APEX FusedRMSNorm not available, using native implementation
/home/ubuntu/vibevoice/vibevoice/processor/vibevoice_asr_processor.py:23: UserWarning: audio_utils not available, will fall back to soundfile for audio loading
  warnings.warn("audio_utils not available, will fall back to soundfile for audio loading")
03/16/2026 06:47:26 - INFO - __main__ - Training/evaluation parameters CustomTrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
average_tokens_across_devices=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
ce_loss_weight=0.04,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=0,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
ddpm_batch_mul=4,
debug=[],
debug_ce_details=False,
debug_ce_every_n_steps=200,
debug_ce_max_examples=1,
debug_ce_topk=5,
debug_save=False,
deepspeed=None,
diffusion_loss_weight=1.4,
disable_tqdm=False,
do_eval=False,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=None,
eval_strategy=no,
eval_use_gather_object=False,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=4,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
gradient_clipping=True,
greater_is_better=None,
group_by_length=False,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=None,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_for_metrics=[],
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=2.5e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/home/ubuntu/vibevoice_finetune_output/runs/Mar16_06-47-25_0321-dsm2-nvdgxa100-prxmx70052,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=10,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=0.8,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=10.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/home/ubuntu/vibevoice_finetune_output,
overwrite_output_dir=False,
past_index=-1,
per_device_eval_batch_size=8,
per_device_train_batch_size=8,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=False,
report_to=[],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=/home/ubuntu/vibevoice_finetune_output,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=200,
save_strategy=steps,
save_total_limit=None,
seed=42,
skip_memory_metrics=True,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tp_size=0,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.0,
)
03/16/2026 06:47:26 - INFO - __main__ - Gradient clipping enabled: max_grad_norm=0.8
03/16/2026 06:47:26 - INFO - vibevoice.processor.vibevoice_processor - Loading tokenizer from Qwen/Qwen2.5-1.5B
The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization. 
The tokenizer class you load from this checkpoint is 'Qwen2Tokenizer'. 
The class this function is called from is 'VibeVoiceTextTokenizerFast'.
03/16/2026 06:47:26 - WARNING - transformers.tokenization_utils_base - The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization. 
The tokenizer class you load from this checkpoint is 'Qwen2Tokenizer'. 
The class this function is called from is 'VibeVoiceTextTokenizerFast'.
Tied input and output embeddings using standard assignment.

Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards:  67%|██████▋   | 2/3 [00:00<00:00, 11.23it/s]
Loading checkpoint shards: 100%|██████████| 3/3 [00:00<00:00,  7.92it/s]
Tied input and output embeddings using standard assignment.
03/16/2026 06:47:29 - INFO - __main__ - Patched acoustic_tokenizer.encode() to return [[...]] for legacy indexing.
03/16/2026 06:47:29 - INFO - __main__ - LM head diagnostics -> shared_params=True, values_equal=True, tie_word_embeddings=True
03/16/2026 06:47:29 - INFO - __main__ - LM head requires_grad before freeze: True
03/16/2026 06:47:29 - INFO - __main__ - Special token check -> speech_start_id=151652, decoded='<|vision_start|>', exists=True, in_vocab_range=True, emb_vs_head_row_equal=True
03/16/2026 06:47:29 - INFO - __main__ - Special token check -> speech_diffusion_id=151654, decoded='<|vision_pad|>', exists=True, in_vocab_range=True, emb_vs_head_row_equal=True
03/16/2026 06:47:29 - INFO - __main__ - Special token check -> speech_end_id=151653, decoded='<|vision_end|>', exists=True, in_vocab_range=True, emb_vs_head_row_equal=True
03/16/2026 06:47:29 - INFO - __main__ - === TOKENIZER DIAGNOSTICS ===
03/16/2026 06:47:29 - INFO - __main__ - Tokenizer class: VibeVoiceTextTokenizerFast
03/16/2026 06:47:29 - INFO - __main__ - Tokenizer vocab_size: 151643
03/16/2026 06:47:30 - INFO - __main__ - Simple text CE loss: 14.8125
Tied input and output embeddings using standard assignment.
03/16/2026 06:47:30 - INFO - __main__ - Trainable by block -> LLM-LoRA: 9,232,384 | diff_head: 123,279,360 | ac_conn: 0 | se_conn: 0
03/16/2026 06:47:30 - INFO - __main__ - TOTAL trainable: 132,511,744
03/16/2026 06:47:31 - INFO - __main__ - LoRA debug: found 392 LoRA params (A=196, B=196); trainable=392. Initial lora_B_zero=196.

  0%|          | 0/730 [00:00<?, ?it/s]
                                       
{'debug/num_tok_total': 2294.0, 'debug/num_tok_loss': 1603.0, 'debug/num_lat_total': 2294.0, 'debug/num_lat_loss': 1603.0, 'epoch': 0}

  0%|          | 0/730 [00:02<?, ?it/s]
                                       
{'train/ce_loss': 23.25, 'train/diffusion_loss': 0.5052313208580017, 'epoch': 0}

  0%|          | 0/730 [00:02<?, ?it/s]
                                       
{'train/learning_rate_real': 0.0, 'epoch': 0}

  0%|          | 0/730 [00:02<?, ?it/s]
                                       
{'debug/num_tok_total': 2404.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2404.0, 'debug/num_lat_loss': 1770.0, 'epoch': 0}

  0%|          | 0/730 [00:03<?, ?it/s]
                                       
{'train/ce_loss': 23.375, 'train/diffusion_loss': 0.5152708888053894, 'epoch': 0}

  0%|          | 0/730 [00:03<?, ?it/s]
                                       
{'train/learning_rate_real': 0.0, 'epoch': 0}

  0%|          | 0/730 [00:03<?, ?it/s]
                                       
{'debug/num_tok_total': 2651.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2651.0, 'debug/num_lat_loss': 1787.0, 'epoch': 0}

  0%|          | 0/730 [00:04<?, ?it/s]
                                       
{'train/ce_loss': 23.75, 'train/diffusion_loss': 0.49939244985580444, 'epoch': 0}

  0%|          | 0/730 [00:04<?, ?it/s]
                                       
{'train/learning_rate_real': 0.0, 'epoch': 0}

  0%|          | 0/730 [00:04<?, ?it/s]
                                       
{'debug/num_tok_total': 2892.0, 'debug/num_tok_loss': 1809.0, 'debug/num_lat_total': 2892.0, 'debug/num_lat_loss': 1809.0, 'epoch': 0}

  0%|          | 0/730 [00:06<?, ?it/s]
                                       
{'train/ce_loss': 23.5, 'train/diffusion_loss': 0.4265933632850647, 'epoch': 0}

  0%|          | 0/730 [00:06<?, ?it/s]
                                       
{'train/learning_rate_real': 0.0, 'epoch': 0}

  0%|          | 0/730 [00:06<?, ?it/s]03/16/2026 06:47:38 - INFO - __main__ - LoRA debug step 1: changed A 0/196, changed B 0/196, lora_B_zero_now=196.

  0%|          | 1/730 [00:06<1:21:04,  6.67s/it]
                                                 
{'debug/num_tok_total': 3242.0, 'debug/num_tok_loss': 1753.0, 'debug/num_lat_total': 3242.0, 'debug/num_lat_loss': 1753.0, 'epoch': 0.01}

  0%|          | 1/730 [00:07<1:21:04,  6.67s/it]
                                                 
{'train/ce_loss': 23.625, 'train/diffusion_loss': 0.35353660583496094, 'epoch': 0.01}

  0%|          | 1/730 [00:08<1:21:04,  6.67s/it]
                                                 
{'train/learning_rate_real': 1.1363636363636364e-06, 'epoch': 0.01}

  0%|          | 1/730 [00:08<1:21:04,  6.67s/it]
                                                 
{'debug/num_tok_total': 2013.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2013.0, 'debug/num_lat_loss': 1794.0, 'epoch': 0.01}

  0%|          | 1/730 [00:09<1:21:04,  6.67s/it]
                                                 
{'train/ce_loss': 23.625, 'train/diffusion_loss': 0.5851874947547913, 'epoch': 0.01}

  0%|          | 1/730 [00:09<1:21:04,  6.67s/it]
                                                 
{'train/learning_rate_real': 1.1363636363636364e-06, 'epoch': 0.01}

  0%|          | 1/730 [00:09<1:21:04,  6.67s/it]
                                                 
{'debug/num_tok_total': 2652.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2652.0, 'debug/num_lat_loss': 1788.0, 'epoch': 0.01}

  0%|          | 1/730 [00:10<1:21:04,  6.67s/it]
                                                 
{'train/ce_loss': 23.625, 'train/diffusion_loss': 0.49107739329338074, 'epoch': 0.01}

  0%|          | 1/730 [00:10<1:21:04,  6.67s/it]
                                                 
{'train/learning_rate_real': 1.1363636363636364e-06, 'epoch': 0.01}

  0%|          | 1/730 [00:10<1:21:04,  6.67s/it]
                                                 
{'debug/num_tok_total': 2911.0, 'debug/num_tok_loss': 1818.0, 'debug/num_lat_total': 2911.0, 'debug/num_lat_loss': 1818.0, 'epoch': 0.01}

  0%|          | 1/730 [00:11<1:21:04,  6.67s/it]
                                                 
{'train/ce_loss': 23.625, 'train/diffusion_loss': 0.461406946182251, 'epoch': 0.01}

  0%|          | 1/730 [00:11<1:21:04,  6.67s/it]
                                                 
{'train/learning_rate_real': 1.1363636363636364e-06, 'epoch': 0.01}

  0%|          | 1/730 [00:11<1:21:04,  6.67s/it]
  0%|          | 2/730 [00:12<1:12:33,  5.98s/it]
                                                 
{'debug/num_tok_total': 2128.0, 'debug/num_tok_loss': 1642.0, 'debug/num_lat_total': 2128.0, 'debug/num_lat_loss': 1642.0, 'epoch': 0.03}

  0%|          | 2/730 [00:13<1:12:33,  5.98s/it]
                                                 
{'train/ce_loss': 23.375, 'train/diffusion_loss': 0.5282515287399292, 'epoch': 0.03}

  0%|          | 2/730 [00:13<1:12:33,  5.98s/it]
                                                 
{'train/learning_rate_real': 2.2727272727272728e-06, 'epoch': 0.03}

  0%|          | 2/730 [00:13<1:12:33,  5.98s/it]
                                                 
{'debug/num_tok_total': 2405.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2405.0, 'debug/num_lat_loss': 1772.0, 'epoch': 0.03}

  0%|          | 2/730 [00:14<1:12:33,  5.98s/it]
                                                 
{'train/ce_loss': 23.5, 'train/diffusion_loss': 0.5025462508201599, 'epoch': 0.03}

  0%|          | 2/730 [00:14<1:12:33,  5.98s/it]
                                                 
{'train/learning_rate_real': 2.2727272727272728e-06, 'epoch': 0.03}

  0%|          | 2/730 [00:14<1:12:33,  5.98s/it]
                                                 
{'debug/num_tok_total': 2867.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2867.0, 'debug/num_lat_loss': 1783.0, 'epoch': 0.03}

  0%|          | 2/730 [00:15<1:12:33,  5.98s/it]
                                                 
{'train/ce_loss': 23.125, 'train/diffusion_loss': 0.42426061630249023, 'epoch': 0.03}

  0%|          | 2/730 [00:15<1:12:33,  5.98s/it]
                                                 
{'train/learning_rate_real': 2.2727272727272728e-06, 'epoch': 0.03}

  0%|          | 2/730 [00:15<1:12:33,  5.98s/it]
                                                 
{'debug/num_tok_total': 2234.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2234.0, 'debug/num_lat_loss': 1785.0, 'epoch': 0.03}

  0%|          | 2/730 [00:16<1:12:33,  5.98s/it]
                                                 
{'train/ce_loss': 24.125, 'train/diffusion_loss': 0.5727062225341797, 'epoch': 0.03}

  0%|          | 2/730 [00:16<1:12:33,  5.98s/it]
                                                 
{'train/learning_rate_real': 2.2727272727272728e-06, 'epoch': 0.03}

  0%|          | 2/730 [00:16<1:12:33,  5.98s/it]
  0%|          | 3/730 [00:17<1:08:33,  5.66s/it]
                                                 
{'debug/num_tok_total': 3015.0, 'debug/num_tok_loss': 1648.0, 'debug/num_lat_total': 3015.0, 'debug/num_lat_loss': 1648.0, 'epoch': 0.04}

  0%|          | 3/730 [00:18<1:08:33,  5.66s/it]
                                                 
{'train/ce_loss': 23.625, 'train/diffusion_loss': 0.3512691855430603, 'epoch': 0.04}

  0%|          | 3/730 [00:18<1:08:33,  5.66s/it]
                                                 
{'train/learning_rate_real': 3.409090909090909e-06, 'epoch': 0.04}

  0%|          | 3/730 [00:18<1:08:33,  5.66s/it]
                                                 
{'debug/num_tok_total': 2399.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2399.0, 'debug/num_lat_loss': 1764.0, 'epoch': 0.04}

  0%|          | 3/730 [00:19<1:08:33,  5.66s/it]
                                                 
{'train/ce_loss': 23.875, 'train/diffusion_loss': 0.48421576619148254, 'epoch': 0.04}

  0%|          | 3/730 [00:19<1:08:33,  5.66s/it]
                                                 
{'train/learning_rate_real': 3.409090909090909e-06, 'epoch': 0.04}

  0%|          | 3/730 [00:19<1:08:33,  5.66s/it]
                                                 
{'debug/num_tok_total': 2825.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 2825.0, 'debug/num_lat_loss': 1769.0, 'epoch': 0.04}

  0%|          | 3/730 [00:21<1:08:33,  5.66s/it]
                                                 
{'train/ce_loss': 23.75, 'train/diffusion_loss': 0.4260019361972809, 'epoch': 0.04}

  0%|          | 3/730 [00:21<1:08:33,  5.66s/it]
                                                 
{'train/learning_rate_real': 3.409090909090909e-06, 'epoch': 0.04}

  0%|          | 3/730 [00:21<1:08:33,  5.66s/it]
                                                 
{'debug/num_tok_total': 2866.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2866.0, 'debug/num_lat_loss': 1792.0, 'epoch': 0.04}

  0%|          | 3/730 [00:22<1:08:33,  5.66s/it]
                                                 
{'train/ce_loss': 23.625, 'train/diffusion_loss': 0.42226922512054443, 'epoch': 0.04}

  0%|          | 3/730 [00:22<1:08:33,  5.66s/it]
                                                 
{'train/learning_rate_real': 3.409090909090909e-06, 'epoch': 0.04}

  0%|          | 3/730 [00:22<1:08:33,  5.66s/it]
  1%|          | 4/730 [00:22<1:07:30,  5.58s/it]
                                                 
{'debug/num_tok_total': 2354.0, 'debug/num_tok_loss': 1633.0, 'debug/num_lat_total': 2354.0, 'debug/num_lat_loss': 1633.0, 'epoch': 0.05}

  1%|          | 4/730 [00:24<1:07:30,  5.58s/it]
                                                 
{'train/ce_loss': 23.25, 'train/diffusion_loss': 0.4710257947444916, 'epoch': 0.05}

  1%|          | 4/730 [00:24<1:07:30,  5.58s/it]
                                                 
{'train/learning_rate_real': 4.5454545454545455e-06, 'epoch': 0.05}

  1%|          | 4/730 [00:24<1:07:30,  5.58s/it]
                                                 
{'debug/num_tok_total': 2670.0, 'debug/num_tok_loss': 1811.0, 'debug/num_lat_total': 2670.0, 'debug/num_lat_loss': 1811.0, 'epoch': 0.05}

  1%|          | 4/730 [00:25<1:07:30,  5.58s/it]
                                                 
{'train/ce_loss': 24.0, 'train/diffusion_loss': 0.4669632911682129, 'epoch': 0.05}

  1%|          | 4/730 [00:25<1:07:30,  5.58s/it]
                                                 
{'train/learning_rate_real': 4.5454545454545455e-06, 'epoch': 0.05}

  1%|          | 4/730 [00:25<1:07:30,  5.58s/it]
                                                 
{'debug/num_tok_total': 2660.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 2660.0, 'debug/num_lat_loss': 1801.0, 'epoch': 0.05}

  1%|          | 4/730 [00:26<1:07:30,  5.58s/it]
                                                 
{'train/ce_loss': 23.75, 'train/diffusion_loss': 0.4852781295776367, 'epoch': 0.05}

  1%|          | 4/730 [00:26<1:07:30,  5.58s/it]
                                                 
{'train/learning_rate_real': 4.5454545454545455e-06, 'epoch': 0.05}

  1%|          | 4/730 [00:26<1:07:30,  5.58s/it]
                                                 
{'debug/num_tok_total': 2247.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2247.0, 'debug/num_lat_loss': 1799.0, 'epoch': 0.05}

  1%|          | 4/730 [00:27<1:07:30,  5.58s/it]
                                                 
{'train/ce_loss': 23.5, 'train/diffusion_loss': 0.5597466826438904, 'epoch': 0.05}

  1%|          | 4/730 [00:27<1:07:30,  5.58s/it]
                                                 
{'train/learning_rate_real': 4.5454545454545455e-06, 'epoch': 0.05}

  1%|          | 4/730 [00:27<1:07:30,  5.58s/it]
  1%|          | 5/730 [00:28<1:06:18,  5.49s/it]
                                                 
{'debug/num_tok_total': 2583.0, 'debug/num_tok_loss': 1755.0, 'debug/num_lat_total': 2583.0, 'debug/num_lat_loss': 1755.0, 'epoch': 0.07}

  1%|          | 5/730 [00:29<1:06:18,  5.49s/it]
                                                 
{'train/ce_loss': 23.625, 'train/diffusion_loss': 0.48908230662345886, 'epoch': 0.07}

  1%|          | 5/730 [00:29<1:06:18,  5.49s/it]
                                                 
{'train/learning_rate_real': 5.681818181818182e-06, 'epoch': 0.07}

  1%|          | 5/730 [00:29<1:06:18,  5.49s/it]
                                                 
{'debug/num_tok_total': 2606.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2606.0, 'debug/num_lat_loss': 1766.0, 'epoch': 0.07}

  1%|          | 5/730 [00:30<1:06:18,  5.49s/it]
                                                 
{'train/ce_loss': 23.375, 'train/diffusion_loss': 0.4757000505924225, 'epoch': 0.07}

  1%|          | 5/730 [00:30<1:06:18,  5.49s/it]
                                                 
{'train/learning_rate_real': 5.681818181818182e-06, 'epoch': 0.07}

  1%|          | 5/730 [00:30<1:06:18,  5.49s/it]
                                                 
{'debug/num_tok_total': 2459.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 2459.0, 'debug/num_lat_loss': 1797.0, 'epoch': 0.07}

  1%|          | 5/730 [00:31<1:06:18,  5.49s/it]
                                                 
{'train/ce_loss': 23.75, 'train/diffusion_loss': 0.5041497945785522, 'epoch': 0.07}

  1%|          | 5/730 [00:31<1:06:18,  5.49s/it]
                                                 
{'train/learning_rate_real': 5.681818181818182e-06, 'epoch': 0.07}

  1%|          | 5/730 [00:31<1:06:18,  5.49s/it]
                                                 
{'debug/num_tok_total': 1823.0, 'debug/num_tok_loss': 1617.0, 'debug/num_lat_total': 1823.0, 'debug/num_lat_loss': 1617.0, 'epoch': 0.07}

  1%|          | 5/730 [00:32<1:06:18,  5.49s/it]
                                                 
{'train/ce_loss': 23.625, 'train/diffusion_loss': 0.5805519223213196, 'epoch': 0.07}

  1%|          | 5/730 [00:32<1:06:18,  5.49s/it]
                                                 
{'train/learning_rate_real': 5.681818181818182e-06, 'epoch': 0.07}

  1%|          | 5/730 [00:32<1:06:18,  5.49s/it]
  1%|          | 6/730 [00:33<1:04:25,  5.34s/it]
                                                 
{'debug/num_tok_total': 2836.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2836.0, 'debug/num_lat_loss': 1775.0, 'epoch': 0.08}

  1%|          | 6/730 [00:34<1:04:25,  5.34s/it]
                                                 
{'train/ce_loss': 23.25, 'train/diffusion_loss': 0.4666789472103119, 'epoch': 0.08}

  1%|          | 6/730 [00:34<1:04:25,  5.34s/it]
                                                 
{'train/learning_rate_real': 6.818181818181818e-06, 'epoch': 0.08}

  1%|          | 6/730 [00:34<1:04:25,  5.34s/it]
                                                 
{'debug/num_tok_total': 2437.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2437.0, 'debug/num_lat_loss': 1788.0, 'epoch': 0.08}

  1%|          | 6/730 [00:35<1:04:25,  5.34s/it]
                                                 
{'train/ce_loss': 23.5, 'train/diffusion_loss': 0.5358713865280151, 'epoch': 0.08}

  1%|          | 6/730 [00:35<1:04:25,  5.34s/it]
                                                 
{'train/learning_rate_real': 6.818181818181818e-06, 'epoch': 0.08}

  1%|          | 6/730 [00:35<1:04:25,  5.34s/it]
                                                 
{'debug/num_tok_total': 2344.0, 'debug/num_tok_loss': 1634.0, 'debug/num_lat_total': 2344.0, 'debug/num_lat_loss': 1634.0, 'epoch': 0.08}

  1%|          | 6/730 [00:36<1:04:25,  5.34s/it]
                                                 
{'train/ce_loss': 23.25, 'train/diffusion_loss': 0.4909515976905823, 'epoch': 0.08}

  1%|          | 6/730 [00:36<1:04:25,  5.34s/it]
                                                 
{'train/learning_rate_real': 6.818181818181818e-06, 'epoch': 0.08}

  1%|          | 6/730 [00:36<1:04:25,  5.34s/it]
                                                 
{'debug/num_tok_total': 3088.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 3088.0, 'debug/num_lat_loss': 1788.0, 'epoch': 0.08}

  1%|          | 6/730 [00:38<1:04:25,  5.34s/it]
                                                 
{'train/ce_loss': 23.5, 'train/diffusion_loss': 0.3803665339946747, 'epoch': 0.08}

  1%|          | 6/730 [00:38<1:04:25,  5.34s/it]
                                                 
{'train/learning_rate_real': 6.818181818181818e-06, 'epoch': 0.08}

  1%|          | 6/730 [00:38<1:04:25,  5.34s/it]
  1%|          | 7/730 [00:39<1:05:53,  5.47s/it]
                                                 
{'debug/num_tok_total': 2623.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2623.0, 'debug/num_lat_loss': 1774.0, 'epoch': 0.1}

  1%|          | 7/730 [00:40<1:05:53,  5.47s/it]
                                                 
{'train/ce_loss': 23.25, 'train/diffusion_loss': 0.4644874334335327, 'epoch': 0.1}

  1%|          | 7/730 [00:40<1:05:53,  5.47s/it]
                                                 
{'train/learning_rate_real': 7.954545454545455e-06, 'epoch': 0.1}

  1%|          | 7/730 [00:40<1:05:53,  5.47s/it]
                                                 
{'debug/num_tok_total': 2829.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 2829.0, 'debug/num_lat_loss': 1765.0, 'epoch': 0.1}

  1%|          | 7/730 [00:41<1:05:53,  5.47s/it]
                                                 
{'train/ce_loss': 23.375, 'train/diffusion_loss': 0.42349711060523987, 'epoch': 0.1}

  1%|          | 7/730 [00:41<1:05:53,  5.47s/it]
                                                 
{'train/learning_rate_real': 7.954545454545455e-06, 'epoch': 0.1}

  1%|          | 7/730 [00:41<1:05:53,  5.47s/it]
                                                 
{'debug/num_tok_total': 2279.0, 'debug/num_tok_loss': 1600.0, 'debug/num_lat_total': 2279.0, 'debug/num_lat_loss': 1600.0, 'epoch': 0.1}

  1%|          | 7/730 [00:42<1:05:53,  5.47s/it]
                                                 
{'train/ce_loss': 23.0, 'train/diffusion_loss': 0.4753822982311249, 'epoch': 0.1}

  1%|          | 7/730 [00:42<1:05:53,  5.47s/it]
                                                 
{'train/learning_rate_real': 7.954545454545455e-06, 'epoch': 0.1}

  1%|          | 7/730 [00:42<1:05:53,  5.47s/it]
                                                 
{'debug/num_tok_total': 2679.0, 'debug/num_tok_loss': 1806.0, 'debug/num_lat_total': 2679.0, 'debug/num_lat_loss': 1806.0, 'epoch': 0.1}

  1%|          | 7/730 [00:44<1:05:53,  5.47s/it]
                                                 
{'train/ce_loss': 23.5, 'train/diffusion_loss': 0.4810636341571808, 'epoch': 0.1}

  1%|          | 7/730 [00:44<1:05:53,  5.47s/it]
                                                 
{'train/learning_rate_real': 7.954545454545455e-06, 'epoch': 0.1}

  1%|          | 7/730 [00:44<1:05:53,  5.47s/it]
  1%|          | 8/730 [00:44<1:05:57,  5.48s/it]
                                                 
{'debug/num_tok_total': 2348.0, 'debug/num_tok_loss': 1699.0, 'debug/num_lat_total': 2348.0, 'debug/num_lat_loss': 1699.0, 'epoch': 0.11}

  1%|          | 8/730 [00:45<1:05:57,  5.48s/it]
                                                 
{'train/ce_loss': 23.0, 'train/diffusion_loss': 0.5147892832756042, 'epoch': 0.11}

  1%|          | 8/730 [00:45<1:05:57,  5.48s/it]
                                                 
{'train/learning_rate_real': 9.090909090909091e-06, 'epoch': 0.11}

  1%|          | 8/730 [00:45<1:05:57,  5.48s/it]
                                                 
{'debug/num_tok_total': 3080.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 3080.0, 'debug/num_lat_loss': 1780.0, 'epoch': 0.11}

  1%|          | 8/730 [00:47<1:05:57,  5.48s/it]
                                                 
{'train/ce_loss': 23.25, 'train/diffusion_loss': 0.3648412227630615, 'epoch': 0.11}

  1%|          | 8/730 [00:47<1:05:57,  5.48s/it]
                                                 
{'train/learning_rate_real': 9.090909090909091e-06, 'epoch': 0.11}

  1%|          | 8/730 [00:47<1:05:57,  5.48s/it]
                                                 
{'debug/num_tok_total': 2632.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 2632.0, 'debug/num_lat_loss': 1765.0, 'epoch': 0.11}

  1%|          | 8/730 [00:48<1:05:57,  5.48s/it]
                                                 
{'train/ce_loss': 23.25, 'train/diffusion_loss': 0.4737395942211151, 'epoch': 0.11}

  1%|          | 8/730 [00:48<1:05:57,  5.48s/it]
                                                 
{'train/learning_rate_real': 9.090909090909091e-06, 'epoch': 0.11}

  1%|          | 8/730 [00:48<1:05:57,  5.48s/it]
                                                 
{'debug/num_tok_total': 2647.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2647.0, 'debug/num_lat_loss': 1788.0, 'epoch': 0.11}

  1%|          | 8/730 [00:49<1:05:57,  5.48s/it]
                                                 
{'train/ce_loss': 23.125, 'train/diffusion_loss': 0.4529899060726166, 'epoch': 0.11}

  1%|          | 8/730 [00:49<1:05:57,  5.48s/it]
                                                 
{'train/learning_rate_real': 9.090909090909091e-06, 'epoch': 0.11}

  1%|          | 8/730 [00:49<1:05:57,  5.48s/it]
  1%|          | 9/730 [00:49<1:05:50,  5.48s/it]
                                                 
{'debug/num_tok_total': 2645.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2645.0, 'debug/num_lat_loss': 1787.0, 'epoch': 0.12}

  1%|          | 9/730 [00:51<1:05:50,  5.48s/it]
                                                 
{'train/ce_loss': 22.875, 'train/diffusion_loss': 0.460472971200943, 'epoch': 0.12}

  1%|          | 9/730 [00:51<1:05:50,  5.48s/it]
                                                 
{'train/learning_rate_real': 1.0227272727272729e-05, 'epoch': 0.12}

  1%|          | 9/730 [00:51<1:05:50,  5.48s/it]
                                                 
{'debug/num_tok_total': 2845.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2845.0, 'debug/num_lat_loss': 1775.0, 'epoch': 0.12}

  1%|          | 9/730 [00:52<1:05:50,  5.48s/it]
                                                 
{'train/ce_loss': 22.875, 'train/diffusion_loss': 0.39518892765045166, 'epoch': 0.12}

  1%|          | 9/730 [00:52<1:05:50,  5.48s/it]
                                                 
{'train/learning_rate_real': 1.0227272727272729e-05, 'epoch': 0.12}

  1%|          | 9/730 [00:52<1:05:50,  5.48s/it]
                                                 
{'debug/num_tok_total': 2689.0, 'debug/num_tok_loss': 1814.0, 'debug/num_lat_total': 2689.0, 'debug/num_lat_loss': 1814.0, 'epoch': 0.12}

  1%|          | 9/730 [00:53<1:05:50,  5.48s/it]
                                                 
{'train/ce_loss': 22.875, 'train/diffusion_loss': 0.4926188588142395, 'epoch': 0.12}

  1%|          | 9/730 [00:53<1:05:50,  5.48s/it]
                                                 
{'train/learning_rate_real': 1.0227272727272729e-05, 'epoch': 0.12}

  1%|          | 9/730 [00:53<1:05:50,  5.48s/it]
                                                 
{'debug/num_tok_total': 2250.0, 'debug/num_tok_loss': 1575.0, 'debug/num_lat_total': 2250.0, 'debug/num_lat_loss': 1575.0, 'epoch': 0.12}

  1%|          | 9/730 [00:54<1:05:50,  5.48s/it]
                                                 
{'train/ce_loss': 22.625, 'train/diffusion_loss': 0.48270726203918457, 'epoch': 0.12}

  1%|          | 9/730 [00:54<1:05:50,  5.48s/it]
                                                 
{'train/learning_rate_real': 1.0227272727272729e-05, 'epoch': 0.12}

  1%|          | 9/730 [00:54<1:05:50,  5.48s/it]03/16/2026 06:48:27 - INFO - __main__ - LoRA debug step 10: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

  1%|▏         | 10/730 [00:55<1:05:22,  5.45s/it]
                                                  
{'loss': 6.3985, 'grad_norm': 11.971195220947266, 'learning_rate': 1.0227272727272729e-05, 'epoch': 0.14}

  1%|▏         | 10/730 [00:55<1:05:22,  5.45s/it]
                                                  
{'debug/num_tok_total': 3053.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 3053.0, 'debug/num_lat_loss': 1770.0, 'epoch': 0.14}

  1%|▏         | 10/730 [00:56<1:05:22,  5.45s/it]
                                                  
{'train/ce_loss': 22.875, 'train/diffusion_loss': 0.35791292786598206, 'epoch': 0.14}

  1%|▏         | 10/730 [00:56<1:05:22,  5.45s/it]
                                                  
{'train/learning_rate_real': 1.1363636363636365e-05, 'epoch': 0.14}

  1%|▏         | 10/730 [00:56<1:05:22,  5.45s/it]
                                                  
{'debug/num_tok_total': 3087.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 3087.0, 'debug/num_lat_loss': 1795.0, 'epoch': 0.14}

  1%|▏         | 10/730 [00:57<1:05:22,  5.45s/it]
                                                  
{'train/ce_loss': 22.875, 'train/diffusion_loss': 0.38981443643569946, 'epoch': 0.14}

  1%|▏         | 10/730 [00:57<1:05:22,  5.45s/it]
                                                  
{'train/learning_rate_real': 1.1363636363636365e-05, 'epoch': 0.14}

  1%|▏         | 10/730 [00:57<1:05:22,  5.45s/it]
                                                  
{'debug/num_tok_total': 2231.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2231.0, 'debug/num_lat_loss': 1793.0, 'epoch': 0.14}

  1%|▏         | 10/730 [00:59<1:05:22,  5.45s/it]
                                                  
{'train/ce_loss': 22.5, 'train/diffusion_loss': 0.5622515678405762, 'epoch': 0.14}

  1%|▏         | 10/730 [00:59<1:05:22,  5.45s/it]
                                                  
{'train/learning_rate_real': 1.1363636363636365e-05, 'epoch': 0.14}

  1%|▏         | 10/730 [00:59<1:05:22,  5.45s/it]
                                                  
{'debug/num_tok_total': 2867.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2867.0, 'debug/num_lat_loss': 1788.0, 'epoch': 0.14}

  1%|▏         | 10/730 [01:00<1:05:22,  5.45s/it]
                                                  
{'train/ce_loss': 22.625, 'train/diffusion_loss': 0.3926439881324768, 'epoch': 0.14}

  1%|▏         | 10/730 [01:00<1:05:22,  5.45s/it]
                                                  
{'train/learning_rate_real': 1.1363636363636365e-05, 'epoch': 0.14}

  1%|▏         | 10/730 [01:00<1:05:22,  5.45s/it]
  2%|▏         | 11/730 [01:00<1:05:24,  5.46s/it]
                                                  
{'debug/num_tok_total': 2437.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2437.0, 'debug/num_lat_loss': 1781.0, 'epoch': 0.15}

  2%|▏         | 11/730 [01:02<1:05:24,  5.46s/it]
                                                  
{'train/ce_loss': 22.375, 'train/diffusion_loss': 0.4806886315345764, 'epoch': 0.15}

  2%|▏         | 11/730 [01:02<1:05:24,  5.46s/it]
                                                  
{'train/learning_rate_real': 1.25e-05, 'epoch': 0.15}

  2%|▏         | 11/730 [01:02<1:05:24,  5.46s/it]
                                                  
{'debug/num_tok_total': 2848.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2848.0, 'debug/num_lat_loss': 1776.0, 'epoch': 0.15}

  2%|▏         | 11/730 [01:03<1:05:24,  5.46s/it]
                                                  
{'train/ce_loss': 22.125, 'train/diffusion_loss': 0.4312594532966614, 'epoch': 0.15}

  2%|▏         | 11/730 [01:03<1:05:24,  5.46s/it]
                                                  
{'train/learning_rate_real': 1.25e-05, 'epoch': 0.15}

  2%|▏         | 11/730 [01:03<1:05:24,  5.46s/it]
                                                  
{'debug/num_tok_total': 2875.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2875.0, 'debug/num_lat_loss': 1788.0, 'epoch': 0.15}

  2%|▏         | 11/730 [01:04<1:05:24,  5.46s/it]
                                                  
{'train/ce_loss': 22.125, 'train/diffusion_loss': 0.4150593876838684, 'epoch': 0.15}

  2%|▏         | 11/730 [01:04<1:05:24,  5.46s/it]
                                                  
{'train/learning_rate_real': 1.25e-05, 'epoch': 0.15}

  2%|▏         | 11/730 [01:04<1:05:24,  5.46s/it]
                                                  
{'debug/num_tok_total': 2418.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2418.0, 'debug/num_lat_loss': 1785.0, 'epoch': 0.15}

  2%|▏         | 11/730 [01:05<1:05:24,  5.46s/it]
                                                  
{'train/ce_loss': 22.0, 'train/diffusion_loss': 0.49140727519989014, 'epoch': 0.15}

  2%|▏         | 11/730 [01:05<1:05:24,  5.46s/it]
                                                  
{'train/learning_rate_real': 1.25e-05, 'epoch': 0.15}

  2%|▏         | 11/730 [01:05<1:05:24,  5.46s/it]
  2%|▏         | 12/730 [01:06<1:05:12,  5.45s/it]
                                                  
{'debug/num_tok_total': 2536.0, 'debug/num_tok_loss': 1732.0, 'debug/num_lat_total': 2536.0, 'debug/num_lat_loss': 1732.0, 'epoch': 0.16}

  2%|▏         | 12/730 [01:07<1:05:12,  5.45s/it]
                                                  
{'train/ce_loss': 21.5, 'train/diffusion_loss': 0.4987460970878601, 'epoch': 0.16}

  2%|▏         | 12/730 [01:07<1:05:12,  5.45s/it]
                                                  
{'train/learning_rate_real': 1.3636363636363637e-05, 'epoch': 0.16}

  2%|▏         | 12/730 [01:07<1:05:12,  5.45s/it]
                                                  
{'debug/num_tok_total': 2399.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2399.0, 'debug/num_lat_loss': 1763.0, 'epoch': 0.16}

  2%|▏         | 12/730 [01:08<1:05:12,  5.45s/it]
                                                  
{'train/ce_loss': 21.375, 'train/diffusion_loss': 0.4842108190059662, 'epoch': 0.16}

  2%|▏         | 12/730 [01:08<1:05:12,  5.45s/it]
                                                  
{'train/learning_rate_real': 1.3636363636363637e-05, 'epoch': 0.16}

  2%|▏         | 12/730 [01:08<1:05:12,  5.45s/it]
                                                  
{'debug/num_tok_total': 2363.0, 'debug/num_tok_loss': 1702.0, 'debug/num_lat_total': 2363.0, 'debug/num_lat_loss': 1702.0, 'epoch': 0.16}

  2%|▏         | 12/730 [01:09<1:05:12,  5.45s/it]
                                                  
{'train/ce_loss': 21.5, 'train/diffusion_loss': 0.4798933267593384, 'epoch': 0.16}

  2%|▏         | 12/730 [01:09<1:05:12,  5.45s/it]
                                                  
{'train/learning_rate_real': 1.3636363636363637e-05, 'epoch': 0.16}

  2%|▏         | 12/730 [01:09<1:05:12,  5.45s/it]
                                                  
{'debug/num_tok_total': 2623.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2623.0, 'debug/num_lat_loss': 1776.0, 'epoch': 0.16}

  2%|▏         | 12/730 [01:10<1:05:12,  5.45s/it]
                                                  
{'train/ce_loss': 21.625, 'train/diffusion_loss': 0.48570314049720764, 'epoch': 0.16}

  2%|▏         | 12/730 [01:10<1:05:12,  5.45s/it]
                                                  
{'train/learning_rate_real': 1.3636363636363637e-05, 'epoch': 0.16}

  2%|▏         | 12/730 [01:10<1:05:12,  5.45s/it]
  2%|▏         | 13/730 [01:11<1:04:00,  5.36s/it]
                                                  
{'debug/num_tok_total': 2404.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 2404.0, 'debug/num_lat_loss': 1765.0, 'epoch': 0.18}

  2%|▏         | 13/730 [01:12<1:04:00,  5.36s/it]
                                                  
{'train/ce_loss': 20.75, 'train/diffusion_loss': 0.4996339678764343, 'epoch': 0.18}

  2%|▏         | 13/730 [01:12<1:04:00,  5.36s/it]
                                                  
{'train/learning_rate_real': 1.4772727272727274e-05, 'epoch': 0.18}

  2%|▏         | 13/730 [01:12<1:04:00,  5.36s/it]
                                                  
{'debug/num_tok_total': 2452.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2452.0, 'debug/num_lat_loss': 1796.0, 'epoch': 0.18}

  2%|▏         | 13/730 [01:13<1:04:00,  5.36s/it]
                                                  
{'train/ce_loss': 20.75, 'train/diffusion_loss': 0.5086291432380676, 'epoch': 0.18}

  2%|▏         | 13/730 [01:13<1:04:00,  5.36s/it]
                                                  
{'train/learning_rate_real': 1.4772727272727274e-05, 'epoch': 0.18}

  2%|▏         | 13/730 [01:13<1:04:00,  5.36s/it]
                                                  
{'debug/num_tok_total': 2656.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2656.0, 'debug/num_lat_loss': 1789.0, 'epoch': 0.18}

  2%|▏         | 13/730 [01:14<1:04:00,  5.36s/it]
                                                  
{'train/ce_loss': 20.875, 'train/diffusion_loss': 0.4561443328857422, 'epoch': 0.18}

  2%|▏         | 13/730 [01:14<1:04:00,  5.36s/it]
                                                  
{'train/learning_rate_real': 1.4772727272727274e-05, 'epoch': 0.18}

  2%|▏         | 13/730 [01:14<1:04:00,  5.36s/it]
                                                  
{'debug/num_tok_total': 3249.0, 'debug/num_tok_loss': 1655.0, 'debug/num_lat_total': 3249.0, 'debug/num_lat_loss': 1655.0, 'epoch': 0.18}

  2%|▏         | 13/730 [01:16<1:04:00,  5.36s/it]
                                                  
{'train/ce_loss': 21.125, 'train/diffusion_loss': 0.26086410880088806, 'epoch': 0.18}

  2%|▏         | 13/730 [01:16<1:04:00,  5.36s/it]
                                                  
{'train/learning_rate_real': 1.4772727272727274e-05, 'epoch': 0.18}

  2%|▏         | 13/730 [01:16<1:04:00,  5.36s/it]
  2%|▏         | 14/730 [01:16<1:04:05,  5.37s/it]
                                                  
{'debug/num_tok_total': 2870.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2870.0, 'debug/num_lat_loss': 1781.0, 'epoch': 0.19}

  2%|▏         | 14/730 [01:18<1:04:05,  5.37s/it]
                                                  
{'train/ce_loss': 20.0, 'train/diffusion_loss': 0.408465713262558, 'epoch': 0.19}

  2%|▏         | 14/730 [01:18<1:04:05,  5.37s/it]
                                                  
{'train/learning_rate_real': 1.590909090909091e-05, 'epoch': 0.19}

  2%|▏         | 14/730 [01:18<1:04:05,  5.37s/it]
                                                  
{'debug/num_tok_total': 2379.0, 'debug/num_tok_loss': 1749.0, 'debug/num_lat_total': 2379.0, 'debug/num_lat_loss': 1749.0, 'epoch': 0.19}

  2%|▏         | 14/730 [01:19<1:04:05,  5.37s/it]
                                                  
{'train/ce_loss': 19.875, 'train/diffusion_loss': 0.4767815172672272, 'epoch': 0.19}

  2%|▏         | 14/730 [01:19<1:04:05,  5.37s/it]
                                                  
{'train/learning_rate_real': 1.590909090909091e-05, 'epoch': 0.19}

  2%|▏         | 14/730 [01:19<1:04:05,  5.37s/it]
                                                  
{'debug/num_tok_total': 2612.0, 'debug/num_tok_loss': 1728.0, 'debug/num_lat_total': 2612.0, 'debug/num_lat_loss': 1728.0, 'epoch': 0.19}

  2%|▏         | 14/730 [01:20<1:04:05,  5.37s/it]
                                                  
{'train/ce_loss': 20.0, 'train/diffusion_loss': 0.45503005385398865, 'epoch': 0.19}

  2%|▏         | 14/730 [01:20<1:04:05,  5.37s/it]
                                                  
{'train/learning_rate_real': 1.590909090909091e-05, 'epoch': 0.19}

  2%|▏         | 14/730 [01:20<1:04:05,  5.37s/it]
                                                  
{'debug/num_tok_total': 2892.0, 'debug/num_tok_loss': 1806.0, 'debug/num_lat_total': 2892.0, 'debug/num_lat_loss': 1806.0, 'epoch': 0.19}

  2%|▏         | 14/730 [01:21<1:04:05,  5.37s/it]
                                                  
{'train/ce_loss': 20.125, 'train/diffusion_loss': 0.4370043873786926, 'epoch': 0.19}

  2%|▏         | 14/730 [01:21<1:04:05,  5.37s/it]
                                                  
{'train/learning_rate_real': 1.590909090909091e-05, 'epoch': 0.19}

  2%|▏         | 14/730 [01:21<1:04:05,  5.37s/it]
  2%|▏         | 15/730 [01:22<1:04:07,  5.38s/it]
                                                  
{'debug/num_tok_total': 2617.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2617.0, 'debug/num_lat_loss': 1775.0, 'epoch': 0.21}

  2%|▏         | 15/730 [01:23<1:04:07,  5.38s/it]
                                                  
{'train/ce_loss': 18.875, 'train/diffusion_loss': 0.45287543535232544, 'epoch': 0.21}

  2%|▏         | 15/730 [01:23<1:04:07,  5.38s/it]
                                                  
{'train/learning_rate_real': 1.7045454545454546e-05, 'epoch': 0.21}

  2%|▏         | 15/730 [01:23<1:04:07,  5.38s/it]
                                                  
{'debug/num_tok_total': 2454.0, 'debug/num_tok_loss': 1804.0, 'debug/num_lat_total': 2454.0, 'debug/num_lat_loss': 1804.0, 'epoch': 0.21}

  2%|▏         | 15/730 [01:24<1:04:07,  5.38s/it]
                                                  
{'train/ce_loss': 18.875, 'train/diffusion_loss': 0.5138697028160095, 'epoch': 0.21}

  2%|▏         | 15/730 [01:24<1:04:07,  5.38s/it]
                                                  
{'train/learning_rate_real': 1.7045454545454546e-05, 'epoch': 0.21}

  2%|▏         | 15/730 [01:24<1:04:07,  5.38s/it]
                                                  
{'debug/num_tok_total': 2184.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 2184.0, 'debug/num_lat_loss': 1767.0, 'epoch': 0.21}

  2%|▏         | 15/730 [01:25<1:04:07,  5.38s/it]
                                                  
{'train/ce_loss': 18.75, 'train/diffusion_loss': 0.5638265013694763, 'epoch': 0.21}

  2%|▏         | 15/730 [01:25<1:04:07,  5.38s/it]
                                                  
{'train/learning_rate_real': 1.7045454545454546e-05, 'epoch': 0.21}

  2%|▏         | 15/730 [01:25<1:04:07,  5.38s/it]
                                                  
{'debug/num_tok_total': 1998.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 1998.0, 'debug/num_lat_loss': 1773.0, 'epoch': 0.21}

  2%|▏         | 15/730 [01:26<1:04:07,  5.38s/it]
                                                  
{'train/ce_loss': 18.625, 'train/diffusion_loss': 0.5613153576850891, 'epoch': 0.21}

  2%|▏         | 15/730 [01:26<1:04:07,  5.38s/it]
                                                  
{'train/learning_rate_real': 1.7045454545454546e-05, 'epoch': 0.21}

  2%|▏         | 15/730 [01:26<1:04:07,  5.38s/it]
  2%|▏         | 16/730 [01:27<1:02:35,  5.26s/it]
                                                  
{'debug/num_tok_total': 2632.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 2632.0, 'debug/num_lat_loss': 1768.0, 'epoch': 0.22}

  2%|▏         | 16/730 [01:28<1:02:35,  5.26s/it]
                                                  
{'train/ce_loss': 18.0, 'train/diffusion_loss': 0.4571579396724701, 'epoch': 0.22}

  2%|▏         | 16/730 [01:28<1:02:35,  5.26s/it]
                                                  
{'train/learning_rate_real': 1.8181818181818182e-05, 'epoch': 0.22}

  2%|▏         | 16/730 [01:28<1:02:35,  5.26s/it]
                                                  
{'debug/num_tok_total': 3095.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 3095.0, 'debug/num_lat_loss': 1792.0, 'epoch': 0.22}

  2%|▏         | 16/730 [01:29<1:02:35,  5.26s/it]
                                                  
{'train/ce_loss': 18.25, 'train/diffusion_loss': 0.3644043803215027, 'epoch': 0.22}

  2%|▏         | 16/730 [01:29<1:02:35,  5.26s/it]
                                                  
{'train/learning_rate_real': 1.8181818181818182e-05, 'epoch': 0.22}

  2%|▏         | 16/730 [01:29<1:02:35,  5.26s/it]
                                                  
{'debug/num_tok_total': 2632.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2632.0, 'debug/num_lat_loss': 1771.0, 'epoch': 0.22}

  2%|▏         | 16/730 [01:31<1:02:35,  5.26s/it]
                                                  
{'train/ce_loss': 18.125, 'train/diffusion_loss': 0.4329890012741089, 'epoch': 0.22}

  2%|▏         | 16/730 [01:31<1:02:35,  5.26s/it]
                                                  
{'train/learning_rate_real': 1.8181818181818182e-05, 'epoch': 0.22}

  2%|▏         | 16/730 [01:31<1:02:35,  5.26s/it]
                                                  
{'debug/num_tok_total': 2653.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2653.0, 'debug/num_lat_loss': 1784.0, 'epoch': 0.22}

  2%|▏         | 16/730 [01:32<1:02:35,  5.26s/it]
                                                  
{'train/ce_loss': 17.75, 'train/diffusion_loss': 0.4439554512500763, 'epoch': 0.22}

  2%|▏         | 16/730 [01:32<1:02:35,  5.26s/it]
                                                  
{'train/learning_rate_real': 1.8181818181818182e-05, 'epoch': 0.22}

  2%|▏         | 16/730 [01:32<1:02:35,  5.26s/it]
  2%|▏         | 17/730 [01:32<1:03:54,  5.38s/it]
                                                  
{'debug/num_tok_total': 2760.0, 'debug/num_tok_loss': 1679.0, 'debug/num_lat_total': 2760.0, 'debug/num_lat_loss': 1679.0, 'epoch': 0.23}

  2%|▏         | 17/730 [01:34<1:03:54,  5.38s/it]
                                                  
{'train/ce_loss': 17.125, 'train/diffusion_loss': 0.41194167733192444, 'epoch': 0.23}

  2%|▏         | 17/730 [01:34<1:03:54,  5.38s/it]
                                                  
{'train/learning_rate_real': 1.9318181818181818e-05, 'epoch': 0.23}

  2%|▏         | 17/730 [01:34<1:03:54,  5.38s/it]
                                                  
{'debug/num_tok_total': 2848.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2848.0, 'debug/num_lat_loss': 1775.0, 'epoch': 0.23}

  2%|▏         | 17/730 [01:35<1:03:54,  5.38s/it]
                                                  
{'train/ce_loss': 16.875, 'train/diffusion_loss': 0.4312850832939148, 'epoch': 0.23}

  2%|▏         | 17/730 [01:35<1:03:54,  5.38s/it]
                                                  
{'train/learning_rate_real': 1.9318181818181818e-05, 'epoch': 0.23}

  2%|▏         | 17/730 [01:35<1:03:54,  5.38s/it]
                                                  
{'debug/num_tok_total': 2575.0, 'debug/num_tok_loss': 1754.0, 'debug/num_lat_total': 2575.0, 'debug/num_lat_loss': 1754.0, 'epoch': 0.23}

  2%|▏         | 17/730 [01:36<1:03:54,  5.38s/it]
                                                  
{'train/ce_loss': 16.875, 'train/diffusion_loss': 0.4552731513977051, 'epoch': 0.23}

  2%|▏         | 17/730 [01:36<1:03:54,  5.38s/it]
                                                  
{'train/learning_rate_real': 1.9318181818181818e-05, 'epoch': 0.23}

  2%|▏         | 17/730 [01:36<1:03:54,  5.38s/it]
                                                  
{'debug/num_tok_total': 3136.0, 'debug/num_tok_loss': 1823.0, 'debug/num_lat_total': 3136.0, 'debug/num_lat_loss': 1823.0, 'epoch': 0.23}

  2%|▏         | 17/730 [01:37<1:03:54,  5.38s/it]
                                                  
{'train/ce_loss': 17.375, 'train/diffusion_loss': 0.37979450821876526, 'epoch': 0.23}

  2%|▏         | 17/730 [01:37<1:03:54,  5.38s/it]
                                                  
{'train/learning_rate_real': 1.9318181818181818e-05, 'epoch': 0.23}

  2%|▏         | 17/730 [01:37<1:03:54,  5.38s/it]
  2%|▏         | 18/730 [01:38<1:04:35,  5.44s/it]
                                                  
{'debug/num_tok_total': 2678.0, 'debug/num_tok_loss': 1807.0, 'debug/num_lat_total': 2678.0, 'debug/num_lat_loss': 1807.0, 'epoch': 0.25}

  2%|▏         | 18/730 [01:39<1:04:35,  5.44s/it]
                                                  
{'train/ce_loss': 15.6875, 'train/diffusion_loss': 0.45173999667167664, 'epoch': 0.25}

  2%|▏         | 18/730 [01:39<1:04:35,  5.44s/it]
                                                  
{'train/learning_rate_real': 2.0454545454545457e-05, 'epoch': 0.25}

  2%|▏         | 18/730 [01:39<1:04:35,  5.44s/it]
                                                  
{'debug/num_tok_total': 2446.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2446.0, 'debug/num_lat_loss': 1796.0, 'epoch': 0.25}

  2%|▏         | 18/730 [01:40<1:04:35,  5.44s/it]
                                                  
{'train/ce_loss': 15.625, 'train/diffusion_loss': 0.4798762798309326, 'epoch': 0.25}

  2%|▏         | 18/730 [01:40<1:04:35,  5.44s/it]
                                                  
{'train/learning_rate_real': 2.0454545454545457e-05, 'epoch': 0.25}

  2%|▏         | 18/730 [01:40<1:04:35,  5.44s/it]
                                                  
{'debug/num_tok_total': 2654.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2654.0, 'debug/num_lat_loss': 1792.0, 'epoch': 0.25}

  2%|▏         | 18/730 [01:41<1:04:35,  5.44s/it]
                                                  
{'train/ce_loss': 15.6875, 'train/diffusion_loss': 0.4615324139595032, 'epoch': 0.25}

  2%|▏         | 18/730 [01:41<1:04:35,  5.44s/it]
                                                  
{'train/learning_rate_real': 2.0454545454545457e-05, 'epoch': 0.25}

  2%|▏         | 18/730 [01:41<1:04:35,  5.44s/it]
                                                  
{'debug/num_tok_total': 2720.0, 'debug/num_tok_loss': 1628.0, 'debug/num_lat_total': 2720.0, 'debug/num_lat_loss': 1628.0, 'epoch': 0.25}

  2%|▏         | 18/730 [01:43<1:04:35,  5.44s/it]
                                                  
{'train/ce_loss': 16.25, 'train/diffusion_loss': 0.365399032831192, 'epoch': 0.25}

  2%|▏         | 18/730 [01:43<1:04:35,  5.44s/it]
                                                  
{'train/learning_rate_real': 2.0454545454545457e-05, 'epoch': 0.25}

  2%|▏         | 18/730 [01:43<1:04:35,  5.44s/it]
  3%|▎         | 19/730 [01:43<1:04:06,  5.41s/it]
                                                  
{'debug/num_tok_total': 2431.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2431.0, 'debug/num_lat_loss': 1787.0, 'epoch': 0.26}

  3%|▎         | 19/730 [01:44<1:04:06,  5.41s/it]
                                                  
{'train/ce_loss': 14.6875, 'train/diffusion_loss': 0.4907709062099457, 'epoch': 0.26}

  3%|▎         | 19/730 [01:45<1:04:06,  5.41s/it]
                                                  
{'train/learning_rate_real': 2.1590909090909093e-05, 'epoch': 0.26}

  3%|▎         | 19/730 [01:45<1:04:06,  5.41s/it]
                                                  
{'debug/num_tok_total': 2186.0, 'debug/num_tok_loss': 1752.0, 'debug/num_lat_total': 2186.0, 'debug/num_lat_loss': 1752.0, 'epoch': 0.26}

  3%|▎         | 19/730 [01:46<1:04:06,  5.41s/it]
                                                  
{'train/ce_loss': 14.125, 'train/diffusion_loss': 0.542559027671814, 'epoch': 0.26}

  3%|▎         | 19/730 [01:46<1:04:06,  5.41s/it]
                                                  
{'train/learning_rate_real': 2.1590909090909093e-05, 'epoch': 0.26}

  3%|▎         | 19/730 [01:46<1:04:06,  5.41s/it]
                                                  
{'debug/num_tok_total': 2674.0, 'debug/num_tok_loss': 1687.0, 'debug/num_lat_total': 2674.0, 'debug/num_lat_loss': 1687.0, 'epoch': 0.26}

  3%|▎         | 19/730 [01:47<1:04:06,  5.41s/it]
                                                  
{'train/ce_loss': 15.0625, 'train/diffusion_loss': 0.38787195086479187, 'epoch': 0.26}

  3%|▎         | 19/730 [01:47<1:04:06,  5.41s/it]
                                                  
{'train/learning_rate_real': 2.1590909090909093e-05, 'epoch': 0.26}

  3%|▎         | 19/730 [01:47<1:04:06,  5.41s/it]
                                                  
{'debug/num_tok_total': 2032.0, 'debug/num_tok_loss': 1807.0, 'debug/num_lat_total': 2032.0, 'debug/num_lat_loss': 1807.0, 'epoch': 0.26}

  3%|▎         | 19/730 [01:48<1:04:06,  5.41s/it]
                                                  
{'train/ce_loss': 13.875, 'train/diffusion_loss': 0.5763474106788635, 'epoch': 0.26}

  3%|▎         | 19/730 [01:48<1:04:06,  5.41s/it]
                                                  
{'train/learning_rate_real': 2.1590909090909093e-05, 'epoch': 0.26}

  3%|▎         | 19/730 [01:48<1:04:06,  5.41s/it]03/16/2026 06:49:20 - INFO - __main__ - LoRA debug step 20: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

  3%|▎         | 20/730 [01:48<1:03:02,  5.33s/it]
                                                  
{'loss': 5.6093, 'grad_norm': 8.30610466003418, 'learning_rate': 2.1590909090909093e-05, 'epoch': 0.27}

  3%|▎         | 20/730 [01:48<1:03:02,  5.33s/it]
                                                  
{'debug/num_tok_total': 3130.0, 'debug/num_tok_loss': 1812.0, 'debug/num_lat_total': 3130.0, 'debug/num_lat_loss': 1812.0, 'epoch': 0.27}

  3%|▎         | 20/730 [01:50<1:03:02,  5.33s/it]
                                                  
{'train/ce_loss': 14.8125, 'train/diffusion_loss': 0.3510192036628723, 'epoch': 0.27}

  3%|▎         | 20/730 [01:50<1:03:02,  5.33s/it]
                                                  
{'train/learning_rate_real': 2.272727272727273e-05, 'epoch': 0.27}

  3%|▎         | 20/730 [01:50<1:03:02,  5.33s/it]
                                                  
{'debug/num_tok_total': 2193.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 2193.0, 'debug/num_lat_loss': 1768.0, 'epoch': 0.27}

  3%|▎         | 20/730 [01:51<1:03:02,  5.33s/it]
                                                  
{'train/ce_loss': 13.4375, 'train/diffusion_loss': 0.5544577836990356, 'epoch': 0.27}

  3%|▎         | 20/730 [01:51<1:03:02,  5.33s/it]
                                                  
{'train/learning_rate_real': 2.272727272727273e-05, 'epoch': 0.27}

  3%|▎         | 20/730 [01:51<1:03:02,  5.33s/it]
                                                  
{'debug/num_tok_total': 2394.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2394.0, 'debug/num_lat_loss': 1764.0, 'epoch': 0.27}

  3%|▎         | 20/730 [01:52<1:03:02,  5.33s/it]
                                                  
{'train/ce_loss': 13.75, 'train/diffusion_loss': 0.48900994658470154, 'epoch': 0.27}

  3%|▎         | 20/730 [01:52<1:03:02,  5.33s/it]
                                                  
{'train/learning_rate_real': 2.272727272727273e-05, 'epoch': 0.27}

  3%|▎         | 20/730 [01:52<1:03:02,  5.33s/it]
                                                  
{'debug/num_tok_total': 2868.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2868.0, 'debug/num_lat_loss': 1790.0, 'epoch': 0.27}

  3%|▎         | 20/730 [01:53<1:03:02,  5.33s/it]
                                                  
{'train/ce_loss': 14.25, 'train/diffusion_loss': 0.3776327073574066, 'epoch': 0.27}

  3%|▎         | 20/730 [01:53<1:03:02,  5.33s/it]
                                                  
{'train/learning_rate_real': 2.272727272727273e-05, 'epoch': 0.27}

  3%|▎         | 20/730 [01:53<1:03:02,  5.33s/it]
  3%|▎         | 21/730 [01:54<1:03:14,  5.35s/it]
                                                  
{'debug/num_tok_total': 2857.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2857.0, 'debug/num_lat_loss': 1789.0, 'epoch': 0.29}

  3%|▎         | 21/730 [01:55<1:03:14,  5.35s/it]
                                                  
{'train/ce_loss': 13.875, 'train/diffusion_loss': 0.39368781447410583, 'epoch': 0.29}

  3%|▎         | 21/730 [01:55<1:03:14,  5.35s/it]
                                                  
{'train/learning_rate_real': 2.3863636363636365e-05, 'epoch': 0.29}

  3%|▎         | 21/730 [01:55<1:03:14,  5.35s/it]
                                                  
{'debug/num_tok_total': 2192.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 2192.0, 'debug/num_lat_loss': 1769.0, 'epoch': 0.29}

  3%|▎         | 21/730 [01:56<1:03:14,  5.35s/it]
                                                  
{'train/ce_loss': 12.875, 'train/diffusion_loss': 0.5420176982879639, 'epoch': 0.29}

  3%|▎         | 21/730 [01:56<1:03:14,  5.35s/it]
                                                  
{'train/learning_rate_real': 2.3863636363636365e-05, 'epoch': 0.29}

  3%|▎         | 21/730 [01:56<1:03:14,  5.35s/it]
                                                  
{'debug/num_tok_total': 3104.0, 'debug/num_tok_loss': 1798.0, 'debug/num_lat_total': 3104.0, 'debug/num_lat_loss': 1798.0, 'epoch': 0.29}

  3%|▎         | 21/730 [01:57<1:03:14,  5.35s/it]
                                                  
{'train/ce_loss': 14.0625, 'train/diffusion_loss': 0.3666694760322571, 'epoch': 0.29}

  3%|▎         | 21/730 [01:57<1:03:14,  5.35s/it]
                                                  
{'train/learning_rate_real': 2.3863636363636365e-05, 'epoch': 0.29}

  3%|▎         | 21/730 [01:57<1:03:14,  5.35s/it]
                                                  
{'debug/num_tok_total': 2789.0, 'debug/num_tok_loss': 1634.0, 'debug/num_lat_total': 2789.0, 'debug/num_lat_loss': 1634.0, 'epoch': 0.29}

  3%|▎         | 21/730 [01:59<1:03:14,  5.35s/it]
                                                  
{'train/ce_loss': 14.0, 'train/diffusion_loss': 0.37210342288017273, 'epoch': 0.29}

  3%|▎         | 21/730 [01:59<1:03:14,  5.35s/it]
                                                  
{'train/learning_rate_real': 2.3863636363636365e-05, 'epoch': 0.29}

  3%|▎         | 21/730 [01:59<1:03:14,  5.35s/it]
  3%|▎         | 22/730 [01:59<1:03:13,  5.36s/it]
                                                  
{'debug/num_tok_total': 2434.0, 'debug/num_tok_loss': 1586.0, 'debug/num_lat_total': 2434.0, 'debug/num_lat_loss': 1586.0, 'epoch': 0.3}

  3%|▎         | 22/730 [02:00<1:03:13,  5.36s/it]
                                                  
{'train/ce_loss': 13.1875, 'train/diffusion_loss': 0.4085391163825989, 'epoch': 0.3}

  3%|▎         | 22/730 [02:00<1:03:13,  5.36s/it]
                                                  
{'train/learning_rate_real': 2.5e-05, 'epoch': 0.3}

  3%|▎         | 22/730 [02:00<1:03:13,  5.36s/it]
                                                  
{'debug/num_tok_total': 2223.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 2223.0, 'debug/num_lat_loss': 1801.0, 'epoch': 0.3}

  3%|▎         | 22/730 [02:01<1:03:13,  5.36s/it]
                                                  
{'train/ce_loss': 12.4375, 'train/diffusion_loss': 0.5267496705055237, 'epoch': 0.3}

  3%|▎         | 22/730 [02:01<1:03:13,  5.36s/it]
                                                  
{'train/learning_rate_real': 2.5e-05, 'epoch': 0.3}

  3%|▎         | 22/730 [02:01<1:03:13,  5.36s/it]
                                                  
{'debug/num_tok_total': 3025.0, 'debug/num_tok_loss': 1751.0, 'debug/num_lat_total': 3025.0, 'debug/num_lat_loss': 1751.0, 'epoch': 0.3}

  3%|▎         | 22/730 [02:03<1:03:13,  5.36s/it]
                                                  
{'train/ce_loss': 13.5625, 'train/diffusion_loss': 0.3490737974643707, 'epoch': 0.3}

  3%|▎         | 22/730 [02:03<1:03:13,  5.36s/it]
                                                  
{'train/learning_rate_real': 2.5e-05, 'epoch': 0.3}

  3%|▎         | 22/730 [02:03<1:03:13,  5.36s/it]
                                                  
{'debug/num_tok_total': 2684.0, 'debug/num_tok_loss': 1596.0, 'debug/num_lat_total': 2684.0, 'debug/num_lat_loss': 1596.0, 'epoch': 0.3}

  3%|▎         | 22/730 [02:04<1:03:13,  5.36s/it]
                                                  
{'train/ce_loss': 13.3125, 'train/diffusion_loss': 0.3688155710697174, 'epoch': 0.3}

  3%|▎         | 22/730 [02:04<1:03:13,  5.36s/it]
                                                  
{'train/learning_rate_real': 2.5e-05, 'epoch': 0.3}

  3%|▎         | 22/730 [02:04<1:03:13,  5.36s/it]
  3%|▎         | 23/730 [02:05<1:02:54,  5.34s/it]
                                                  
{'debug/num_tok_total': 2224.0, 'debug/num_tok_loss': 1584.0, 'debug/num_lat_total': 2224.0, 'debug/num_lat_loss': 1584.0, 'epoch': 0.32}

  3%|▎         | 23/730 [02:06<1:02:54,  5.34s/it]
                                                  
{'train/ce_loss': 12.5, 'train/diffusion_loss': 0.4947201907634735, 'epoch': 0.32}

  3%|▎         | 23/730 [02:06<1:02:54,  5.34s/it]
                                                  
{'train/learning_rate_real': 2.4999876941239957e-05, 'epoch': 0.32}

  3%|▎         | 23/730 [02:06<1:02:54,  5.34s/it]
                                                  
{'debug/num_tok_total': 2617.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2617.0, 'debug/num_lat_loss': 1770.0, 'epoch': 0.32}

  3%|▎         | 23/730 [02:07<1:02:54,  5.34s/it]
                                                  
{'train/ce_loss': 12.6875, 'train/diffusion_loss': 0.42563173174858093, 'epoch': 0.32}

  3%|▎         | 23/730 [02:07<1:02:54,  5.34s/it]
                                                  
{'train/learning_rate_real': 2.4999876941239957e-05, 'epoch': 0.32}

  3%|▎         | 23/730 [02:07<1:02:54,  5.34s/it]
                                                  
{'debug/num_tok_total': 2644.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2644.0, 'debug/num_lat_loss': 1785.0, 'epoch': 0.32}

  3%|▎         | 23/730 [02:08<1:02:54,  5.34s/it]
                                                  
{'train/ce_loss': 12.5625, 'train/diffusion_loss': 0.4141148328781128, 'epoch': 0.32}

  3%|▎         | 23/730 [02:08<1:02:54,  5.34s/it]
                                                  
{'train/learning_rate_real': 2.4999876941239957e-05, 'epoch': 0.32}

  3%|▎         | 23/730 [02:08<1:02:54,  5.34s/it]
                                                  
{'debug/num_tok_total': 3114.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 3114.0, 'debug/num_lat_loss': 1801.0, 'epoch': 0.32}

  3%|▎         | 23/730 [02:09<1:02:54,  5.34s/it]
                                                  
{'train/ce_loss': 13.3125, 'train/diffusion_loss': 0.3588603138923645, 'epoch': 0.32}

  3%|▎         | 23/730 [02:10<1:02:54,  5.34s/it]
                                                  
{'train/learning_rate_real': 2.4999876941239957e-05, 'epoch': 0.32}

  3%|▎         | 23/730 [02:10<1:02:54,  5.34s/it]
  3%|▎         | 24/730 [02:10<1:03:19,  5.38s/it]
                                                  
{'debug/num_tok_total': 2680.0, 'debug/num_tok_loss': 1803.0, 'debug/num_lat_total': 2680.0, 'debug/num_lat_loss': 1803.0, 'epoch': 0.33}

  3%|▎         | 24/730 [02:11<1:03:19,  5.38s/it]
                                                  
{'train/ce_loss': 12.1875, 'train/diffusion_loss': 0.4522026479244232, 'epoch': 0.33}

  3%|▎         | 24/730 [02:11<1:03:19,  5.38s/it]
                                                  
{'train/learning_rate_real': 2.4999507767382776e-05, 'epoch': 0.33}

  3%|▎         | 24/730 [02:11<1:03:19,  5.38s/it]
                                                  
{'debug/num_tok_total': 2847.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2847.0, 'debug/num_lat_loss': 1776.0, 'epoch': 0.33}

  3%|▎         | 24/730 [02:12<1:03:19,  5.38s/it]
                                                  
{'train/ce_loss': 12.3125, 'train/diffusion_loss': 0.37764832377433777, 'epoch': 0.33}

  3%|▎         | 24/730 [02:13<1:03:19,  5.38s/it]
                                                  
{'train/learning_rate_real': 2.4999507767382776e-05, 'epoch': 0.33}

  3%|▎         | 24/730 [02:13<1:03:19,  5.38s/it]
                                                  
{'debug/num_tok_total': 2842.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2842.0, 'debug/num_lat_loss': 1779.0, 'epoch': 0.33}

  3%|▎         | 24/730 [02:14<1:03:19,  5.38s/it]
                                                  
{'train/ce_loss': 12.3125, 'train/diffusion_loss': 0.4105679392814636, 'epoch': 0.33}

  3%|▎         | 24/730 [02:14<1:03:19,  5.38s/it]
                                                  
{'train/learning_rate_real': 2.4999507767382776e-05, 'epoch': 0.33}

  3%|▎         | 24/730 [02:14<1:03:19,  5.38s/it]
                                                  
{'debug/num_tok_total': 3031.0, 'debug/num_tok_loss': 1759.0, 'debug/num_lat_total': 3031.0, 'debug/num_lat_loss': 1759.0, 'epoch': 0.33}

  3%|▎         | 24/730 [02:15<1:03:19,  5.38s/it]
                                                  
{'train/ce_loss': 12.4375, 'train/diffusion_loss': 0.3406731188297272, 'epoch': 0.33}

  3%|▎         | 24/730 [02:15<1:03:19,  5.38s/it]
                                                  
{'train/learning_rate_real': 2.4999507767382776e-05, 'epoch': 0.33}

  3%|▎         | 24/730 [02:15<1:03:19,  5.38s/it]
  3%|▎         | 25/730 [02:16<1:03:56,  5.44s/it]
                                                  
{'debug/num_tok_total': 2465.0, 'debug/num_tok_loss': 1803.0, 'debug/num_lat_total': 2465.0, 'debug/num_lat_loss': 1803.0, 'epoch': 0.34}

  3%|▎         | 25/730 [02:17<1:03:56,  5.44s/it]
                                                  
{'train/ce_loss': 11.625, 'train/diffusion_loss': 0.4853813946247101, 'epoch': 0.34}

  3%|▎         | 25/730 [02:17<1:03:56,  5.44s/it]
                                                  
{'train/learning_rate_real': 2.4998892485697274e-05, 'epoch': 0.34}

  3%|▎         | 25/730 [02:17<1:03:56,  5.44s/it]
                                                  
{'debug/num_tok_total': 2635.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2635.0, 'debug/num_lat_loss': 1787.0, 'epoch': 0.34}

  3%|▎         | 25/730 [02:18<1:03:56,  5.44s/it]
                                                  
{'train/ce_loss': 11.6875, 'train/diffusion_loss': 0.44053584337234497, 'epoch': 0.34}

  3%|▎         | 25/730 [02:18<1:03:56,  5.44s/it]
                                                  
{'train/learning_rate_real': 2.4998892485697274e-05, 'epoch': 0.34}

  3%|▎         | 25/730 [02:18<1:03:56,  5.44s/it]
                                                  
{'debug/num_tok_total': 3020.0, 'debug/num_tok_loss': 1753.0, 'debug/num_lat_total': 3020.0, 'debug/num_lat_loss': 1753.0, 'epoch': 0.34}

  3%|▎         | 25/730 [02:19<1:03:56,  5.44s/it]
                                                  
{'train/ce_loss': 12.125, 'train/diffusion_loss': 0.3806762397289276, 'epoch': 0.34}

  3%|▎         | 25/730 [02:19<1:03:56,  5.44s/it]
                                                  
{'train/learning_rate_real': 2.4998892485697274e-05, 'epoch': 0.34}

  3%|▎         | 25/730 [02:19<1:03:56,  5.44s/it]
                                                  
{'debug/num_tok_total': 2919.0, 'debug/num_tok_loss': 1819.0, 'debug/num_lat_total': 2919.0, 'debug/num_lat_loss': 1819.0, 'epoch': 0.34}

  3%|▎         | 25/730 [02:20<1:03:56,  5.44s/it]
                                                  
{'train/ce_loss': 11.8125, 'train/diffusion_loss': 0.4265860915184021, 'epoch': 0.34}

  3%|▎         | 25/730 [02:20<1:03:56,  5.44s/it]
                                                  
{'train/learning_rate_real': 2.4998892485697274e-05, 'epoch': 0.34}

  3%|▎         | 25/730 [02:20<1:03:56,  5.44s/it]
  4%|▎         | 26/730 [02:21<1:03:34,  5.42s/it]
                                                  
{'debug/num_tok_total': 2678.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2678.0, 'debug/num_lat_loss': 1802.0, 'epoch': 0.36}

  4%|▎         | 26/730 [02:22<1:03:34,  5.42s/it]
                                                  
{'train/ce_loss': 11.375, 'train/diffusion_loss': 0.4423990845680237, 'epoch': 0.36}

  4%|▎         | 26/730 [02:22<1:03:34,  5.42s/it]
                                                  
{'train/learning_rate_real': 2.4998031108297975e-05, 'epoch': 0.36}

  4%|▎         | 26/730 [02:22<1:03:34,  5.42s/it]
                                                  
{'debug/num_tok_total': 2840.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2840.0, 'debug/num_lat_loss': 1771.0, 'epoch': 0.36}

  4%|▎         | 26/730 [02:23<1:03:34,  5.42s/it]
                                                  
{'train/ce_loss': 11.5, 'train/diffusion_loss': 0.3932268023490906, 'epoch': 0.36}

  4%|▎         | 26/730 [02:23<1:03:34,  5.42s/it]
                                                  
{'train/learning_rate_real': 2.4998031108297975e-05, 'epoch': 0.36}

  4%|▎         | 26/730 [02:23<1:03:34,  5.42s/it]
                                                  
{'debug/num_tok_total': 3091.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 3091.0, 'debug/num_lat_loss': 1794.0, 'epoch': 0.36}

  4%|▎         | 26/730 [02:25<1:03:34,  5.42s/it]
                                                  
{'train/ce_loss': 11.625, 'train/diffusion_loss': 0.3441810607910156, 'epoch': 0.36}

  4%|▎         | 26/730 [02:25<1:03:34,  5.42s/it]
                                                  
{'train/learning_rate_real': 2.4998031108297975e-05, 'epoch': 0.36}

  4%|▎         | 26/730 [02:25<1:03:34,  5.42s/it]
                                                  
{'debug/num_tok_total': 2841.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2841.0, 'debug/num_lat_loss': 1774.0, 'epoch': 0.36}

  4%|▎         | 26/730 [02:26<1:03:34,  5.42s/it]
                                                  
{'train/ce_loss': 11.5, 'train/diffusion_loss': 0.4140407145023346, 'epoch': 0.36}

  4%|▎         | 26/730 [02:26<1:03:34,  5.42s/it]
                                                  
{'train/learning_rate_real': 2.4998031108297975e-05, 'epoch': 0.36}

  4%|▎         | 26/730 [02:26<1:03:34,  5.42s/it]
  4%|▎         | 27/730 [02:26<1:03:37,  5.43s/it]
                                                  
{'debug/num_tok_total': 2690.0, 'debug/num_tok_loss': 1805.0, 'debug/num_lat_total': 2690.0, 'debug/num_lat_loss': 1805.0, 'epoch': 0.37}

  4%|▎         | 27/730 [02:28<1:03:37,  5.43s/it]
                                                  
{'train/ce_loss': 11.125, 'train/diffusion_loss': 0.4142424166202545, 'epoch': 0.37}

  4%|▎         | 27/730 [02:28<1:03:37,  5.43s/it]
                                                  
{'train/learning_rate_real': 2.4996923652144887e-05, 'epoch': 0.37}

  4%|▎         | 27/730 [02:28<1:03:37,  5.43s/it]
                                                  
{'debug/num_tok_total': 2457.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2457.0, 'debug/num_lat_loss': 1794.0, 'epoch': 0.37}

  4%|▎         | 27/730 [02:29<1:03:37,  5.43s/it]
                                                  
{'train/ce_loss': 11.125, 'train/diffusion_loss': 0.4762253761291504, 'epoch': 0.37}

  4%|▎         | 27/730 [02:29<1:03:37,  5.43s/it]
                                                  
{'train/learning_rate_real': 2.4996923652144887e-05, 'epoch': 0.37}

  4%|▎         | 27/730 [02:29<1:03:37,  5.43s/it]
                                                  
{'debug/num_tok_total': 2393.0, 'debug/num_tok_loss': 1754.0, 'debug/num_lat_total': 2393.0, 'debug/num_lat_loss': 1754.0, 'epoch': 0.37}

  4%|▎         | 27/730 [02:30<1:03:37,  5.43s/it]
                                                  
{'train/ce_loss': 11.0, 'train/diffusion_loss': 0.46639570593833923, 'epoch': 0.37}

  4%|▎         | 27/730 [02:30<1:03:37,  5.43s/it]
                                                  
{'train/learning_rate_real': 2.4996923652144887e-05, 'epoch': 0.37}

  4%|▎         | 27/730 [02:30<1:03:37,  5.43s/it]
                                                  
{'debug/num_tok_total': 2277.0, 'debug/num_tok_loss': 1621.0, 'debug/num_lat_total': 2277.0, 'debug/num_lat_loss': 1621.0, 'epoch': 0.37}

  4%|▎         | 27/730 [02:31<1:03:37,  5.43s/it]
                                                  
{'train/ce_loss': 11.0625, 'train/diffusion_loss': 0.4759414494037628, 'epoch': 0.37}

  4%|▎         | 27/730 [02:31<1:03:37,  5.43s/it]
                                                  
{'train/learning_rate_real': 2.4996923652144887e-05, 'epoch': 0.37}

  4%|▎         | 27/730 [02:31<1:03:37,  5.43s/it]
  4%|▍         | 28/730 [02:31<1:02:23,  5.33s/it]
                                                  
{'debug/num_tok_total': 2847.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2847.0, 'debug/num_lat_loss': 1771.0, 'epoch': 0.38}

  4%|▍         | 28/730 [02:33<1:02:23,  5.33s/it]
                                                  
{'train/ce_loss': 11.0, 'train/diffusion_loss': 0.40890657901763916, 'epoch': 0.38}

  4%|▍         | 28/730 [02:33<1:02:23,  5.33s/it]
                                                  
{'train/learning_rate_real': 2.4995570139043158e-05, 'epoch': 0.38}

  4%|▍         | 28/730 [02:33<1:02:23,  5.33s/it]
                                                  
{'debug/num_tok_total': 2790.0, 'debug/num_tok_loss': 1735.0, 'debug/num_lat_total': 2790.0, 'debug/num_lat_loss': 1735.0, 'epoch': 0.38}

  4%|▍         | 28/730 [02:34<1:02:23,  5.33s/it]
                                                  
{'train/ce_loss': 11.0, 'train/diffusion_loss': 0.3836289644241333, 'epoch': 0.38}

  4%|▍         | 28/730 [02:34<1:02:23,  5.33s/it]
                                                  
{'train/learning_rate_real': 2.4995570139043158e-05, 'epoch': 0.38}

  4%|▍         | 28/730 [02:34<1:02:23,  5.33s/it]
                                                  
{'debug/num_tok_total': 3062.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 3062.0, 'debug/num_lat_loss': 1773.0, 'epoch': 0.38}

  4%|▍         | 28/730 [02:35<1:02:23,  5.33s/it]
                                                  
{'train/ce_loss': 11.0625, 'train/diffusion_loss': 0.32098737359046936, 'epoch': 0.38}

  4%|▍         | 28/730 [02:35<1:02:23,  5.33s/it]
                                                  
{'train/learning_rate_real': 2.4995570139043158e-05, 'epoch': 0.38}

  4%|▍         | 28/730 [02:35<1:02:23,  5.33s/it]
                                                  
{'debug/num_tok_total': 3061.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 3061.0, 'debug/num_lat_loss': 1781.0, 'epoch': 0.38}

  4%|▍         | 28/730 [02:37<1:02:23,  5.33s/it]
                                                  
{'train/ce_loss': 11.0, 'train/diffusion_loss': 0.35937121510505676, 'epoch': 0.38}

  4%|▍         | 28/730 [02:37<1:02:23,  5.33s/it]
                                                  
{'train/learning_rate_real': 2.4995570139043158e-05, 'epoch': 0.38}

  4%|▍         | 28/730 [02:37<1:02:23,  5.33s/it]
  4%|▍         | 29/730 [02:37<1:03:08,  5.40s/it]
                                                  
{'debug/num_tok_total': 2669.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 2669.0, 'debug/num_lat_loss': 1797.0, 'epoch': 0.4}

  4%|▍         | 29/730 [02:38<1:03:08,  5.40s/it]
                                                  
{'train/ce_loss': 10.75, 'train/diffusion_loss': 0.4344680905342102, 'epoch': 0.4}

  4%|▍         | 29/730 [02:38<1:03:08,  5.40s/it]
                                                  
{'train/learning_rate_real': 2.499397059564265e-05, 'epoch': 0.4}

  4%|▍         | 29/730 [02:38<1:03:08,  5.40s/it]
                                                  
{'debug/num_tok_total': 3022.0, 'debug/num_tok_loss': 1756.0, 'debug/num_lat_total': 3022.0, 'debug/num_lat_loss': 1756.0, 'epoch': 0.4}

  4%|▍         | 29/730 [02:39<1:03:08,  5.40s/it]
                                                  
{'train/ce_loss': 10.8125, 'train/diffusion_loss': 0.34724363684654236, 'epoch': 0.4}

  4%|▍         | 29/730 [02:40<1:03:08,  5.40s/it]
                                                  
{'train/learning_rate_real': 2.499397059564265e-05, 'epoch': 0.4}

  4%|▍         | 29/730 [02:40<1:03:08,  5.40s/it]
                                                  
{'debug/num_tok_total': 2866.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2866.0, 'debug/num_lat_loss': 1795.0, 'epoch': 0.4}

  4%|▍         | 29/730 [02:41<1:03:08,  5.40s/it]
                                                  
{'train/ce_loss': 10.75, 'train/diffusion_loss': 0.39015740156173706, 'epoch': 0.4}

  4%|▍         | 29/730 [02:41<1:03:08,  5.40s/it]
                                                  
{'train/learning_rate_real': 2.499397059564265e-05, 'epoch': 0.4}

  4%|▍         | 29/730 [02:41<1:03:08,  5.40s/it]
                                                  
{'debug/num_tok_total': 2869.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2869.0, 'debug/num_lat_loss': 1794.0, 'epoch': 0.4}

  4%|▍         | 29/730 [02:42<1:03:08,  5.40s/it]
                                                  
{'train/ce_loss': 10.75, 'train/diffusion_loss': 0.37296244502067566, 'epoch': 0.4}

  4%|▍         | 29/730 [02:42<1:03:08,  5.40s/it]
                                                  
{'train/learning_rate_real': 2.499397059564265e-05, 'epoch': 0.4}

  4%|▍         | 29/730 [02:42<1:03:08,  5.40s/it]03/16/2026 06:50:14 - INFO - __main__ - LoRA debug step 30: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

  4%|▍         | 30/730 [02:43<1:03:11,  5.42s/it]
                                                  
{'loss': 4.2717, 'grad_norm': 3.1832733154296875, 'learning_rate': 2.499397059564265e-05, 'epoch': 0.41}

  4%|▍         | 30/730 [02:43<1:03:11,  5.42s/it]
                                                  
{'debug/num_tok_total': 2707.0, 'debug/num_tok_loss': 1707.0, 'debug/num_lat_total': 2707.0, 'debug/num_lat_loss': 1707.0, 'epoch': 0.41}

  4%|▍         | 30/730 [02:44<1:03:11,  5.42s/it]
                                                  
{'train/ce_loss': 10.625, 'train/diffusion_loss': 0.38847485184669495, 'epoch': 0.41}

  4%|▍         | 30/730 [02:44<1:03:11,  5.42s/it]
                                                  
{'train/learning_rate_real': 2.499212505343742e-05, 'epoch': 0.41}

  4%|▍         | 30/730 [02:44<1:03:11,  5.42s/it]
                                                  
{'debug/num_tok_total': 2631.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2631.0, 'debug/num_lat_loss': 1776.0, 'epoch': 0.41}

  4%|▍         | 30/730 [02:45<1:03:11,  5.42s/it]
                                                  
{'train/ce_loss': 10.5625, 'train/diffusion_loss': 0.42114052176475525, 'epoch': 0.41}

  4%|▍         | 30/730 [02:45<1:03:11,  5.42s/it]
                                                  
{'train/learning_rate_real': 2.499212505343742e-05, 'epoch': 0.41}

  4%|▍         | 30/730 [02:45<1:03:11,  5.42s/it]
                                                  
{'debug/num_tok_total': 2236.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2236.0, 'debug/num_lat_loss': 1796.0, 'epoch': 0.41}

  4%|▍         | 30/730 [02:46<1:03:11,  5.42s/it]
                                                  
{'train/ce_loss': 10.375, 'train/diffusion_loss': 0.5161031484603882, 'epoch': 0.41}

  4%|▍         | 30/730 [02:46<1:03:11,  5.42s/it]
                                                  
{'train/learning_rate_real': 2.499212505343742e-05, 'epoch': 0.41}

  4%|▍         | 30/730 [02:46<1:03:11,  5.42s/it]
                                                  
{'debug/num_tok_total': 2354.0, 'debug/num_tok_loss': 1548.0, 'debug/num_lat_total': 2354.0, 'debug/num_lat_loss': 1548.0, 'epoch': 0.41}

  4%|▍         | 30/730 [02:47<1:03:11,  5.42s/it]
                                                  
{'train/ce_loss': 10.5625, 'train/diffusion_loss': 0.404449462890625, 'epoch': 0.41}

  4%|▍         | 30/730 [02:47<1:03:11,  5.42s/it]
                                                  
{'train/learning_rate_real': 2.499212505343742e-05, 'epoch': 0.41}

  4%|▍         | 30/730 [02:47<1:03:11,  5.42s/it]
  4%|▍         | 31/730 [02:48<1:02:23,  5.36s/it]
                                                  
{'debug/num_tok_total': 2417.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2417.0, 'debug/num_lat_loss': 1786.0, 'epoch': 0.42}

  4%|▍         | 31/730 [02:49<1:02:23,  5.36s/it]
                                                  
{'train/ce_loss': 10.25, 'train/diffusion_loss': 0.48773378133773804, 'epoch': 0.42}

  4%|▍         | 31/730 [02:49<1:02:23,  5.36s/it]
                                                  
{'train/learning_rate_real': 2.4990033548765084e-05, 'epoch': 0.42}

  4%|▍         | 31/730 [02:49<1:02:23,  5.36s/it]
                                                  
{'debug/num_tok_total': 2422.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2422.0, 'debug/num_lat_loss': 1780.0, 'epoch': 0.42}

  4%|▍         | 31/730 [02:50<1:02:23,  5.36s/it]
                                                  
{'train/ce_loss': 10.3125, 'train/diffusion_loss': 0.4736463129520416, 'epoch': 0.42}

  4%|▍         | 31/730 [02:50<1:02:23,  5.36s/it]
                                                  
{'train/learning_rate_real': 2.4990033548765084e-05, 'epoch': 0.42}

  4%|▍         | 31/730 [02:50<1:02:23,  5.36s/it]
                                                  
{'debug/num_tok_total': 2889.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2889.0, 'debug/num_lat_loss': 1795.0, 'epoch': 0.42}

  4%|▍         | 31/730 [02:51<1:02:23,  5.36s/it]
                                                  
{'train/ce_loss': 10.4375, 'train/diffusion_loss': 0.347330778837204, 'epoch': 0.42}

  4%|▍         | 31/730 [02:51<1:02:23,  5.36s/it]
                                                  
{'train/learning_rate_real': 2.4990033548765084e-05, 'epoch': 0.42}

  4%|▍         | 31/730 [02:51<1:02:23,  5.36s/it]
                                                  
{'debug/num_tok_total': 2391.0, 'debug/num_tok_loss': 1755.0, 'debug/num_lat_total': 2391.0, 'debug/num_lat_loss': 1755.0, 'epoch': 0.42}

  4%|▍         | 31/730 [02:52<1:02:23,  5.36s/it]
                                                  
{'train/ce_loss': 10.25, 'train/diffusion_loss': 0.48943084478378296, 'epoch': 0.42}

  4%|▍         | 31/730 [02:52<1:02:23,  5.36s/it]
                                                  
{'train/learning_rate_real': 2.4990033548765084e-05, 'epoch': 0.42}

  4%|▍         | 31/730 [02:52<1:02:23,  5.36s/it]
  4%|▍         | 32/730 [02:53<1:01:35,  5.29s/it]
                                                  
{'debug/num_tok_total': 2192.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 2192.0, 'debug/num_lat_loss': 1767.0, 'epoch': 0.44}

  4%|▍         | 32/730 [02:54<1:01:35,  5.29s/it]
                                                  
{'train/ce_loss': 10.0, 'train/diffusion_loss': 0.5378103852272034, 'epoch': 0.44}

  4%|▍         | 32/730 [02:54<1:01:35,  5.29s/it]
                                                  
{'train/learning_rate_real': 2.4987696122806127e-05, 'epoch': 0.44}

  4%|▍         | 32/730 [02:54<1:01:35,  5.29s/it]
                                                  
{'debug/num_tok_total': 2235.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2235.0, 'debug/num_lat_loss': 1793.0, 'epoch': 0.44}

  4%|▍         | 32/730 [02:55<1:01:35,  5.29s/it]
                                                  
{'train/ce_loss': 10.0, 'train/diffusion_loss': 0.546953022480011, 'epoch': 0.44}

  4%|▍         | 32/730 [02:55<1:01:35,  5.29s/it]
                                                  
{'train/learning_rate_real': 2.4987696122806127e-05, 'epoch': 0.44}

  4%|▍         | 32/730 [02:55<1:01:35,  5.29s/it]
                                                  
{'debug/num_tok_total': 3086.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 3086.0, 'debug/num_lat_loss': 1784.0, 'epoch': 0.44}

  4%|▍         | 32/730 [02:56<1:01:35,  5.29s/it]
                                                  
{'train/ce_loss': 10.25, 'train/diffusion_loss': 0.3321462869644165, 'epoch': 0.44}

  4%|▍         | 32/730 [02:56<1:01:35,  5.29s/it]
                                                  
{'train/learning_rate_real': 2.4987696122806127e-05, 'epoch': 0.44}

  4%|▍         | 32/730 [02:56<1:01:35,  5.29s/it]
                                                  
{'debug/num_tok_total': 2842.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2842.0, 'debug/num_lat_loss': 1780.0, 'epoch': 0.44}

  4%|▍         | 32/730 [02:58<1:01:35,  5.29s/it]
                                                  
{'train/ce_loss': 10.1875, 'train/diffusion_loss': 0.39034295082092285, 'epoch': 0.44}

  4%|▍         | 32/730 [02:58<1:01:35,  5.29s/it]
                                                  
{'train/learning_rate_real': 2.4987696122806127e-05, 'epoch': 0.44}

  4%|▍         | 32/730 [02:58<1:01:35,  5.29s/it]
  5%|▍         | 33/730 [02:58<1:01:06,  5.26s/it]
                                                  
{'debug/num_tok_total': 2399.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 2399.0, 'debug/num_lat_loss': 1769.0, 'epoch': 0.45}

  5%|▍         | 33/730 [02:59<1:01:06,  5.26s/it]
                                                  
{'train/ce_loss': 9.9375, 'train/diffusion_loss': 0.4693682789802551, 'epoch': 0.45}

  5%|▍         | 33/730 [02:59<1:01:06,  5.26s/it]
                                                  
{'train/learning_rate_real': 2.498511282158305e-05, 'epoch': 0.45}

  5%|▍         | 33/730 [02:59<1:01:06,  5.26s/it]
                                                  
{'debug/num_tok_total': 2620.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 2620.0, 'debug/num_lat_loss': 1768.0, 'epoch': 0.45}

  5%|▍         | 33/730 [03:00<1:01:06,  5.26s/it]
                                                  
{'train/ce_loss': 9.9375, 'train/diffusion_loss': 0.39866846799850464, 'epoch': 0.45}

  5%|▍         | 33/730 [03:00<1:01:06,  5.26s/it]
                                                  
{'train/learning_rate_real': 2.498511282158305e-05, 'epoch': 0.45}

  5%|▍         | 33/730 [03:00<1:01:06,  5.26s/it]
                                                  
{'debug/num_tok_total': 2706.0, 'debug/num_tok_loss': 1599.0, 'debug/num_lat_total': 2706.0, 'debug/num_lat_loss': 1599.0, 'epoch': 0.45}

  5%|▍         | 33/730 [03:02<1:01:06,  5.26s/it]
                                                  
{'train/ce_loss': 10.0, 'train/diffusion_loss': 0.3576337397098541, 'epoch': 0.45}

  5%|▍         | 33/730 [03:02<1:01:06,  5.26s/it]
                                                  
{'train/learning_rate_real': 2.498511282158305e-05, 'epoch': 0.45}

  5%|▍         | 33/730 [03:02<1:01:06,  5.26s/it]
                                                  
{'debug/num_tok_total': 1977.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 1977.0, 'debug/num_lat_loss': 1770.0, 'epoch': 0.45}

  5%|▍         | 33/730 [03:03<1:01:06,  5.26s/it]
                                                  
{'train/ce_loss': 9.75, 'train/diffusion_loss': 0.5810636878013611, 'epoch': 0.45}

  5%|▍         | 33/730 [03:03<1:01:06,  5.26s/it]
                                                  
{'train/learning_rate_real': 2.498511282158305e-05, 'epoch': 0.45}

  5%|▍         | 33/730 [03:03<1:01:06,  5.26s/it]
  5%|▍         | 34/730 [03:03<1:00:22,  5.21s/it]
                                                  
{'debug/num_tok_total': 2402.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2402.0, 'debug/num_lat_loss': 1764.0, 'epoch': 0.47}

  5%|▍         | 34/730 [03:04<1:00:22,  5.21s/it]
                                                  
{'train/ce_loss': 9.6875, 'train/diffusion_loss': 0.4539051651954651, 'epoch': 0.47}

  5%|▍         | 34/730 [03:04<1:00:22,  5.21s/it]
                                                  
{'train/learning_rate_real': 2.4982283695959525e-05, 'epoch': 0.47}

  5%|▍         | 34/730 [03:04<1:00:22,  5.21s/it]
                                                  
{'debug/num_tok_total': 2834.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2834.0, 'debug/num_lat_loss': 1763.0, 'epoch': 0.47}

  5%|▍         | 34/730 [03:06<1:00:22,  5.21s/it]
                                                  
{'train/ce_loss': 9.6875, 'train/diffusion_loss': 0.3914901614189148, 'epoch': 0.47}

  5%|▍         | 34/730 [03:06<1:00:22,  5.21s/it]
                                                  
{'train/learning_rate_real': 2.4982283695959525e-05, 'epoch': 0.47}

  5%|▍         | 34/730 [03:06<1:00:22,  5.21s/it]
                                                  
{'debug/num_tok_total': 3059.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 3059.0, 'debug/num_lat_loss': 1775.0, 'epoch': 0.47}

  5%|▍         | 34/730 [03:07<1:00:22,  5.21s/it]
                                                  
{'train/ce_loss': 9.8125, 'train/diffusion_loss': 0.3305813670158386, 'epoch': 0.47}

  5%|▍         | 34/730 [03:07<1:00:22,  5.21s/it]
                                                  
{'train/learning_rate_real': 2.4982283695959525e-05, 'epoch': 0.47}

  5%|▍         | 34/730 [03:07<1:00:22,  5.21s/it]
                                                  
{'debug/num_tok_total': 2895.0, 'debug/num_tok_loss': 1798.0, 'debug/num_lat_total': 2895.0, 'debug/num_lat_loss': 1798.0, 'epoch': 0.47}

  5%|▍         | 34/730 [03:08<1:00:22,  5.21s/it]
                                                  
{'train/ce_loss': 9.75, 'train/diffusion_loss': 0.3716033399105072, 'epoch': 0.47}

  5%|▍         | 34/730 [03:08<1:00:22,  5.21s/it]
                                                  
{'train/learning_rate_real': 2.4982283695959525e-05, 'epoch': 0.47}

  5%|▍         | 34/730 [03:08<1:00:22,  5.21s/it]
  5%|▍         | 35/730 [03:09<1:01:18,  5.29s/it]
                                                  
{'debug/num_tok_total': 3350.0, 'debug/num_tok_loss': 1817.0, 'debug/num_lat_total': 3350.0, 'debug/num_lat_loss': 1817.0, 'epoch': 0.48}

  5%|▍         | 35/730 [03:10<1:01:18,  5.29s/it]
                                                  
{'train/ce_loss': 9.5625, 'train/diffusion_loss': 0.27160805463790894, 'epoch': 0.48}

  5%|▍         | 35/730 [03:10<1:01:18,  5.29s/it]
                                                  
{'train/learning_rate_real': 2.4979208801639335e-05, 'epoch': 0.48}

  5%|▍         | 35/730 [03:10<1:01:18,  5.29s/it]
                                                  
{'debug/num_tok_total': 2438.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2438.0, 'debug/num_lat_loss': 1785.0, 'epoch': 0.48}

  5%|▍         | 35/730 [03:11<1:01:18,  5.29s/it]
                                                  
{'train/ce_loss': 9.4375, 'train/diffusion_loss': 0.48116302490234375, 'epoch': 0.48}

  5%|▍         | 35/730 [03:11<1:01:18,  5.29s/it]
                                                  
{'train/learning_rate_real': 2.4979208801639335e-05, 'epoch': 0.48}

  5%|▍         | 35/730 [03:11<1:01:18,  5.29s/it]
                                                  
{'debug/num_tok_total': 2077.0, 'debug/num_tok_loss': 1652.0, 'debug/num_lat_total': 2077.0, 'debug/num_lat_loss': 1652.0, 'epoch': 0.48}

  5%|▍         | 35/730 [03:12<1:01:18,  5.29s/it]
                                                  
{'train/ce_loss': 9.375, 'train/diffusion_loss': 0.5048086643218994, 'epoch': 0.48}

  5%|▍         | 35/730 [03:12<1:01:18,  5.29s/it]
                                                  
{'train/learning_rate_real': 2.4979208801639335e-05, 'epoch': 0.48}

  5%|▍         | 35/730 [03:12<1:01:18,  5.29s/it]
                                                  
{'debug/num_tok_total': 2331.0, 'debug/num_tok_loss': 1626.0, 'debug/num_lat_total': 2331.0, 'debug/num_lat_loss': 1626.0, 'epoch': 0.48}

  5%|▍         | 35/730 [03:14<1:01:18,  5.29s/it]
                                                  
{'train/ce_loss': 9.5, 'train/diffusion_loss': 0.43859153985977173, 'epoch': 0.48}

  5%|▍         | 35/730 [03:14<1:01:18,  5.29s/it]
                                                  
{'train/learning_rate_real': 2.4979208801639335e-05, 'epoch': 0.48}

  5%|▍         | 35/730 [03:14<1:01:18,  5.29s/it]
  5%|▍         | 36/730 [03:14<1:01:25,  5.31s/it]
                                                  
{'debug/num_tok_total': 2455.0, 'debug/num_tok_loss': 1804.0, 'debug/num_lat_total': 2455.0, 'debug/num_lat_loss': 1804.0, 'epoch': 0.49}

  5%|▍         | 36/730 [03:15<1:01:25,  5.31s/it]
                                                  
{'train/ce_loss': 9.25, 'train/diffusion_loss': 0.4600470960140228, 'epoch': 0.49}

  5%|▍         | 36/730 [03:15<1:01:25,  5.31s/it]
                                                  
{'train/learning_rate_real': 2.497588819916531e-05, 'epoch': 0.49}

  5%|▍         | 36/730 [03:15<1:01:25,  5.31s/it]
                                                  
{'debug/num_tok_total': 2444.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2444.0, 'debug/num_lat_loss': 1783.0, 'epoch': 0.49}

  5%|▍         | 36/730 [03:16<1:01:25,  5.31s/it]
                                                  
{'train/ce_loss': 9.125, 'train/diffusion_loss': 0.5146046876907349, 'epoch': 0.49}

  5%|▍         | 36/730 [03:16<1:01:25,  5.31s/it]
                                                  
{'train/learning_rate_real': 2.497588819916531e-05, 'epoch': 0.49}

  5%|▍         | 36/730 [03:16<1:01:25,  5.31s/it]
                                                  
{'debug/num_tok_total': 2006.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2006.0, 'debug/num_lat_loss': 1794.0, 'epoch': 0.49}

  5%|▍         | 36/730 [03:17<1:01:25,  5.31s/it]
                                                  
{'train/ce_loss': 9.0, 'train/diffusion_loss': 0.5873913168907166, 'epoch': 0.49}

  5%|▍         | 36/730 [03:17<1:01:25,  5.31s/it]
                                                  
{'train/learning_rate_real': 2.497588819916531e-05, 'epoch': 0.49}

  5%|▍         | 36/730 [03:17<1:01:25,  5.31s/it]
                                                  
{'debug/num_tok_total': 2867.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2867.0, 'debug/num_lat_loss': 1790.0, 'epoch': 0.49}

  5%|▍         | 36/730 [03:19<1:01:25,  5.31s/it]
                                                  
{'train/ce_loss': 9.25, 'train/diffusion_loss': 0.36510205268859863, 'epoch': 0.49}

  5%|▍         | 36/730 [03:19<1:01:25,  5.31s/it]
                                                  
{'train/learning_rate_real': 2.497588819916531e-05, 'epoch': 0.49}

  5%|▍         | 36/730 [03:19<1:01:25,  5.31s/it]
  5%|▌         | 37/730 [03:19<1:00:56,  5.28s/it]
                                                  
{'debug/num_tok_total': 2899.0, 'debug/num_tok_loss': 1805.0, 'debug/num_lat_total': 2899.0, 'debug/num_lat_loss': 1805.0, 'epoch': 0.51}

  5%|▌         | 37/730 [03:20<1:00:56,  5.28s/it]
                                                  
{'train/ce_loss': 9.0, 'train/diffusion_loss': 0.3962245285511017, 'epoch': 0.51}

  5%|▌         | 37/730 [03:20<1:00:56,  5.28s/it]
                                                  
{'train/learning_rate_real': 2.4972321953918126e-05, 'epoch': 0.51}

  5%|▌         | 37/730 [03:20<1:00:56,  5.28s/it]
                                                  
{'debug/num_tok_total': 2637.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2637.0, 'debug/num_lat_loss': 1787.0, 'epoch': 0.51}

  5%|▌         | 37/730 [03:22<1:00:56,  5.28s/it]
                                                  
{'train/ce_loss': 8.9375, 'train/diffusion_loss': 0.41625067591667175, 'epoch': 0.51}

  5%|▌         | 37/730 [03:22<1:00:56,  5.28s/it]
                                                  
{'train/learning_rate_real': 2.4972321953918126e-05, 'epoch': 0.51}

  5%|▌         | 37/730 [03:22<1:00:56,  5.28s/it]
                                                  
{'debug/num_tok_total': 2415.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2415.0, 'debug/num_lat_loss': 1764.0, 'epoch': 0.51}

  5%|▌         | 37/730 [03:23<1:00:56,  5.28s/it]
                                                  
{'train/ce_loss': 8.875, 'train/diffusion_loss': 0.45504698157310486, 'epoch': 0.51}

  5%|▌         | 37/730 [03:23<1:00:56,  5.28s/it]
                                                  
{'train/learning_rate_real': 2.4972321953918126e-05, 'epoch': 0.51}

  5%|▌         | 37/730 [03:23<1:00:56,  5.28s/it]
                                                  
{'debug/num_tok_total': 2875.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2875.0, 'debug/num_lat_loss': 1796.0, 'epoch': 0.51}

  5%|▌         | 37/730 [03:24<1:00:56,  5.28s/it]
                                                  
{'train/ce_loss': 8.9375, 'train/diffusion_loss': 0.3914295434951782, 'epoch': 0.51}

  5%|▌         | 37/730 [03:24<1:00:56,  5.28s/it]
                                                  
{'train/learning_rate_real': 2.4972321953918126e-05, 'epoch': 0.51}

  5%|▌         | 37/730 [03:24<1:00:56,  5.28s/it]
  5%|▌         | 38/730 [03:25<1:01:42,  5.35s/it]
                                                  
{'debug/num_tok_total': 2664.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2664.0, 'debug/num_lat_loss': 1788.0, 'epoch': 0.52}

  5%|▌         | 38/730 [03:26<1:01:42,  5.35s/it]
                                                  
{'train/ce_loss': 8.625, 'train/diffusion_loss': 0.4318960905075073, 'epoch': 0.52}

  5%|▌         | 38/730 [03:26<1:01:42,  5.35s/it]
                                                  
{'train/learning_rate_real': 2.496851013611502e-05, 'epoch': 0.52}

  5%|▌         | 38/730 [03:26<1:01:42,  5.35s/it]
                                                  
{'debug/num_tok_total': 2958.0, 'debug/num_tok_loss': 1677.0, 'debug/num_lat_total': 2958.0, 'debug/num_lat_loss': 1677.0, 'epoch': 0.52}

  5%|▌         | 38/730 [03:27<1:01:42,  5.35s/it]
                                                  
{'train/ce_loss': 8.8125, 'train/diffusion_loss': 0.31698915362358093, 'epoch': 0.52}

  5%|▌         | 38/730 [03:27<1:01:42,  5.35s/it]
                                                  
{'train/learning_rate_real': 2.496851013611502e-05, 'epoch': 0.52}

  5%|▌         | 38/730 [03:27<1:01:42,  5.35s/it]
                                                  
{'debug/num_tok_total': 2453.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2453.0, 'debug/num_lat_loss': 1799.0, 'epoch': 0.52}

  5%|▌         | 38/730 [03:28<1:01:42,  5.35s/it]
                                                  
{'train/ce_loss': 8.5625, 'train/diffusion_loss': 0.481392502784729, 'epoch': 0.52}

  5%|▌         | 38/730 [03:28<1:01:42,  5.35s/it]
                                                  
{'train/learning_rate_real': 2.496851013611502e-05, 'epoch': 0.52}

  5%|▌         | 38/730 [03:28<1:01:42,  5.35s/it]
                                                  
{'debug/num_tok_total': 2440.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2440.0, 'debug/num_lat_loss': 1785.0, 'epoch': 0.52}

  5%|▌         | 38/730 [03:30<1:01:42,  5.35s/it]
                                                  
{'train/ce_loss': 8.6875, 'train/diffusion_loss': 0.46959739923477173, 'epoch': 0.52}

  5%|▌         | 38/730 [03:30<1:01:42,  5.35s/it]
                                                  
{'train/learning_rate_real': 2.496851013611502e-05, 'epoch': 0.52}

  5%|▌         | 38/730 [03:30<1:01:42,  5.35s/it]
  5%|▌         | 39/730 [03:30<1:01:52,  5.37s/it]
                                                  
{'debug/num_tok_total': 3080.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 3080.0, 'debug/num_lat_loss': 1783.0, 'epoch': 0.53}

  5%|▌         | 39/730 [03:31<1:01:52,  5.37s/it]
                                                  
{'train/ce_loss': 8.5, 'train/diffusion_loss': 0.35770803689956665, 'epoch': 0.53}

  5%|▌         | 39/730 [03:31<1:01:52,  5.37s/it]
                                                  
{'train/learning_rate_real': 2.4964452820808397e-05, 'epoch': 0.53}

  5%|▌         | 39/730 [03:31<1:01:52,  5.37s/it]
                                                  
{'debug/num_tok_total': 2442.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2442.0, 'debug/num_lat_loss': 1791.0, 'epoch': 0.53}

  5%|▌         | 39/730 [03:33<1:01:52,  5.37s/it]
                                                  
{'train/ce_loss': 8.3125, 'train/diffusion_loss': 0.4941246509552002, 'epoch': 0.53}

  5%|▌         | 39/730 [03:33<1:01:52,  5.37s/it]
                                                  
{'train/learning_rate_real': 2.4964452820808397e-05, 'epoch': 0.53}

  5%|▌         | 39/730 [03:33<1:01:52,  5.37s/it]
                                                  
{'debug/num_tok_total': 2457.0, 'debug/num_tok_loss': 1806.0, 'debug/num_lat_total': 2457.0, 'debug/num_lat_loss': 1806.0, 'epoch': 0.53}

  5%|▌         | 39/730 [03:34<1:01:52,  5.37s/it]
                                                  
{'train/ce_loss': 8.375, 'train/diffusion_loss': 0.4619571268558502, 'epoch': 0.53}

  5%|▌         | 39/730 [03:34<1:01:52,  5.37s/it]
                                                  
{'train/learning_rate_real': 2.4964452820808397e-05, 'epoch': 0.53}

  5%|▌         | 39/730 [03:34<1:01:52,  5.37s/it]
                                                  
{'debug/num_tok_total': 2303.0, 'debug/num_tok_loss': 1661.0, 'debug/num_lat_total': 2303.0, 'debug/num_lat_loss': 1661.0, 'epoch': 0.53}

  5%|▌         | 39/730 [03:35<1:01:52,  5.37s/it]
                                                  
{'train/ce_loss': 8.375, 'train/diffusion_loss': 0.4603547155857086, 'epoch': 0.53}

  5%|▌         | 39/730 [03:35<1:01:52,  5.37s/it]
                                                  
{'train/learning_rate_real': 2.4964452820808397e-05, 'epoch': 0.53}

  5%|▌         | 39/730 [03:35<1:01:52,  5.37s/it]03/16/2026 06:51:07 - INFO - __main__ - LoRA debug step 40: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

  5%|▌         | 40/730 [03:35<1:01:09,  5.32s/it]
                                                  
{'loss': 3.9661, 'grad_norm': 2.600710391998291, 'learning_rate': 2.4964452820808397e-05, 'epoch': 0.55}

  5%|▌         | 40/730 [03:35<1:01:09,  5.32s/it]
                                                  
{'debug/num_tok_total': 2843.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2843.0, 'debug/num_lat_loss': 1779.0, 'epoch': 0.55}

  5%|▌         | 40/730 [03:37<1:01:09,  5.32s/it]
                                                  
{'train/ce_loss': 8.125, 'train/diffusion_loss': 0.37036633491516113, 'epoch': 0.55}

  5%|▌         | 40/730 [03:37<1:01:09,  5.32s/it]
                                                  
{'train/learning_rate_real': 2.4960150087884376e-05, 'epoch': 0.55}

  5%|▌         | 40/730 [03:37<1:01:09,  5.32s/it]
                                                  
{'debug/num_tok_total': 2626.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2626.0, 'debug/num_lat_loss': 1776.0, 'epoch': 0.55}

  5%|▌         | 40/730 [03:38<1:01:09,  5.32s/it]
                                                  
{'train/ce_loss': 8.125, 'train/diffusion_loss': 0.4317276179790497, 'epoch': 0.55}

  5%|▌         | 40/730 [03:38<1:01:09,  5.32s/it]
                                                  
{'train/learning_rate_real': 2.4960150087884376e-05, 'epoch': 0.55}

  5%|▌         | 40/730 [03:38<1:01:09,  5.32s/it]
                                                  
{'debug/num_tok_total': 2725.0, 'debug/num_tok_loss': 1606.0, 'debug/num_lat_total': 2725.0, 'debug/num_lat_loss': 1606.0, 'epoch': 0.55}

  5%|▌         | 40/730 [03:39<1:01:09,  5.32s/it]
                                                  
{'train/ce_loss': 8.1875, 'train/diffusion_loss': 0.33192673325538635, 'epoch': 0.55}

  5%|▌         | 40/730 [03:39<1:01:09,  5.32s/it]
                                                  
{'train/learning_rate_real': 2.4960150087884376e-05, 'epoch': 0.55}

  5%|▌         | 40/730 [03:39<1:01:09,  5.32s/it]
                                                  
{'debug/num_tok_total': 2447.0, 'debug/num_tok_loss': 1804.0, 'debug/num_lat_total': 2447.0, 'debug/num_lat_loss': 1804.0, 'epoch': 0.55}

  5%|▌         | 40/730 [03:40<1:01:09,  5.32s/it]
                                                  
{'train/ce_loss': 8.1875, 'train/diffusion_loss': 0.4681604206562042, 'epoch': 0.55}

  5%|▌         | 40/730 [03:40<1:01:09,  5.32s/it]
                                                  
{'train/learning_rate_real': 2.4960150087884376e-05, 'epoch': 0.55}

  5%|▌         | 40/730 [03:40<1:01:09,  5.32s/it]
  6%|▌         | 41/730 [03:41<1:01:09,  5.33s/it]
                                                  
{'debug/num_tok_total': 2449.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2449.0, 'debug/num_lat_loss': 1787.0, 'epoch': 0.56}

  6%|▌         | 41/730 [03:42<1:01:09,  5.33s/it]
                                                  
{'train/ce_loss': 7.8125, 'train/diffusion_loss': 0.450018048286438, 'epoch': 0.56}

  6%|▌         | 41/730 [03:42<1:01:09,  5.33s/it]
                                                  
{'train/learning_rate_real': 2.495560202206119e-05, 'epoch': 0.56}

  6%|▌         | 41/730 [03:42<1:01:09,  5.33s/it]
                                                  
{'debug/num_tok_total': 2474.0, 'debug/num_tok_loss': 1818.0, 'debug/num_lat_total': 2474.0, 'debug/num_lat_loss': 1818.0, 'epoch': 0.56}

  6%|▌         | 41/730 [03:43<1:01:09,  5.33s/it]
                                                  
{'train/ce_loss': 7.84375, 'train/diffusion_loss': 0.4612468481063843, 'epoch': 0.56}

  6%|▌         | 41/730 [03:43<1:01:09,  5.33s/it]
                                                  
{'train/learning_rate_real': 2.495560202206119e-05, 'epoch': 0.56}

  6%|▌         | 41/730 [03:43<1:01:09,  5.33s/it]
                                                  
{'debug/num_tok_total': 2614.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2614.0, 'debug/num_lat_loss': 1776.0, 'epoch': 0.56}

  6%|▌         | 41/730 [03:44<1:01:09,  5.33s/it]
                                                  
{'train/ce_loss': 7.90625, 'train/diffusion_loss': 0.4192771017551422, 'epoch': 0.56}

  6%|▌         | 41/730 [03:44<1:01:09,  5.33s/it]
                                                  
{'train/learning_rate_real': 2.495560202206119e-05, 'epoch': 0.56}

  6%|▌         | 41/730 [03:44<1:01:09,  5.33s/it]
                                                  
{'debug/num_tok_total': 2437.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2437.0, 'debug/num_lat_loss': 1787.0, 'epoch': 0.56}

  6%|▌         | 41/730 [03:46<1:01:09,  5.33s/it]
                                                  
{'train/ce_loss': 7.8125, 'train/diffusion_loss': 0.47766613960266113, 'epoch': 0.56}

  6%|▌         | 41/730 [03:46<1:01:09,  5.33s/it]
                                                  
{'train/learning_rate_real': 2.495560202206119e-05, 'epoch': 0.56}

  6%|▌         | 41/730 [03:46<1:01:09,  5.33s/it]
  6%|▌         | 42/730 [03:46<1:02:07,  5.42s/it]
                                                  
{'debug/num_tok_total': 2647.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2647.0, 'debug/num_lat_loss': 1786.0, 'epoch': 0.58}

  6%|▌         | 42/730 [03:47<1:02:07,  5.42s/it]
                                                  
{'train/ce_loss': 7.53125, 'train/diffusion_loss': 0.4023301303386688, 'epoch': 0.58}

  6%|▌         | 42/730 [03:48<1:02:07,  5.42s/it]
                                                  
{'train/learning_rate_real': 2.4950808712887533e-05, 'epoch': 0.58}

  6%|▌         | 42/730 [03:48<1:02:07,  5.42s/it]
                                                  
{'debug/num_tok_total': 3016.0, 'debug/num_tok_loss': 1758.0, 'debug/num_lat_total': 3016.0, 'debug/num_lat_loss': 1758.0, 'epoch': 0.58}

  6%|▌         | 42/730 [03:49<1:02:07,  5.42s/it]
                                                  
{'train/ce_loss': 7.59375, 'train/diffusion_loss': 0.360873818397522, 'epoch': 0.58}

  6%|▌         | 42/730 [03:49<1:02:07,  5.42s/it]
                                                  
{'train/learning_rate_real': 2.4950808712887533e-05, 'epoch': 0.58}

  6%|▌         | 42/730 [03:49<1:02:07,  5.42s/it]
                                                  
{'debug/num_tok_total': 2704.0, 'debug/num_tok_loss': 1818.0, 'debug/num_lat_total': 2704.0, 'debug/num_lat_loss': 1818.0, 'epoch': 0.58}

  6%|▌         | 42/730 [03:50<1:02:07,  5.42s/it]
                                                  
{'train/ce_loss': 7.53125, 'train/diffusion_loss': 0.42654749751091003, 'epoch': 0.58}

  6%|▌         | 42/730 [03:50<1:02:07,  5.42s/it]
                                                  
{'train/learning_rate_real': 2.4950808712887533e-05, 'epoch': 0.58}

  6%|▌         | 42/730 [03:50<1:02:07,  5.42s/it]
                                                  
{'debug/num_tok_total': 2207.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2207.0, 'debug/num_lat_loss': 1777.0, 'epoch': 0.58}

  6%|▌         | 42/730 [03:51<1:02:07,  5.42s/it]
                                                  
{'train/ce_loss': 7.71875, 'train/diffusion_loss': 0.5486947894096375, 'epoch': 0.58}

  6%|▌         | 42/730 [03:51<1:02:07,  5.42s/it]
                                                  
{'train/learning_rate_real': 2.4950808712887533e-05, 'epoch': 0.58}

  6%|▌         | 42/730 [03:51<1:02:07,  5.42s/it]
  6%|▌         | 43/730 [03:52<1:02:06,  5.42s/it]
                                                  
{'debug/num_tok_total': 2862.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2862.0, 'debug/num_lat_loss': 1781.0, 'epoch': 0.59}

  6%|▌         | 43/730 [03:53<1:02:06,  5.42s/it]
                                                  
{'train/ce_loss': 7.65625, 'train/diffusion_loss': 0.3878563344478607, 'epoch': 0.59}

  6%|▌         | 43/730 [03:53<1:02:06,  5.42s/it]
                                                  
{'train/learning_rate_real': 2.4945770254740794e-05, 'epoch': 0.59}

  6%|▌         | 43/730 [03:53<1:02:06,  5.42s/it]
                                                  
{'debug/num_tok_total': 2436.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2436.0, 'debug/num_lat_loss': 1787.0, 'epoch': 0.59}

  6%|▌         | 43/730 [03:54<1:02:06,  5.42s/it]
                                                  
{'train/ce_loss': 7.34375, 'train/diffusion_loss': 0.4670861065387726, 'epoch': 0.59}

  6%|▌         | 43/730 [03:54<1:02:06,  5.42s/it]
                                                  
{'train/learning_rate_real': 2.4945770254740794e-05, 'epoch': 0.59}

  6%|▌         | 43/730 [03:54<1:02:06,  5.42s/it]
                                                  
{'debug/num_tok_total': 2687.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 2687.0, 'debug/num_lat_loss': 1801.0, 'epoch': 0.59}

  6%|▌         | 43/730 [03:55<1:02:06,  5.42s/it]
                                                  
{'train/ce_loss': 7.3125, 'train/diffusion_loss': 0.4248601496219635, 'epoch': 0.59}

  6%|▌         | 43/730 [03:55<1:02:06,  5.42s/it]
                                                  
{'train/learning_rate_real': 2.4945770254740794e-05, 'epoch': 0.59}

  6%|▌         | 43/730 [03:55<1:02:06,  5.42s/it]
                                                  
{'debug/num_tok_total': 2381.0, 'debug/num_tok_loss': 1754.0, 'debug/num_lat_total': 2381.0, 'debug/num_lat_loss': 1754.0, 'epoch': 0.59}

  6%|▌         | 43/730 [03:57<1:02:06,  5.42s/it]
                                                  
{'train/ce_loss': 7.28125, 'train/diffusion_loss': 0.49052563309669495, 'epoch': 0.59}

  6%|▌         | 43/730 [03:57<1:02:06,  5.42s/it]
                                                  
{'train/learning_rate_real': 2.4945770254740794e-05, 'epoch': 0.59}

  6%|▌         | 43/730 [03:57<1:02:06,  5.42s/it]
  6%|▌         | 44/730 [03:57<1:01:48,  5.41s/it]
                                                  
{'debug/num_tok_total': 2239.0, 'debug/num_tok_loss': 1806.0, 'debug/num_lat_total': 2239.0, 'debug/num_lat_loss': 1806.0, 'epoch': 0.6}

  6%|▌         | 44/730 [03:58<1:01:48,  5.41s/it]
                                                  
{'train/ce_loss': 7.03125, 'train/diffusion_loss': 0.5206171870231628, 'epoch': 0.6}

  6%|▌         | 44/730 [03:58<1:01:48,  5.41s/it]
                                                  
{'train/learning_rate_real': 2.4940486746825197e-05, 'epoch': 0.6}

  6%|▌         | 44/730 [03:58<1:01:48,  5.41s/it]
                                                  
{'debug/num_tok_total': 3057.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 3057.0, 'debug/num_lat_loss': 1768.0, 'epoch': 0.6}

  6%|▌         | 44/730 [04:00<1:01:48,  5.41s/it]
                                                  
{'train/ce_loss': 7.21875, 'train/diffusion_loss': 0.3397883474826813, 'epoch': 0.6}

  6%|▌         | 44/730 [04:00<1:01:48,  5.41s/it]
                                                  
{'train/learning_rate_real': 2.4940486746825197e-05, 'epoch': 0.6}

  6%|▌         | 44/730 [04:00<1:01:48,  5.41s/it]
                                                  
{'debug/num_tok_total': 2164.0, 'debug/num_tok_loss': 1652.0, 'debug/num_lat_total': 2164.0, 'debug/num_lat_loss': 1652.0, 'epoch': 0.6}

  6%|▌         | 44/730 [04:01<1:01:48,  5.41s/it]
                                                  
{'train/ce_loss': 7.09375, 'train/diffusion_loss': 0.4852142930030823, 'epoch': 0.6}

  6%|▌         | 44/730 [04:01<1:01:48,  5.41s/it]
                                                  
{'train/learning_rate_real': 2.4940486746825197e-05, 'epoch': 0.6}

  6%|▌         | 44/730 [04:01<1:01:48,  5.41s/it]
                                                  
{'debug/num_tok_total': 3095.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 3095.0, 'debug/num_lat_loss': 1795.0, 'epoch': 0.6}

  6%|▌         | 44/730 [04:02<1:01:48,  5.41s/it]
                                                  
{'train/ce_loss': 7.15625, 'train/diffusion_loss': 0.33311718702316284, 'epoch': 0.6}

  6%|▌         | 44/730 [04:02<1:01:48,  5.41s/it]
                                                  
{'train/learning_rate_real': 2.4940486746825197e-05, 'epoch': 0.6}

  6%|▌         | 44/730 [04:02<1:01:48,  5.41s/it]
  6%|▌         | 45/730 [04:02<1:01:24,  5.38s/it]
                                                  
{'debug/num_tok_total': 2680.0, 'debug/num_tok_loss': 1700.0, 'debug/num_lat_total': 2680.0, 'debug/num_lat_loss': 1700.0, 'epoch': 0.62}

  6%|▌         | 45/730 [04:04<1:01:24,  5.38s/it]
                                                  
{'train/ce_loss': 7.03125, 'train/diffusion_loss': 0.3979421555995941, 'epoch': 0.62}

  6%|▌         | 45/730 [04:04<1:01:24,  5.38s/it]
                                                  
{'train/learning_rate_real': 2.493495829316986e-05, 'epoch': 0.62}

  6%|▌         | 45/730 [04:04<1:01:24,  5.38s/it]
                                                  
{'debug/num_tok_total': 2441.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2441.0, 'debug/num_lat_loss': 1802.0, 'epoch': 0.62}

  6%|▌         | 45/730 [04:05<1:01:24,  5.38s/it]
                                                  
{'train/ce_loss': 6.90625, 'train/diffusion_loss': 0.47810766100883484, 'epoch': 0.62}

  6%|▌         | 45/730 [04:05<1:01:24,  5.38s/it]
                                                  
{'train/learning_rate_real': 2.493495829316986e-05, 'epoch': 0.62}

  6%|▌         | 45/730 [04:05<1:01:24,  5.38s/it]
                                                  
{'debug/num_tok_total': 2632.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2632.0, 'debug/num_lat_loss': 1779.0, 'epoch': 0.62}

  6%|▌         | 45/730 [04:06<1:01:24,  5.38s/it]
                                                  
{'train/ce_loss': 6.90625, 'train/diffusion_loss': 0.4057077169418335, 'epoch': 0.62}

  6%|▌         | 45/730 [04:06<1:01:24,  5.38s/it]
                                                  
{'train/learning_rate_real': 2.493495829316986e-05, 'epoch': 0.62}

  6%|▌         | 45/730 [04:06<1:01:24,  5.38s/it]
                                                  
{'debug/num_tok_total': 2506.0, 'debug/num_tok_loss': 1716.0, 'debug/num_lat_total': 2506.0, 'debug/num_lat_loss': 1716.0, 'epoch': 0.62}

  6%|▌         | 45/730 [04:07<1:01:24,  5.38s/it]
                                                  
{'train/ce_loss': 7.03125, 'train/diffusion_loss': 0.4449847936630249, 'epoch': 0.62}

  6%|▌         | 45/730 [04:07<1:01:24,  5.38s/it]
                                                  
{'train/learning_rate_real': 2.493495829316986e-05, 'epoch': 0.62}

  6%|▌         | 45/730 [04:07<1:01:24,  5.38s/it]
  6%|▋         | 46/730 [04:08<1:01:38,  5.41s/it]
                                                  
{'debug/num_tok_total': 2764.0, 'debug/num_tok_loss': 1629.0, 'debug/num_lat_total': 2764.0, 'debug/num_lat_loss': 1629.0, 'epoch': 0.63}

  6%|▋         | 46/730 [04:09<1:01:38,  5.41s/it]
                                                  
{'train/ce_loss': 6.84375, 'train/diffusion_loss': 0.3362199664115906, 'epoch': 0.63}

  6%|▋         | 46/730 [04:09<1:01:38,  5.41s/it]
                                                  
{'train/learning_rate_real': 2.4929185002626714e-05, 'epoch': 0.63}

  6%|▋         | 46/730 [04:09<1:01:38,  5.41s/it]
                                                  
{'debug/num_tok_total': 2588.0, 'debug/num_tok_loss': 1749.0, 'debug/num_lat_total': 2588.0, 'debug/num_lat_loss': 1749.0, 'epoch': 0.63}

  6%|▋         | 46/730 [04:10<1:01:38,  5.41s/it]
                                                  
{'train/ce_loss': 6.78125, 'train/diffusion_loss': 0.447072297334671, 'epoch': 0.63}

  6%|▋         | 46/730 [04:10<1:01:38,  5.41s/it]
                                                  
{'train/learning_rate_real': 2.4929185002626714e-05, 'epoch': 0.63}

  6%|▋         | 46/730 [04:10<1:01:38,  5.41s/it]
                                                  
{'debug/num_tok_total': 2666.0, 'debug/num_tok_loss': 1798.0, 'debug/num_lat_total': 2666.0, 'debug/num_lat_loss': 1798.0, 'epoch': 0.63}

  6%|▋         | 46/730 [04:12<1:01:38,  5.41s/it]
                                                  
{'train/ce_loss': 6.71875, 'train/diffusion_loss': 0.4139653742313385, 'epoch': 0.63}

  6%|▋         | 46/730 [04:12<1:01:38,  5.41s/it]
                                                  
{'train/learning_rate_real': 2.4929185002626714e-05, 'epoch': 0.63}

  6%|▋         | 46/730 [04:12<1:01:38,  5.41s/it]
                                                  
{'debug/num_tok_total': 2716.0, 'debug/num_tok_loss': 1716.0, 'debug/num_lat_total': 2716.0, 'debug/num_lat_loss': 1716.0, 'epoch': 0.63}

  6%|▋         | 46/730 [04:13<1:01:38,  5.41s/it]
                                                  
{'train/ce_loss': 6.875, 'train/diffusion_loss': 0.3880141079425812, 'epoch': 0.63}

  6%|▋         | 46/730 [04:13<1:01:38,  5.41s/it]
                                                  
{'train/learning_rate_real': 2.4929185002626714e-05, 'epoch': 0.63}

  6%|▋         | 46/730 [04:13<1:01:38,  5.41s/it]
  6%|▋         | 47/730 [04:13<1:02:13,  5.47s/it]
                                                  
{'debug/num_tok_total': 2618.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2618.0, 'debug/num_lat_loss': 1763.0, 'epoch': 0.64}

  6%|▋         | 47/730 [04:15<1:02:13,  5.47s/it]
                                                  
{'train/ce_loss': 6.5, 'train/diffusion_loss': 0.4298379421234131, 'epoch': 0.64}

  6%|▋         | 47/730 [04:15<1:02:13,  5.47s/it]
                                                  
{'train/learning_rate_real': 2.4923166988868407e-05, 'epoch': 0.64}

  6%|▋         | 47/730 [04:15<1:02:13,  5.47s/it]
                                                  
{'debug/num_tok_total': 3106.0, 'debug/num_tok_loss': 1803.0, 'debug/num_lat_total': 3106.0, 'debug/num_lat_loss': 1803.0, 'epoch': 0.64}

  6%|▋         | 47/730 [04:16<1:02:13,  5.47s/it]
                                                  
{'train/ce_loss': 6.59375, 'train/diffusion_loss': 0.3285239040851593, 'epoch': 0.64}

  6%|▋         | 47/730 [04:16<1:02:13,  5.47s/it]
                                                  
{'train/learning_rate_real': 2.4923166988868407e-05, 'epoch': 0.64}

  6%|▋         | 47/730 [04:16<1:02:13,  5.47s/it]
                                                  
{'debug/num_tok_total': 2233.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2233.0, 'debug/num_lat_loss': 1793.0, 'epoch': 0.64}

  6%|▋         | 47/730 [04:17<1:02:13,  5.47s/it]
                                                  
{'train/ce_loss': 6.5625, 'train/diffusion_loss': 0.5352428555488586, 'epoch': 0.64}

  6%|▋         | 47/730 [04:17<1:02:13,  5.47s/it]
                                                  
{'train/learning_rate_real': 2.4923166988868407e-05, 'epoch': 0.64}

  6%|▋         | 47/730 [04:17<1:02:13,  5.47s/it]
                                                  
{'debug/num_tok_total': 2406.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2406.0, 'debug/num_lat_loss': 1764.0, 'epoch': 0.64}

  6%|▋         | 47/730 [04:18<1:02:13,  5.47s/it]
                                                  
{'train/ce_loss': 6.84375, 'train/diffusion_loss': 0.45154592394828796, 'epoch': 0.64}

  6%|▋         | 47/730 [04:18<1:02:13,  5.47s/it]
                                                  
{'train/learning_rate_real': 2.4923166988868407e-05, 'epoch': 0.64}

  6%|▋         | 47/730 [04:18<1:02:13,  5.47s/it]
  7%|▋         | 48/730 [04:19<1:01:10,  5.38s/it]
                                                  
{'debug/num_tok_total': 2466.0, 'debug/num_tok_loss': 1803.0, 'debug/num_lat_total': 2466.0, 'debug/num_lat_loss': 1803.0, 'epoch': 0.66}

  7%|▋         | 48/730 [04:20<1:01:10,  5.38s/it]
                                                  
{'train/ce_loss': 6.4375, 'train/diffusion_loss': 0.4415627121925354, 'epoch': 0.66}

  7%|▋         | 48/730 [04:20<1:01:10,  5.38s/it]
                                                  
{'train/learning_rate_real': 2.491690437038602e-05, 'epoch': 0.66}

  7%|▋         | 48/730 [04:20<1:01:10,  5.38s/it]
                                                  
{'debug/num_tok_total': 3149.0, 'debug/num_tok_loss': 1604.0, 'debug/num_lat_total': 3149.0, 'debug/num_lat_loss': 1604.0, 'epoch': 0.66}

  7%|▋         | 48/730 [04:21<1:01:10,  5.38s/it]
                                                  
{'train/ce_loss': 6.5625, 'train/diffusion_loss': 0.24725854396820068, 'epoch': 0.66}

  7%|▋         | 48/730 [04:21<1:01:10,  5.38s/it]
                                                  
{'train/learning_rate_real': 2.491690437038602e-05, 'epoch': 0.66}

  7%|▋         | 48/730 [04:21<1:01:10,  5.38s/it]
                                                  
{'debug/num_tok_total': 2419.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2419.0, 'debug/num_lat_loss': 1783.0, 'epoch': 0.66}

  7%|▋         | 48/730 [04:23<1:01:10,  5.38s/it]
                                                  
{'train/ce_loss': 6.40625, 'train/diffusion_loss': 0.4734286665916443, 'epoch': 0.66}

  7%|▋         | 48/730 [04:23<1:01:10,  5.38s/it]
                                                  
{'train/learning_rate_real': 2.491690437038602e-05, 'epoch': 0.66}

  7%|▋         | 48/730 [04:23<1:01:10,  5.38s/it]
                                                  
{'debug/num_tok_total': 3086.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 3086.0, 'debug/num_lat_loss': 1785.0, 'epoch': 0.66}

  7%|▋         | 48/730 [04:24<1:01:10,  5.38s/it]
                                                  
{'train/ce_loss': 6.46875, 'train/diffusion_loss': 0.32624179124832153, 'epoch': 0.66}

  7%|▋         | 48/730 [04:24<1:01:10,  5.38s/it]
                                                  
{'train/learning_rate_real': 2.491690437038602e-05, 'epoch': 0.66}

  7%|▋         | 48/730 [04:24<1:01:10,  5.38s/it]
  7%|▋         | 49/730 [04:24<1:01:52,  5.45s/it]
                                                  
{'debug/num_tok_total': 2873.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2873.0, 'debug/num_lat_loss': 1792.0, 'epoch': 0.67}

  7%|▋         | 49/730 [04:25<1:01:52,  5.45s/it]
                                                  
{'train/ce_loss': 6.25, 'train/diffusion_loss': 0.38346582651138306, 'epoch': 0.67}

  7%|▋         | 49/730 [04:26<1:01:52,  5.45s/it]
                                                  
{'train/learning_rate_real': 2.491039727048677e-05, 'epoch': 0.67}

  7%|▋         | 49/730 [04:26<1:01:52,  5.45s/it]
                                                  
{'debug/num_tok_total': 2609.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2609.0, 'debug/num_lat_loss': 1763.0, 'epoch': 0.67}

  7%|▋         | 49/730 [04:27<1:01:52,  5.45s/it]
                                                  
{'train/ce_loss': 6.25, 'train/diffusion_loss': 0.4216737449169159, 'epoch': 0.67}

  7%|▋         | 49/730 [04:27<1:01:52,  5.45s/it]
                                                  
{'train/learning_rate_real': 2.491039727048677e-05, 'epoch': 0.67}

  7%|▋         | 49/730 [04:27<1:01:52,  5.45s/it]
                                                  
{'debug/num_tok_total': 2620.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 2620.0, 'debug/num_lat_loss': 1769.0, 'epoch': 0.67}

  7%|▋         | 49/730 [04:28<1:01:52,  5.45s/it]
                                                  
{'train/ce_loss': 6.25, 'train/diffusion_loss': 0.4077589511871338, 'epoch': 0.67}

  7%|▋         | 49/730 [04:28<1:01:52,  5.45s/it]
                                                  
{'train/learning_rate_real': 2.491039727048677e-05, 'epoch': 0.67}

  7%|▋         | 49/730 [04:28<1:01:52,  5.45s/it]
                                                  
{'debug/num_tok_total': 2331.0, 'debug/num_tok_loss': 1635.0, 'debug/num_lat_total': 2331.0, 'debug/num_lat_loss': 1635.0, 'epoch': 0.67}

  7%|▋         | 49/730 [04:29<1:01:52,  5.45s/it]
                                                  
{'train/ce_loss': 6.28125, 'train/diffusion_loss': 0.43508994579315186, 'epoch': 0.67}

  7%|▋         | 49/730 [04:29<1:01:52,  5.45s/it]
                                                  
{'train/learning_rate_real': 2.491039727048677e-05, 'epoch': 0.67}

  7%|▋         | 49/730 [04:29<1:01:52,  5.45s/it]03/16/2026 06:52:02 - INFO - __main__ - LoRA debug step 50: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

  7%|▋         | 50/730 [04:30<1:01:30,  5.43s/it]
                                                  
{'loss': 3.4905, 'grad_norm': 2.2437052726745605, 'learning_rate': 2.491039727048677e-05, 'epoch': 0.68}

  7%|▋         | 50/730 [04:30<1:01:30,  5.43s/it]
                                                  
{'debug/num_tok_total': 3297.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 3297.0, 'debug/num_lat_loss': 1782.0, 'epoch': 0.68}

  7%|▋         | 50/730 [04:31<1:01:30,  5.43s/it]
                                                  
{'train/ce_loss': 6.15625, 'train/diffusion_loss': 0.3150237500667572, 'epoch': 0.68}

  7%|▋         | 50/730 [04:31<1:01:30,  5.43s/it]
                                                  
{'train/learning_rate_real': 2.490364581729156e-05, 'epoch': 0.68}

  7%|▋         | 50/730 [04:31<1:01:30,  5.43s/it]
                                                  
{'debug/num_tok_total': 2550.0, 'debug/num_tok_loss': 1631.0, 'debug/num_lat_total': 2550.0, 'debug/num_lat_loss': 1631.0, 'epoch': 0.68}

  7%|▋         | 50/730 [04:32<1:01:30,  5.43s/it]
                                                  
{'train/ce_loss': 6.21875, 'train/diffusion_loss': 0.39398685097694397, 'epoch': 0.68}

  7%|▋         | 50/730 [04:32<1:01:30,  5.43s/it]
                                                  
{'train/learning_rate_real': 2.490364581729156e-05, 'epoch': 0.68}

  7%|▋         | 50/730 [04:32<1:01:30,  5.43s/it]
                                                  
{'debug/num_tok_total': 2623.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2623.0, 'debug/num_lat_loss': 1764.0, 'epoch': 0.68}

  7%|▋         | 50/730 [04:34<1:01:30,  5.43s/it]
                                                  
{'train/ce_loss': 6.03125, 'train/diffusion_loss': 0.41709935665130615, 'epoch': 0.68}

  7%|▋         | 50/730 [04:34<1:01:30,  5.43s/it]
                                                  
{'train/learning_rate_real': 2.490364581729156e-05, 'epoch': 0.68}

  7%|▋         | 50/730 [04:34<1:01:30,  5.43s/it]
                                                  
{'debug/num_tok_total': 2113.0, 'debug/num_tok_loss': 1681.0, 'debug/num_lat_total': 2113.0, 'debug/num_lat_loss': 1681.0, 'epoch': 0.68}

  7%|▋         | 50/730 [04:35<1:01:30,  5.43s/it]
                                                  
{'train/ce_loss': 6.03125, 'train/diffusion_loss': 0.5213543772697449, 'epoch': 0.68}

  7%|▋         | 50/730 [04:35<1:01:30,  5.43s/it]
                                                  
{'train/learning_rate_real': 2.490364581729156e-05, 'epoch': 0.68}

  7%|▋         | 50/730 [04:35<1:01:30,  5.43s/it]
  7%|▋         | 51/730 [04:35<1:01:32,  5.44s/it]
                                                  
{'debug/num_tok_total': 2661.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2661.0, 'debug/num_lat_loss': 1793.0, 'epoch': 0.7}

  7%|▋         | 51/730 [04:36<1:01:32,  5.44s/it]
                                                  
{'train/ce_loss': 5.9375, 'train/diffusion_loss': 0.4208451509475708, 'epoch': 0.7}

  7%|▋         | 51/730 [04:36<1:01:32,  5.44s/it]
                                                  
{'train/learning_rate_real': 2.4896650143732457e-05, 'epoch': 0.7}

  7%|▋         | 51/730 [04:36<1:01:32,  5.44s/it]
                                                  
{'debug/num_tok_total': 2217.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2217.0, 'debug/num_lat_loss': 1788.0, 'epoch': 0.7}

  7%|▋         | 51/730 [04:37<1:01:32,  5.44s/it]
                                                  
{'train/ce_loss': 6.15625, 'train/diffusion_loss': 0.5214932560920715, 'epoch': 0.7}

  7%|▋         | 51/730 [04:37<1:01:32,  5.44s/it]
                                                  
{'train/learning_rate_real': 2.4896650143732457e-05, 'epoch': 0.7}

  7%|▋         | 51/730 [04:37<1:01:32,  5.44s/it]
                                                  
{'debug/num_tok_total': 2379.0, 'debug/num_tok_loss': 1747.0, 'debug/num_lat_total': 2379.0, 'debug/num_lat_loss': 1747.0, 'epoch': 0.7}

  7%|▋         | 51/730 [04:39<1:01:32,  5.44s/it]
                                                  
{'train/ce_loss': 6.0, 'train/diffusion_loss': 0.4774201214313507, 'epoch': 0.7}

  7%|▋         | 51/730 [04:39<1:01:32,  5.44s/it]
                                                  
{'train/learning_rate_real': 2.4896650143732457e-05, 'epoch': 0.7}

  7%|▋         | 51/730 [04:39<1:01:32,  5.44s/it]
                                                  
{'debug/num_tok_total': 2274.0, 'debug/num_tok_loss': 1621.0, 'debug/num_lat_total': 2274.0, 'debug/num_lat_loss': 1621.0, 'epoch': 0.7}

  7%|▋         | 51/730 [04:40<1:01:32,  5.44s/it]
                                                  
{'train/ce_loss': 6.09375, 'train/diffusion_loss': 0.44967830181121826, 'epoch': 0.7}

  7%|▋         | 51/730 [04:40<1:01:32,  5.44s/it]
                                                  
{'train/learning_rate_real': 2.4896650143732457e-05, 'epoch': 0.7}

  7%|▋         | 51/730 [04:40<1:01:32,  5.44s/it]
  7%|▋         | 52/730 [04:40<59:54,  5.30s/it]  
                                                
{'debug/num_tok_total': 2915.0, 'debug/num_tok_loss': 1820.0, 'debug/num_lat_total': 2915.0, 'debug/num_lat_loss': 1820.0, 'epoch': 0.71}

  7%|▋         | 52/730 [04:41<59:54,  5.30s/it]
                                                
{'train/ce_loss': 5.8125, 'train/diffusion_loss': 0.35934463143348694, 'epoch': 0.71}

  7%|▋         | 52/730 [04:41<59:54,  5.30s/it]
                                                
{'train/learning_rate_real': 2.4889410387550093e-05, 'epoch': 0.71}

  7%|▋         | 52/730 [04:41<59:54,  5.30s/it]
                                                
{'debug/num_tok_total': 2635.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2635.0, 'debug/num_lat_loss': 1779.0, 'epoch': 0.71}

  7%|▋         | 52/730 [04:43<59:54,  5.30s/it]
                                                
{'train/ce_loss': 5.75, 'train/diffusion_loss': 0.4354902505874634, 'epoch': 0.71}

  7%|▋         | 52/730 [04:43<59:54,  5.30s/it]
                                                
{'train/learning_rate_real': 2.4889410387550093e-05, 'epoch': 0.71}

  7%|▋         | 52/730 [04:43<59:54,  5.30s/it]
                                                
{'debug/num_tok_total': 2865.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2865.0, 'debug/num_lat_loss': 1788.0, 'epoch': 0.71}

  7%|▋         | 52/730 [04:44<59:54,  5.30s/it]
                                                
{'train/ce_loss': 5.78125, 'train/diffusion_loss': 0.3834609389305115, 'epoch': 0.71}

  7%|▋         | 52/730 [04:44<59:54,  5.30s/it]
                                                
{'train/learning_rate_real': 2.4889410387550093e-05, 'epoch': 0.71}

  7%|▋         | 52/730 [04:44<59:54,  5.30s/it]
                                                
{'debug/num_tok_total': 2840.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2840.0, 'debug/num_lat_loss': 1773.0, 'epoch': 0.71}

  7%|▋         | 52/730 [04:45<59:54,  5.30s/it]
                                                
{'train/ce_loss': 5.90625, 'train/diffusion_loss': 0.3739621043205261, 'epoch': 0.71}

  7%|▋         | 52/730 [04:45<59:54,  5.30s/it]
                                                
{'train/learning_rate_real': 2.4889410387550093e-05, 'epoch': 0.71}

  7%|▋         | 52/730 [04:45<59:54,  5.30s/it]
  7%|▋         | 53/730 [04:46<1:00:49,  5.39s/it]
                                                  
{'debug/num_tok_total': 2414.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2414.0, 'debug/num_lat_loss': 1773.0, 'epoch': 0.73}

  7%|▋         | 53/730 [04:47<1:00:49,  5.39s/it]
                                                  
{'train/ce_loss': 5.96875, 'train/diffusion_loss': 0.4601896107196808, 'epoch': 0.73}

  7%|▋         | 53/730 [04:47<1:00:49,  5.39s/it]
                                                  
{'train/learning_rate_real': 2.488192669129093e-05, 'epoch': 0.73}

  7%|▋         | 53/730 [04:47<1:00:49,  5.39s/it]
                                                  
{'debug/num_tok_total': 2902.0, 'debug/num_tok_loss': 1807.0, 'debug/num_lat_total': 2902.0, 'debug/num_lat_loss': 1807.0, 'epoch': 0.73}

  7%|▋         | 53/730 [04:48<1:00:49,  5.39s/it]
                                                  
{'train/ce_loss': 6.0, 'train/diffusion_loss': 0.3668011426925659, 'epoch': 0.73}

  7%|▋         | 53/730 [04:48<1:00:49,  5.39s/it]
                                                  
{'train/learning_rate_real': 2.488192669129093e-05, 'epoch': 0.73}

  7%|▋         | 53/730 [04:48<1:00:49,  5.39s/it]
                                                  
{'debug/num_tok_total': 3089.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 3089.0, 'debug/num_lat_loss': 1786.0, 'epoch': 0.73}

  7%|▋         | 53/730 [04:49<1:00:49,  5.39s/it]
                                                  
{'train/ce_loss': 5.78125, 'train/diffusion_loss': 0.3401413857936859, 'epoch': 0.73}

  7%|▋         | 53/730 [04:49<1:00:49,  5.39s/it]
                                                  
{'train/learning_rate_real': 2.488192669129093e-05, 'epoch': 0.73}

  7%|▋         | 53/730 [04:49<1:00:49,  5.39s/it]
                                                  
{'debug/num_tok_total': 3095.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 3095.0, 'debug/num_lat_loss': 1792.0, 'epoch': 0.73}

  7%|▋         | 53/730 [04:51<1:00:49,  5.39s/it]
                                                  
{'train/ce_loss': 6.03125, 'train/diffusion_loss': 0.33927619457244873, 'epoch': 0.73}

  7%|▋         | 53/730 [04:51<1:00:49,  5.39s/it]
                                                  
{'train/learning_rate_real': 2.488192669129093e-05, 'epoch': 0.73}

  7%|▋         | 53/730 [04:51<1:00:49,  5.39s/it]
  7%|▋         | 54/730 [04:51<1:00:32,  5.37s/it]
                                                  
{'debug/num_tok_total': 2812.0, 'debug/num_tok_loss': 1756.0, 'debug/num_lat_total': 2812.0, 'debug/num_lat_loss': 1756.0, 'epoch': 0.74}

  7%|▋         | 54/730 [04:52<1:00:32,  5.37s/it]
                                                  
{'train/ce_loss': 5.90625, 'train/diffusion_loss': 0.39091378450393677, 'epoch': 0.74}

  7%|▋         | 54/730 [04:52<1:00:32,  5.37s/it]
                                                  
{'train/learning_rate_real': 2.4874199202304475e-05, 'epoch': 0.74}

  7%|▋         | 54/730 [04:52<1:00:32,  5.37s/it]
                                                  
{'debug/num_tok_total': 2416.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2416.0, 'debug/num_lat_loss': 1780.0, 'epoch': 0.74}

  7%|▋         | 54/730 [04:53<1:00:32,  5.37s/it]
                                                  
{'train/ce_loss': 5.90625, 'train/diffusion_loss': 0.4684540927410126, 'epoch': 0.74}

  7%|▋         | 54/730 [04:53<1:00:32,  5.37s/it]
                                                  
{'train/learning_rate_real': 2.4874199202304475e-05, 'epoch': 0.74}

  7%|▋         | 54/730 [04:53<1:00:32,  5.37s/it]
                                                  
{'debug/num_tok_total': 2662.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2662.0, 'debug/num_lat_loss': 1790.0, 'epoch': 0.74}

  7%|▋         | 54/730 [04:55<1:00:32,  5.37s/it]
                                                  
{'train/ce_loss': 5.875, 'train/diffusion_loss': 0.43764838576316833, 'epoch': 0.74}

  7%|▋         | 54/730 [04:55<1:00:32,  5.37s/it]
                                                  
{'train/learning_rate_real': 2.4874199202304475e-05, 'epoch': 0.74}

  7%|▋         | 54/730 [04:55<1:00:32,  5.37s/it]
                                                  
{'debug/num_tok_total': 2662.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2662.0, 'debug/num_lat_loss': 1790.0, 'epoch': 0.74}

  7%|▋         | 54/730 [04:56<1:00:32,  5.37s/it]
                                                  
{'train/ce_loss': 5.9375, 'train/diffusion_loss': 0.4463954269886017, 'epoch': 0.74}

  7%|▋         | 54/730 [04:56<1:00:32,  5.37s/it]
                                                  
{'train/learning_rate_real': 2.4874199202304475e-05, 'epoch': 0.74}

  7%|▋         | 54/730 [04:56<1:00:32,  5.37s/it]
  8%|▊         | 55/730 [04:56<1:00:06,  5.34s/it]
                                                  
{'debug/num_tok_total': 2869.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2869.0, 'debug/num_lat_loss': 1788.0, 'epoch': 0.75}

  8%|▊         | 55/730 [04:57<1:00:06,  5.34s/it]
                                                  
{'train/ce_loss': 5.65625, 'train/diffusion_loss': 0.3489157557487488, 'epoch': 0.75}

  8%|▊         | 55/730 [04:57<1:00:06,  5.34s/it]
                                                  
{'train/learning_rate_real': 2.486622807274036e-05, 'epoch': 0.75}

  8%|▊         | 55/730 [04:57<1:00:06,  5.34s/it]
                                                  
{'debug/num_tok_total': 2442.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2442.0, 'debug/num_lat_loss': 1792.0, 'epoch': 0.75}

  8%|▊         | 55/730 [04:59<1:00:06,  5.34s/it]
                                                  
{'train/ce_loss': 5.9375, 'train/diffusion_loss': 0.4745232164859772, 'epoch': 0.75}

  8%|▊         | 55/730 [04:59<1:00:06,  5.34s/it]
                                                  
{'train/learning_rate_real': 2.486622807274036e-05, 'epoch': 0.75}

  8%|▊         | 55/730 [04:59<1:00:06,  5.34s/it]
                                                  
{'debug/num_tok_total': 2697.0, 'debug/num_tok_loss': 1695.0, 'debug/num_lat_total': 2697.0, 'debug/num_lat_loss': 1695.0, 'epoch': 0.75}

  8%|▊         | 55/730 [05:00<1:00:06,  5.34s/it]
                                                  
{'train/ce_loss': 5.53125, 'train/diffusion_loss': 0.3809465765953064, 'epoch': 0.75}

  8%|▊         | 55/730 [05:00<1:00:06,  5.34s/it]
                                                  
{'train/learning_rate_real': 2.486622807274036e-05, 'epoch': 0.75}

  8%|▊         | 55/730 [05:00<1:00:06,  5.34s/it]
                                                  
{'debug/num_tok_total': 1777.0, 'debug/num_tok_loss': 1561.0, 'debug/num_lat_total': 1777.0, 'debug/num_lat_loss': 1561.0, 'epoch': 0.75}

  8%|▊         | 55/730 [05:01<1:00:06,  5.34s/it]
                                                  
{'train/ce_loss': 5.6875, 'train/diffusion_loss': 0.5590707063674927, 'epoch': 0.75}

  8%|▊         | 55/730 [05:01<1:00:06,  5.34s/it]
                                                  
{'train/learning_rate_real': 2.486622807274036e-05, 'epoch': 0.75}

  8%|▊         | 55/730 [05:01<1:00:06,  5.34s/it]
  8%|▊         | 56/730 [05:02<59:39,  5.31s/it]  
                                                
{'debug/num_tok_total': 3064.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 3064.0, 'debug/num_lat_loss': 1772.0, 'epoch': 0.77}

  8%|▊         | 56/730 [05:03<59:39,  5.31s/it]
                                                
{'train/ce_loss': 5.59375, 'train/diffusion_loss': 0.3314301371574402, 'epoch': 0.77}

  8%|▊         | 56/730 [05:03<59:39,  5.31s/it]
                                                
{'train/learning_rate_real': 2.4858013459545352e-05, 'epoch': 0.77}

  8%|▊         | 56/730 [05:03<59:39,  5.31s/it]
                                                
{'debug/num_tok_total': 2649.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2649.0, 'debug/num_lat_loss': 1791.0, 'epoch': 0.77}

  8%|▊         | 56/730 [05:04<59:39,  5.31s/it]
                                                
{'train/ce_loss': 5.4375, 'train/diffusion_loss': 0.3836974501609802, 'epoch': 0.77}

  8%|▊         | 56/730 [05:04<59:39,  5.31s/it]
                                                
{'train/learning_rate_real': 2.4858013459545352e-05, 'epoch': 0.77}

  8%|▊         | 56/730 [05:04<59:39,  5.31s/it]
                                                
{'debug/num_tok_total': 2845.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2845.0, 'debug/num_lat_loss': 1773.0, 'epoch': 0.77}

  8%|▊         | 56/730 [05:05<59:39,  5.31s/it]
                                                
{'train/ce_loss': 5.625, 'train/diffusion_loss': 0.382179856300354, 'epoch': 0.77}

  8%|▊         | 56/730 [05:05<59:39,  5.31s/it]
                                                
{'train/learning_rate_real': 2.4858013459545352e-05, 'epoch': 0.77}

  8%|▊         | 56/730 [05:05<59:39,  5.31s/it]
                                                
{'debug/num_tok_total': 3123.0, 'debug/num_tok_loss': 1808.0, 'debug/num_lat_total': 3123.0, 'debug/num_lat_loss': 1808.0, 'epoch': 0.77}

  8%|▊         | 56/730 [05:07<59:39,  5.31s/it]
                                                
{'train/ce_loss': 5.625, 'train/diffusion_loss': 0.34347909688949585, 'epoch': 0.77}

  8%|▊         | 56/730 [05:07<59:39,  5.31s/it]
                                                
{'train/learning_rate_real': 2.4858013459545352e-05, 'epoch': 0.77}

  8%|▊         | 56/730 [05:07<59:39,  5.31s/it]
  8%|▊         | 57/730 [05:07<1:00:41,  5.41s/it]
                                                  
{'debug/num_tok_total': 2622.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 2622.0, 'debug/num_lat_loss': 1769.0, 'epoch': 0.78}

  8%|▊         | 57/730 [05:08<1:00:41,  5.41s/it]
                                                  
{'train/ce_loss': 5.9375, 'train/diffusion_loss': 0.4226444661617279, 'epoch': 0.78}

  8%|▊         | 57/730 [05:08<1:00:41,  5.41s/it]
                                                  
{'train/learning_rate_real': 2.4849555524460277e-05, 'epoch': 0.78}

  8%|▊         | 57/730 [05:08<1:00:41,  5.41s/it]
                                                  
{'debug/num_tok_total': 3032.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 3032.0, 'debug/num_lat_loss': 1770.0, 'epoch': 0.78}

  8%|▊         | 57/730 [05:10<1:00:41,  5.41s/it]
                                                  
{'train/ce_loss': 5.5625, 'train/diffusion_loss': 0.31934428215026855, 'epoch': 0.78}

  8%|▊         | 57/730 [05:10<1:00:41,  5.41s/it]
                                                  
{'train/learning_rate_real': 2.4849555524460277e-05, 'epoch': 0.78}

  8%|▊         | 57/730 [05:10<1:00:41,  5.41s/it]
                                                  
{'debug/num_tok_total': 2503.0, 'debug/num_tok_loss': 1709.0, 'debug/num_lat_total': 2503.0, 'debug/num_lat_loss': 1709.0, 'epoch': 0.78}

  8%|▊         | 57/730 [05:11<1:00:41,  5.41s/it]
                                                  
{'train/ce_loss': 5.625, 'train/diffusion_loss': 0.4243476390838623, 'epoch': 0.78}

  8%|▊         | 57/730 [05:11<1:00:41,  5.41s/it]
                                                  
{'train/learning_rate_real': 2.4849555524460277e-05, 'epoch': 0.78}

  8%|▊         | 57/730 [05:11<1:00:41,  5.41s/it]
                                                  
{'debug/num_tok_total': 2298.0, 'debug/num_tok_loss': 1664.0, 'debug/num_lat_total': 2298.0, 'debug/num_lat_loss': 1664.0, 'epoch': 0.78}

  8%|▊         | 57/730 [05:12<1:00:41,  5.41s/it]
                                                  
{'train/ce_loss': 5.4375, 'train/diffusion_loss': 0.46369606256484985, 'epoch': 0.78}

  8%|▊         | 57/730 [05:12<1:00:41,  5.41s/it]
                                                  
{'train/learning_rate_real': 2.4849555524460277e-05, 'epoch': 0.78}

  8%|▊         | 57/730 [05:12<1:00:41,  5.41s/it]
  8%|▊         | 58/730 [05:13<1:00:42,  5.42s/it]
                                                  
{'debug/num_tok_total': 2694.0, 'debug/num_tok_loss': 1589.0, 'debug/num_lat_total': 2694.0, 'debug/num_lat_loss': 1589.0, 'epoch': 0.79}

  8%|▊         | 58/730 [05:14<1:00:42,  5.42s/it]
                                                  
{'train/ce_loss': 5.71875, 'train/diffusion_loss': 0.33193787932395935, 'epoch': 0.79}

  8%|▊         | 58/730 [05:14<1:00:42,  5.42s/it]
                                                  
{'train/learning_rate_real': 2.4840854434016808e-05, 'epoch': 0.79}

  8%|▊         | 58/730 [05:14<1:00:42,  5.42s/it]
                                                  
{'debug/num_tok_total': 2214.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2214.0, 'debug/num_lat_loss': 1790.0, 'epoch': 0.79}

  8%|▊         | 58/730 [05:15<1:00:42,  5.42s/it]
                                                  
{'train/ce_loss': 5.40625, 'train/diffusion_loss': 0.4997442066669464, 'epoch': 0.79}

  8%|▊         | 58/730 [05:15<1:00:42,  5.42s/it]
                                                  
{'train/learning_rate_real': 2.4840854434016808e-05, 'epoch': 0.79}

  8%|▊         | 58/730 [05:15<1:00:42,  5.42s/it]
                                                  
{'debug/num_tok_total': 2435.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2435.0, 'debug/num_lat_loss': 1789.0, 'epoch': 0.79}

  8%|▊         | 58/730 [05:16<1:00:42,  5.42s/it]
                                                  
{'train/ce_loss': 5.5, 'train/diffusion_loss': 0.44154196977615356, 'epoch': 0.79}

  8%|▊         | 58/730 [05:16<1:00:42,  5.42s/it]
                                                  
{'train/learning_rate_real': 2.4840854434016808e-05, 'epoch': 0.79}

  8%|▊         | 58/730 [05:16<1:00:42,  5.42s/it]
                                                  
{'debug/num_tok_total': 2635.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2635.0, 'debug/num_lat_loss': 1784.0, 'epoch': 0.79}

  8%|▊         | 58/730 [05:17<1:00:42,  5.42s/it]
                                                  
{'train/ce_loss': 5.625, 'train/diffusion_loss': 0.42629536986351013, 'epoch': 0.79}

  8%|▊         | 58/730 [05:17<1:00:42,  5.42s/it]
                                                  
{'train/learning_rate_real': 2.4840854434016808e-05, 'epoch': 0.79}

  8%|▊         | 58/730 [05:17<1:00:42,  5.42s/it]
  8%|▊         | 59/730 [05:18<59:53,  5.35s/it]  
                                                
{'debug/num_tok_total': 2627.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2627.0, 'debug/num_lat_loss': 1766.0, 'epoch': 0.81}

  8%|▊         | 59/730 [05:19<59:53,  5.35s/it]
                                                
{'train/ce_loss': 5.46875, 'train/diffusion_loss': 0.4058604836463928, 'epoch': 0.81}

  8%|▊         | 59/730 [05:19<59:53,  5.35s/it]
                                                
{'train/learning_rate_real': 2.4831910359534216e-05, 'epoch': 0.81}

  8%|▊         | 59/730 [05:19<59:53,  5.35s/it]
                                                
{'debug/num_tok_total': 2643.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2643.0, 'debug/num_lat_loss': 1787.0, 'epoch': 0.81}

  8%|▊         | 59/730 [05:20<59:53,  5.35s/it]
                                                
{'train/ce_loss': 5.53125, 'train/diffusion_loss': 0.42272061109542847, 'epoch': 0.81}

  8%|▊         | 59/730 [05:20<59:53,  5.35s/it]
                                                
{'train/learning_rate_real': 2.4831910359534216e-05, 'epoch': 0.81}

  8%|▊         | 59/730 [05:20<59:53,  5.35s/it]
                                                
{'debug/num_tok_total': 2473.0, 'debug/num_tok_loss': 1585.0, 'debug/num_lat_total': 2473.0, 'debug/num_lat_loss': 1585.0, 'epoch': 0.81}

  8%|▊         | 59/730 [05:22<59:53,  5.35s/it]
                                                
{'train/ce_loss': 5.71875, 'train/diffusion_loss': 0.40059351921081543, 'epoch': 0.81}

  8%|▊         | 59/730 [05:22<59:53,  5.35s/it]
                                                
{'train/learning_rate_real': 2.4831910359534216e-05, 'epoch': 0.81}

  8%|▊         | 59/730 [05:22<59:53,  5.35s/it]
                                                
{'debug/num_tok_total': 2440.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2440.0, 'debug/num_lat_loss': 1792.0, 'epoch': 0.81}

  8%|▊         | 59/730 [05:23<59:53,  5.35s/it]
                                                
{'train/ce_loss': 5.3125, 'train/diffusion_loss': 0.4610413908958435, 'epoch': 0.81}

  8%|▊         | 59/730 [05:23<59:53,  5.35s/it]
                                                
{'train/learning_rate_real': 2.4831910359534216e-05, 'epoch': 0.81}

  8%|▊         | 59/730 [05:23<59:53,  5.35s/it]03/16/2026 06:52:55 - INFO - __main__ - LoRA debug step 60: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

  8%|▊         | 60/730 [05:23<59:54,  5.37s/it]
                                                
{'loss': 3.2409, 'grad_norm': 1.708295464515686, 'learning_rate': 2.4831910359534216e-05, 'epoch': 0.82}

  8%|▊         | 60/730 [05:23<59:54,  5.37s/it]
                                                
{'debug/num_tok_total': 2527.0, 'debug/num_tok_loss': 1725.0, 'debug/num_lat_total': 2527.0, 'debug/num_lat_loss': 1725.0, 'epoch': 0.82}

  8%|▊         | 60/730 [05:24<59:54,  5.37s/it]
                                                
{'train/ce_loss': 5.5625, 'train/diffusion_loss': 0.3908693790435791, 'epoch': 0.82}

  8%|▊         | 60/730 [05:24<59:54,  5.37s/it]
                                                
{'train/learning_rate_real': 2.4822723477115968e-05, 'epoch': 0.82}

  8%|▊         | 60/730 [05:24<59:54,  5.37s/it]
                                                
{'debug/num_tok_total': 2636.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2636.0, 'debug/num_lat_loss': 1779.0, 'epoch': 0.82}

  8%|▊         | 60/730 [05:26<59:54,  5.37s/it]
                                                
{'train/ce_loss': 5.53125, 'train/diffusion_loss': 0.42371171712875366, 'epoch': 0.82}

  8%|▊         | 60/730 [05:26<59:54,  5.37s/it]
                                                
{'train/learning_rate_real': 2.4822723477115968e-05, 'epoch': 0.82}

  8%|▊         | 60/730 [05:26<59:54,  5.37s/it]
                                                
{'debug/num_tok_total': 2581.0, 'debug/num_tok_loss': 1650.0, 'debug/num_lat_total': 2581.0, 'debug/num_lat_loss': 1650.0, 'epoch': 0.82}

  8%|▊         | 60/730 [05:27<59:54,  5.37s/it]
                                                
{'train/ce_loss': 5.625, 'train/diffusion_loss': 0.40264788269996643, 'epoch': 0.82}

  8%|▊         | 60/730 [05:27<59:54,  5.37s/it]
                                                
{'train/learning_rate_real': 2.4822723477115968e-05, 'epoch': 0.82}

  8%|▊         | 60/730 [05:27<59:54,  5.37s/it]
                                                
{'debug/num_tok_total': 2817.0, 'debug/num_tok_loss': 1750.0, 'debug/num_lat_total': 2817.0, 'debug/num_lat_loss': 1750.0, 'epoch': 0.82}

  8%|▊         | 60/730 [05:28<59:54,  5.37s/it]
                                                
{'train/ce_loss': 5.46875, 'train/diffusion_loss': 0.356502503156662, 'epoch': 0.82}

  8%|▊         | 60/730 [05:28<59:54,  5.37s/it]
                                                
{'train/learning_rate_real': 2.4822723477115968e-05, 'epoch': 0.82}

  8%|▊         | 60/730 [05:28<59:54,  5.37s/it]
  8%|▊         | 61/730 [05:29<59:54,  5.37s/it]
                                                
{'debug/num_tok_total': 3138.0, 'debug/num_tok_loss': 1814.0, 'debug/num_lat_total': 3138.0, 'debug/num_lat_loss': 1814.0, 'epoch': 0.84}

  8%|▊         | 61/730 [05:30<59:54,  5.37s/it]
                                                
{'train/ce_loss': 5.4375, 'train/diffusion_loss': 0.3347739279270172, 'epoch': 0.84}

  8%|▊         | 61/730 [05:30<59:54,  5.37s/it]
                                                
{'train/learning_rate_real': 2.481329396764629e-05, 'epoch': 0.84}

  8%|▊         | 61/730 [05:30<59:54,  5.37s/it]
                                                
{'debug/num_tok_total': 2611.0, 'debug/num_tok_loss': 1758.0, 'debug/num_lat_total': 2611.0, 'debug/num_lat_loss': 1758.0, 'epoch': 0.84}

  8%|▊         | 61/730 [05:31<59:54,  5.37s/it]
                                                
{'train/ce_loss': 5.40625, 'train/diffusion_loss': 0.4167521595954895, 'epoch': 0.84}

  8%|▊         | 61/730 [05:31<59:54,  5.37s/it]
                                                
{'train/learning_rate_real': 2.481329396764629e-05, 'epoch': 0.84}

  8%|▊         | 61/730 [05:31<59:54,  5.37s/it]
                                                
{'debug/num_tok_total': 2322.0, 'debug/num_tok_loss': 1663.0, 'debug/num_lat_total': 2322.0, 'debug/num_lat_loss': 1663.0, 'epoch': 0.84}

  8%|▊         | 61/730 [05:32<59:54,  5.37s/it]
                                                
{'train/ce_loss': 5.375, 'train/diffusion_loss': 0.4386259913444519, 'epoch': 0.84}

  8%|▊         | 61/730 [05:32<59:54,  5.37s/it]
                                                
{'train/learning_rate_real': 2.481329396764629e-05, 'epoch': 0.84}

  8%|▊         | 61/730 [05:32<59:54,  5.37s/it]
                                                
{'debug/num_tok_total': 2614.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2614.0, 'debug/num_lat_loss': 1774.0, 'epoch': 0.84}

  8%|▊         | 61/730 [05:33<59:54,  5.37s/it]
                                                
{'train/ce_loss': 5.3125, 'train/diffusion_loss': 0.4437485337257385, 'epoch': 0.84}

  8%|▊         | 61/730 [05:34<59:54,  5.37s/it]
                                                
{'train/learning_rate_real': 2.481329396764629e-05, 'epoch': 0.84}

  8%|▊         | 61/730 [05:34<59:54,  5.37s/it]
  8%|▊         | 62/730 [05:34<59:36,  5.35s/it]
                                                
{'debug/num_tok_total': 2430.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2430.0, 'debug/num_lat_loss': 1776.0, 'epoch': 0.85}

  8%|▊         | 62/730 [05:35<59:36,  5.35s/it]
                                                
{'train/ce_loss': 5.375, 'train/diffusion_loss': 0.44565480947494507, 'epoch': 0.85}

  8%|▊         | 62/730 [05:35<59:36,  5.35s/it]
                                                
{'train/learning_rate_real': 2.4803622016786578e-05, 'epoch': 0.85}

  8%|▊         | 62/730 [05:35<59:36,  5.35s/it]
                                                
{'debug/num_tok_total': 2550.0, 'debug/num_tok_loss': 1702.0, 'debug/num_lat_total': 2550.0, 'debug/num_lat_loss': 1702.0, 'epoch': 0.85}

  8%|▊         | 62/730 [05:36<59:36,  5.35s/it]
                                                
{'train/ce_loss': 5.625, 'train/diffusion_loss': 0.39768069982528687, 'epoch': 0.85}

  8%|▊         | 62/730 [05:36<59:36,  5.35s/it]
                                                
{'train/learning_rate_real': 2.4803622016786578e-05, 'epoch': 0.85}

  8%|▊         | 62/730 [05:36<59:36,  5.35s/it]
                                                
{'debug/num_tok_total': 2612.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2612.0, 'debug/num_lat_loss': 1772.0, 'epoch': 0.85}

  8%|▊         | 62/730 [05:37<59:36,  5.35s/it]
                                                
{'train/ce_loss': 5.8125, 'train/diffusion_loss': 0.4438689053058624, 'epoch': 0.85}

  8%|▊         | 62/730 [05:37<59:36,  5.35s/it]
                                                
{'train/learning_rate_real': 2.4803622016786578e-05, 'epoch': 0.85}

  8%|▊         | 62/730 [05:37<59:36,  5.35s/it]
                                                
{'debug/num_tok_total': 2835.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2835.0, 'debug/num_lat_loss': 1763.0, 'epoch': 0.85}

  8%|▊         | 62/730 [05:39<59:36,  5.35s/it]
                                                
{'train/ce_loss': 5.3125, 'train/diffusion_loss': 0.4113655686378479, 'epoch': 0.85}

  8%|▊         | 62/730 [05:39<59:36,  5.35s/it]
                                                
{'train/learning_rate_real': 2.4803622016786578e-05, 'epoch': 0.85}

  8%|▊         | 62/730 [05:39<59:36,  5.35s/it]
  9%|▊         | 63/730 [05:39<59:04,  5.31s/it]
                                                
{'debug/num_tok_total': 2392.0, 'debug/num_tok_loss': 1756.0, 'debug/num_lat_total': 2392.0, 'debug/num_lat_loss': 1756.0, 'epoch': 0.86}

  9%|▊         | 63/730 [05:40<59:04,  5.31s/it]
                                                
{'train/ce_loss': 5.3125, 'train/diffusion_loss': 0.4595162272453308, 'epoch': 0.86}

  9%|▊         | 63/730 [05:40<59:04,  5.31s/it]
                                                
{'train/learning_rate_real': 2.479370781497175e-05, 'epoch': 0.86}

  9%|▊         | 63/730 [05:40<59:04,  5.31s/it]
                                                
{'debug/num_tok_total': 2799.0, 'debug/num_tok_loss': 1754.0, 'debug/num_lat_total': 2799.0, 'debug/num_lat_loss': 1754.0, 'epoch': 0.86}

  9%|▊         | 63/730 [05:42<59:04,  5.31s/it]
                                                
{'train/ce_loss': 5.375, 'train/diffusion_loss': 0.38731205463409424, 'epoch': 0.86}

  9%|▊         | 63/730 [05:42<59:04,  5.31s/it]
                                                
{'train/learning_rate_real': 2.479370781497175e-05, 'epoch': 0.86}

  9%|▊         | 63/730 [05:42<59:04,  5.31s/it]
                                                
{'debug/num_tok_total': 2842.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2842.0, 'debug/num_lat_loss': 1775.0, 'epoch': 0.86}

  9%|▊         | 63/730 [05:43<59:04,  5.31s/it]
                                                
{'train/ce_loss': 5.21875, 'train/diffusion_loss': 0.3685234487056732, 'epoch': 0.86}

  9%|▊         | 63/730 [05:43<59:04,  5.31s/it]
                                                
{'train/learning_rate_real': 2.479370781497175e-05, 'epoch': 0.86}

  9%|▊         | 63/730 [05:43<59:04,  5.31s/it]
                                                
{'debug/num_tok_total': 3082.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 3082.0, 'debug/num_lat_loss': 1786.0, 'epoch': 0.86}

  9%|▊         | 63/730 [05:44<59:04,  5.31s/it]
                                                
{'train/ce_loss': 5.375, 'train/diffusion_loss': 0.3305037319660187, 'epoch': 0.86}

  9%|▊         | 63/730 [05:44<59:04,  5.31s/it]
                                                
{'train/learning_rate_real': 2.479370781497175e-05, 'epoch': 0.86}

  9%|▊         | 63/730 [05:44<59:04,  5.31s/it]
  9%|▉         | 64/730 [05:45<59:39,  5.37s/it]
                                                
{'debug/num_tok_total': 2649.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2649.0, 'debug/num_lat_loss': 1787.0, 'epoch': 0.88}

  9%|▉         | 64/730 [05:46<59:39,  5.37s/it]
                                                
{'train/ce_loss': 5.21875, 'train/diffusion_loss': 0.41264399886131287, 'epoch': 0.88}

  9%|▉         | 64/730 [05:46<59:39,  5.37s/it]
                                                
{'train/learning_rate_real': 2.4783551557406515e-05, 'epoch': 0.88}

  9%|▉         | 64/730 [05:46<59:39,  5.37s/it]
                                                
{'debug/num_tok_total': 2663.0, 'debug/num_tok_loss': 1687.0, 'debug/num_lat_total': 2663.0, 'debug/num_lat_loss': 1687.0, 'epoch': 0.88}

  9%|▉         | 64/730 [05:47<59:39,  5.37s/it]
                                                
{'train/ce_loss': 5.21875, 'train/diffusion_loss': 0.36778780817985535, 'epoch': 0.88}

  9%|▉         | 64/730 [05:47<59:39,  5.37s/it]
                                                
{'train/learning_rate_real': 2.4783551557406515e-05, 'epoch': 0.88}

  9%|▉         | 64/730 [05:47<59:39,  5.37s/it]
                                                
{'debug/num_tok_total': 2437.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2437.0, 'debug/num_lat_loss': 1786.0, 'epoch': 0.88}

  9%|▉         | 64/730 [05:48<59:39,  5.37s/it]
                                                
{'train/ce_loss': 5.125, 'train/diffusion_loss': 0.45238378643989563, 'epoch': 0.88}

  9%|▉         | 64/730 [05:48<59:39,  5.37s/it]
                                                
{'train/learning_rate_real': 2.4783551557406515e-05, 'epoch': 0.88}

  9%|▉         | 64/730 [05:48<59:39,  5.37s/it]
                                                
{'debug/num_tok_total': 2450.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 2450.0, 'debug/num_lat_loss': 1801.0, 'epoch': 0.88}

  9%|▉         | 64/730 [05:50<59:39,  5.37s/it]
                                                
{'train/ce_loss': 5.09375, 'train/diffusion_loss': 0.46885961294174194, 'epoch': 0.88}

  9%|▉         | 64/730 [05:50<59:39,  5.37s/it]
                                                
{'train/learning_rate_real': 2.4783551557406515e-05, 'epoch': 0.88}

  9%|▉         | 64/730 [05:50<59:39,  5.37s/it]
  9%|▉         | 65/730 [05:50<59:28,  5.37s/it]
                                                
{'debug/num_tok_total': 2385.0, 'debug/num_tok_loss': 1750.0, 'debug/num_lat_total': 2385.0, 'debug/num_lat_loss': 1750.0, 'epoch': 0.89}

  9%|▉         | 65/730 [05:51<59:28,  5.37s/it]
                                                
{'train/ce_loss': 5.125, 'train/diffusion_loss': 0.45050710439682007, 'epoch': 0.89}

  9%|▉         | 65/730 [05:51<59:28,  5.37s/it]
                                                
{'train/learning_rate_real': 2.4773153444061502e-05, 'epoch': 0.89}

  9%|▉         | 65/730 [05:51<59:28,  5.37s/it]
                                                
{'debug/num_tok_total': 2855.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2855.0, 'debug/num_lat_loss': 1783.0, 'epoch': 0.89}

  9%|▉         | 65/730 [05:53<59:28,  5.37s/it]
                                                
{'train/ce_loss': 5.15625, 'train/diffusion_loss': 0.38604146242141724, 'epoch': 0.89}

  9%|▉         | 65/730 [05:53<59:28,  5.37s/it]
                                                
{'train/learning_rate_real': 2.4773153444061502e-05, 'epoch': 0.89}

  9%|▉         | 65/730 [05:53<59:28,  5.37s/it]
                                                
{'debug/num_tok_total': 2369.0, 'debug/num_tok_loss': 1548.0, 'debug/num_lat_total': 2369.0, 'debug/num_lat_loss': 1548.0, 'epoch': 0.89}

  9%|▉         | 65/730 [05:54<59:28,  5.37s/it]
                                                
{'train/ce_loss': 5.65625, 'train/diffusion_loss': 0.4263710379600525, 'epoch': 0.89}

  9%|▉         | 65/730 [05:54<59:28,  5.37s/it]
                                                
{'train/learning_rate_real': 2.4773153444061502e-05, 'epoch': 0.89}

  9%|▉         | 65/730 [05:54<59:28,  5.37s/it]
                                                
{'debug/num_tok_total': 3074.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 3074.0, 'debug/num_lat_loss': 1786.0, 'epoch': 0.89}

  9%|▉         | 65/730 [05:55<59:28,  5.37s/it]
                                                
{'train/ce_loss': 5.09375, 'train/diffusion_loss': 0.31766679883003235, 'epoch': 0.89}

  9%|▉         | 65/730 [05:55<59:28,  5.37s/it]
                                                
{'train/learning_rate_real': 2.4773153444061502e-05, 'epoch': 0.89}

  9%|▉         | 65/730 [05:55<59:28,  5.37s/it]
  9%|▉         | 66/730 [05:56<1:00:06,  5.43s/it]
                                                  
{'debug/num_tok_total': 2612.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2612.0, 'debug/num_lat_loss': 1766.0, 'epoch': 0.9}

  9%|▉         | 66/730 [05:57<1:00:06,  5.43s/it]
                                                  
{'train/ce_loss': 5.1875, 'train/diffusion_loss': 0.40880653262138367, 'epoch': 0.9}

  9%|▉         | 66/730 [05:57<1:00:06,  5.43s/it]
                                                  
{'train/learning_rate_real': 2.4762513679669348e-05, 'epoch': 0.9}

  9%|▉         | 66/730 [05:57<1:00:06,  5.43s/it]
                                                  
{'debug/num_tok_total': 2648.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2648.0, 'debug/num_lat_loss': 1784.0, 'epoch': 0.9}

  9%|▉         | 66/730 [05:58<1:00:06,  5.43s/it]
                                                  
{'train/ce_loss': 5.15625, 'train/diffusion_loss': 0.39718738198280334, 'epoch': 0.9}

  9%|▉         | 66/730 [05:58<1:00:06,  5.43s/it]
                                                  
{'train/learning_rate_real': 2.4762513679669348e-05, 'epoch': 0.9}

  9%|▉         | 66/730 [05:58<1:00:06,  5.43s/it]
                                                  
{'debug/num_tok_total': 2652.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2652.0, 'debug/num_lat_loss': 1789.0, 'epoch': 0.9}

  9%|▉         | 66/730 [05:59<1:00:06,  5.43s/it]
                                                  
{'train/ce_loss': 5.09375, 'train/diffusion_loss': 0.45373260974884033, 'epoch': 0.9}

  9%|▉         | 66/730 [05:59<1:00:06,  5.43s/it]
                                                  
{'train/learning_rate_real': 2.4762513679669348e-05, 'epoch': 0.9}

  9%|▉         | 66/730 [05:59<1:00:06,  5.43s/it]
                                                  
{'debug/num_tok_total': 2414.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 2414.0, 'debug/num_lat_loss': 1768.0, 'epoch': 0.9}

  9%|▉         | 66/730 [06:01<1:00:06,  5.43s/it]
                                                  
{'train/ce_loss': 5.53125, 'train/diffusion_loss': 0.4466804265975952, 'epoch': 0.9}

  9%|▉         | 66/730 [06:01<1:00:06,  5.43s/it]
                                                  
{'train/learning_rate_real': 2.4762513679669348e-05, 'epoch': 0.9}

  9%|▉         | 66/730 [06:01<1:00:06,  5.43s/it]
  9%|▉         | 67/730 [06:01<1:00:49,  5.50s/it]
                                                  
{'debug/num_tok_total': 3007.0, 'debug/num_tok_loss': 1745.0, 'debug/num_lat_total': 3007.0, 'debug/num_lat_loss': 1745.0, 'epoch': 0.92}

  9%|▉         | 67/730 [06:02<1:00:49,  5.50s/it]
                                                  
{'train/ce_loss': 5.125, 'train/diffusion_loss': 0.28842657804489136, 'epoch': 0.92}

  9%|▉         | 67/730 [06:03<1:00:49,  5.50s/it]
                                                  
{'train/learning_rate_real': 2.4751632473720638e-05, 'epoch': 0.92}

  9%|▉         | 67/730 [06:03<1:00:49,  5.50s/it]
                                                  
{'debug/num_tok_total': 2411.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 2411.0, 'debug/num_lat_loss': 1765.0, 'epoch': 0.92}

  9%|▉         | 67/730 [06:04<1:00:49,  5.50s/it]
                                                  
{'train/ce_loss': 5.15625, 'train/diffusion_loss': 0.4682995080947876, 'epoch': 0.92}

  9%|▉         | 67/730 [06:04<1:00:49,  5.50s/it]
                                                  
{'train/learning_rate_real': 2.4751632473720638e-05, 'epoch': 0.92}

  9%|▉         | 67/730 [06:04<1:00:49,  5.50s/it]
                                                  
{'debug/num_tok_total': 2453.0, 'debug/num_tok_loss': 1805.0, 'debug/num_lat_total': 2453.0, 'debug/num_lat_loss': 1805.0, 'epoch': 0.92}

  9%|▉         | 67/730 [06:05<1:00:49,  5.50s/it]
                                                  
{'train/ce_loss': 5.25, 'train/diffusion_loss': 0.48682138323783875, 'epoch': 0.92}

  9%|▉         | 67/730 [06:05<1:00:49,  5.50s/it]
                                                  
{'train/learning_rate_real': 2.4751632473720638e-05, 'epoch': 0.92}

  9%|▉         | 67/730 [06:05<1:00:49,  5.50s/it]
                                                  
{'debug/num_tok_total': 2531.0, 'debug/num_tok_loss': 1610.0, 'debug/num_lat_total': 2531.0, 'debug/num_lat_loss': 1610.0, 'epoch': 0.92}

  9%|▉         | 67/730 [06:06<1:00:49,  5.50s/it]
                                                  
{'train/ce_loss': 5.1875, 'train/diffusion_loss': 0.41317009925842285, 'epoch': 0.92}

  9%|▉         | 67/730 [06:06<1:00:49,  5.50s/it]
                                                  
{'train/learning_rate_real': 2.4751632473720638e-05, 'epoch': 0.92}

  9%|▉         | 67/730 [06:06<1:00:49,  5.50s/it]
  9%|▉         | 68/730 [06:07<1:00:03,  5.44s/it]
                                                  
{'debug/num_tok_total': 2605.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2605.0, 'debug/num_lat_loss': 1763.0, 'epoch': 0.93}

  9%|▉         | 68/730 [06:08<1:00:03,  5.44s/it]
                                                  
{'train/ce_loss': 5.28125, 'train/diffusion_loss': 0.4040594696998596, 'epoch': 0.93}

  9%|▉         | 68/730 [06:08<1:00:03,  5.44s/it]
                                                  
{'train/learning_rate_real': 2.474051004045981e-05, 'epoch': 0.93}

  9%|▉         | 68/730 [06:08<1:00:03,  5.44s/it]
                                                  
{'debug/num_tok_total': 2861.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2861.0, 'debug/num_lat_loss': 1786.0, 'epoch': 0.93}

  9%|▉         | 68/730 [06:09<1:00:03,  5.44s/it]
                                                  
{'train/ce_loss': 5.3125, 'train/diffusion_loss': 0.3841204047203064, 'epoch': 0.93}

  9%|▉         | 68/730 [06:09<1:00:03,  5.44s/it]
                                                  
{'train/learning_rate_real': 2.474051004045981e-05, 'epoch': 0.93}

  9%|▉         | 68/730 [06:09<1:00:03,  5.44s/it]
                                                  
{'debug/num_tok_total': 2451.0, 'debug/num_tok_loss': 1610.0, 'debug/num_lat_total': 2451.0, 'debug/num_lat_loss': 1610.0, 'epoch': 0.93}

  9%|▉         | 68/730 [06:10<1:00:03,  5.44s/it]
                                                  
{'train/ce_loss': 5.0625, 'train/diffusion_loss': 0.3975687026977539, 'epoch': 0.93}

  9%|▉         | 68/730 [06:10<1:00:03,  5.44s/it]
                                                  
{'train/learning_rate_real': 2.474051004045981e-05, 'epoch': 0.93}

  9%|▉         | 68/730 [06:10<1:00:03,  5.44s/it]
                                                  
{'debug/num_tok_total': 2410.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2410.0, 'debug/num_lat_loss': 1773.0, 'epoch': 0.93}

  9%|▉         | 68/730 [06:11<1:00:03,  5.44s/it]
                                                  
{'train/ce_loss': 5.0, 'train/diffusion_loss': 0.4634510278701782, 'epoch': 0.93}

  9%|▉         | 68/730 [06:11<1:00:03,  5.44s/it]
                                                  
{'train/learning_rate_real': 2.474051004045981e-05, 'epoch': 0.93}

  9%|▉         | 68/730 [06:11<1:00:03,  5.44s/it]
  9%|▉         | 69/730 [06:12<59:11,  5.37s/it]  
                                                
{'debug/num_tok_total': 2693.0, 'debug/num_tok_loss': 1617.0, 'debug/num_lat_total': 2693.0, 'debug/num_lat_loss': 1617.0, 'epoch': 0.95}

  9%|▉         | 69/730 [06:13<59:11,  5.37s/it]
                                                
{'train/ce_loss': 5.25, 'train/diffusion_loss': 0.35224559903144836, 'epoch': 0.95}

  9%|▉         | 69/730 [06:13<59:11,  5.37s/it]
                                                
{'train/learning_rate_real': 2.472914659888092e-05, 'epoch': 0.95}

  9%|▉         | 69/730 [06:13<59:11,  5.37s/it]
                                                
{'debug/num_tok_total': 2292.0, 'debug/num_tok_loss': 1657.0, 'debug/num_lat_total': 2292.0, 'debug/num_lat_loss': 1657.0, 'epoch': 0.95}

  9%|▉         | 69/730 [06:14<59:11,  5.37s/it]
                                                
{'train/ce_loss': 4.96875, 'train/diffusion_loss': 0.4498156011104584, 'epoch': 0.95}

  9%|▉         | 69/730 [06:14<59:11,  5.37s/it]
                                                
{'train/learning_rate_real': 2.472914659888092e-05, 'epoch': 0.95}

  9%|▉         | 69/730 [06:14<59:11,  5.37s/it]
                                                
{'debug/num_tok_total': 2024.0, 'debug/num_tok_loss': 1805.0, 'debug/num_lat_total': 2024.0, 'debug/num_lat_loss': 1805.0, 'epoch': 0.95}

  9%|▉         | 69/730 [06:15<59:11,  5.37s/it]
                                                
{'train/ce_loss': 4.96875, 'train/diffusion_loss': 0.5396973490715027, 'epoch': 0.95}

  9%|▉         | 69/730 [06:15<59:11,  5.37s/it]
                                                
{'train/learning_rate_real': 2.472914659888092e-05, 'epoch': 0.95}

  9%|▉         | 69/730 [06:15<59:11,  5.37s/it]
                                                
{'debug/num_tok_total': 2018.0, 'debug/num_tok_loss': 1592.0, 'debug/num_lat_total': 2018.0, 'debug/num_lat_loss': 1592.0, 'epoch': 0.95}

  9%|▉         | 69/730 [06:16<59:11,  5.37s/it]
                                                
{'train/ce_loss': 5.09375, 'train/diffusion_loss': 0.4777624309062958, 'epoch': 0.95}

  9%|▉         | 69/730 [06:16<59:11,  5.37s/it]
                                                
{'train/learning_rate_real': 2.472914659888092e-05, 'epoch': 0.95}

  9%|▉         | 69/730 [06:16<59:11,  5.37s/it]03/16/2026 06:53:49 - INFO - __main__ - LoRA debug step 70: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 10%|▉         | 70/730 [06:17<57:53,  5.26s/it]
                                                
{'loss': 3.1653, 'grad_norm': 1.7449594736099243, 'learning_rate': 2.472914659888092e-05, 'epoch': 0.96}

 10%|▉         | 70/730 [06:17<57:53,  5.26s/it]
                                                
{'debug/num_tok_total': 2412.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2412.0, 'debug/num_lat_loss': 1780.0, 'epoch': 0.96}

 10%|▉         | 70/730 [06:18<57:53,  5.26s/it]
                                                
{'train/ce_loss': 5.125, 'train/diffusion_loss': 0.4851177930831909, 'epoch': 0.96}

 10%|▉         | 70/730 [06:18<57:53,  5.26s/it]
                                                
{'train/learning_rate_real': 2.4717542372723333e-05, 'epoch': 0.96}

 10%|▉         | 70/730 [06:18<57:53,  5.26s/it]
                                                
{'debug/num_tok_total': 1879.0, 'debug/num_tok_loss': 1670.0, 'debug/num_lat_total': 1879.0, 'debug/num_lat_loss': 1670.0, 'epoch': 0.96}

 10%|▉         | 70/730 [06:19<57:53,  5.26s/it]
                                                
{'train/ce_loss': 5.1875, 'train/diffusion_loss': 0.5450473427772522, 'epoch': 0.96}

 10%|▉         | 70/730 [06:19<57:53,  5.26s/it]
                                                
{'train/learning_rate_real': 2.4717542372723333e-05, 'epoch': 0.96}

 10%|▉         | 70/730 [06:19<57:53,  5.26s/it]
                                                
{'debug/num_tok_total': 2656.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2656.0, 'debug/num_lat_loss': 1792.0, 'epoch': 0.96}

 10%|▉         | 70/730 [06:20<57:53,  5.26s/it]
                                                
{'train/ce_loss': 5.15625, 'train/diffusion_loss': 0.40747514367103577, 'epoch': 0.96}

 10%|▉         | 70/730 [06:20<57:53,  5.26s/it]
                                                
{'train/learning_rate_real': 2.4717542372723333e-05, 'epoch': 0.96}

 10%|▉         | 70/730 [06:20<57:53,  5.26s/it]
                                                
{'debug/num_tok_total': 2617.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2617.0, 'debug/num_lat_loss': 1780.0, 'epoch': 0.96}

 10%|▉         | 70/730 [06:21<57:53,  5.26s/it]
                                                
{'train/ce_loss': 5.0, 'train/diffusion_loss': 0.41900011897087097, 'epoch': 0.96}

 10%|▉         | 70/730 [06:21<57:53,  5.26s/it]
                                                
{'train/learning_rate_real': 2.4717542372723333e-05, 'epoch': 0.96}

 10%|▉         | 70/730 [06:21<57:53,  5.26s/it]
 10%|▉         | 71/730 [06:22<56:55,  5.18s/it]
                                                
{'debug/num_tok_total': 2627.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 2627.0, 'debug/num_lat_loss': 1768.0, 'epoch': 0.97}

 10%|▉         | 71/730 [06:23<56:55,  5.18s/it]
                                                
{'train/ce_loss': 5.625, 'train/diffusion_loss': 0.40764084458351135, 'epoch': 0.97}

 10%|▉         | 71/730 [06:23<56:55,  5.18s/it]
                                                
{'train/learning_rate_real': 2.470569759046732e-05, 'epoch': 0.97}

 10%|▉         | 71/730 [06:23<56:55,  5.18s/it]
                                                
{'debug/num_tok_total': 2681.0, 'debug/num_tok_loss': 1807.0, 'debug/num_lat_total': 2681.0, 'debug/num_lat_loss': 1807.0, 'epoch': 0.97}

 10%|▉         | 71/730 [06:24<56:55,  5.18s/it]
                                                
{'train/ce_loss': 5.03125, 'train/diffusion_loss': 0.4096241891384125, 'epoch': 0.97}

 10%|▉         | 71/730 [06:24<56:55,  5.18s/it]
                                                
{'train/learning_rate_real': 2.470569759046732e-05, 'epoch': 0.97}

 10%|▉         | 71/730 [06:24<56:55,  5.18s/it]
                                                
{'debug/num_tok_total': 2827.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2827.0, 'debug/num_lat_loss': 1772.0, 'epoch': 0.97}

 10%|▉         | 71/730 [06:25<56:55,  5.18s/it]
                                                
{'train/ce_loss': 4.9375, 'train/diffusion_loss': 0.3634895980358124, 'epoch': 0.97}

 10%|▉         | 71/730 [06:25<56:55,  5.18s/it]
                                                
{'train/learning_rate_real': 2.470569759046732e-05, 'epoch': 0.97}

 10%|▉         | 71/730 [06:25<56:55,  5.18s/it]
                                                
{'debug/num_tok_total': 2843.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 2843.0, 'debug/num_lat_loss': 1768.0, 'epoch': 0.97}

 10%|▉         | 71/730 [06:27<56:55,  5.18s/it]
                                                
{'train/ce_loss': 4.90625, 'train/diffusion_loss': 0.3556757867336273, 'epoch': 0.97}

 10%|▉         | 71/730 [06:27<56:55,  5.18s/it]
                                                
{'train/learning_rate_real': 2.470569759046732e-05, 'epoch': 0.97}

 10%|▉         | 71/730 [06:27<56:55,  5.18s/it]
 10%|▉         | 72/730 [06:27<57:57,  5.28s/it]
                                                
{'debug/num_tok_total': 2877.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2877.0, 'debug/num_lat_loss': 1794.0, 'epoch': 0.99}

 10%|▉         | 72/730 [06:28<57:57,  5.28s/it]
                                                
{'train/ce_loss': 5.0625, 'train/diffusion_loss': 0.39283737540245056, 'epoch': 0.99}

 10%|▉         | 72/730 [06:28<57:57,  5.28s/it]
                                                
{'train/learning_rate_real': 2.469361248532955e-05, 'epoch': 0.99}

 10%|▉         | 72/730 [06:28<57:57,  5.28s/it]
                                                
{'debug/num_tok_total': 3048.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 3048.0, 'debug/num_lat_loss': 1770.0, 'epoch': 0.99}

 10%|▉         | 72/730 [06:30<57:57,  5.28s/it]
                                                
{'train/ce_loss': 5.09375, 'train/diffusion_loss': 0.33008840680122375, 'epoch': 0.99}

 10%|▉         | 72/730 [06:30<57:57,  5.28s/it]
                                                
{'train/learning_rate_real': 2.469361248532955e-05, 'epoch': 0.99}

 10%|▉         | 72/730 [06:30<57:57,  5.28s/it]
                                                
{'debug/num_tok_total': 2879.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2879.0, 'debug/num_lat_loss': 1794.0, 'epoch': 0.99}

 10%|▉         | 72/730 [06:31<57:57,  5.28s/it]
                                                
{'train/ce_loss': 4.9375, 'train/diffusion_loss': 0.37761390209198, 'epoch': 0.99}

 10%|▉         | 72/730 [06:31<57:57,  5.28s/it]
                                                
{'train/learning_rate_real': 2.469361248532955e-05, 'epoch': 0.99}

 10%|▉         | 72/730 [06:31<57:57,  5.28s/it]
                                                
{'debug/num_tok_total': 662.0, 'debug/num_tok_loss': 445.0, 'debug/num_lat_total': 662.0, 'debug/num_lat_loss': 445.0, 'epoch': 0.99}

 10%|▉         | 72/730 [06:32<57:57,  5.28s/it]
                                                
{'train/ce_loss': 7.09375, 'train/diffusion_loss': 0.5194283723831177, 'epoch': 0.99}

 10%|▉         | 72/730 [06:32<57:57,  5.28s/it]
                                                
{'train/learning_rate_real': 2.469361248532955e-05, 'epoch': 0.99}

 10%|▉         | 72/730 [06:32<57:57,  5.28s/it]
 10%|█         | 73/730 [06:32<55:23,  5.06s/it]
                                                
{'debug/num_tok_total': 2690.0, 'debug/num_tok_loss': 1812.0, 'debug/num_lat_total': 2690.0, 'debug/num_lat_loss': 1812.0, 'epoch': 1.0}

 10%|█         | 73/730 [06:33<55:23,  5.06s/it]
                                                
{'train/ce_loss': 4.90625, 'train/diffusion_loss': 0.4245181083679199, 'epoch': 1.0}

 10%|█         | 73/730 [06:33<55:23,  5.06s/it]
                                                
{'train/learning_rate_real': 2.4681287295258516e-05, 'epoch': 1.0}

 10%|█         | 73/730 [06:33<55:23,  5.06s/it]
                                                
{'debug/num_tok_total': 2656.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 2656.0, 'debug/num_lat_loss': 1801.0, 'epoch': 1.0}

 10%|█         | 73/730 [06:34<55:23,  5.06s/it]
                                                
{'train/ce_loss': 4.96875, 'train/diffusion_loss': 0.4211677014827728, 'epoch': 1.0}

 10%|█         | 73/730 [06:34<55:23,  5.06s/it]
                                                
{'train/learning_rate_real': 2.4681287295258516e-05, 'epoch': 1.0}

 10%|█         | 73/730 [06:34<55:23,  5.06s/it]
                                                
{'debug/num_tok_total': 2018.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 2018.0, 'debug/num_lat_loss': 1797.0, 'epoch': 1.0}

 10%|█         | 73/730 [06:35<55:23,  5.06s/it]
                                                
{'train/ce_loss': 5.125, 'train/diffusion_loss': 0.5452603697776794, 'epoch': 1.0}

 10%|█         | 73/730 [06:35<55:23,  5.06s/it]
                                                
{'train/learning_rate_real': 2.4681287295258516e-05, 'epoch': 1.0}

 10%|█         | 73/730 [06:35<55:23,  5.06s/it]
                                                
{'debug/num_tok_total': 2644.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2644.0, 'debug/num_lat_loss': 1781.0, 'epoch': 1.0}

 10%|█         | 73/730 [06:37<55:23,  5.06s/it]
                                                
{'train/ce_loss': 5.125, 'train/diffusion_loss': 0.39598238468170166, 'epoch': 1.0}

 10%|█         | 73/730 [06:37<55:23,  5.06s/it]
                                                
{'train/learning_rate_real': 2.4681287295258516e-05, 'epoch': 1.0}

 10%|█         | 73/730 [06:37<55:23,  5.06s/it]
 10%|█         | 74/730 [06:37<55:47,  5.10s/it]
                                                
{'debug/num_tok_total': 2624.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2624.0, 'debug/num_lat_loss': 1771.0, 'epoch': 1.01}

 10%|█         | 74/730 [06:38<55:47,  5.10s/it]
                                                
{'train/ce_loss': 4.90625, 'train/diffusion_loss': 0.41032910346984863, 'epoch': 1.01}

 10%|█         | 74/730 [06:38<55:47,  5.10s/it]
                                                
{'train/learning_rate_real': 2.466872226292983e-05, 'epoch': 1.01}

 10%|█         | 74/730 [06:38<55:47,  5.10s/it]
                                                
{'debug/num_tok_total': 2220.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2220.0, 'debug/num_lat_loss': 1783.0, 'epoch': 1.01}

 10%|█         | 74/730 [06:39<55:47,  5.10s/it]
                                                
{'train/ce_loss': 4.875, 'train/diffusion_loss': 0.5221177935600281, 'epoch': 1.01}

 10%|█         | 74/730 [06:39<55:47,  5.10s/it]
                                                
{'train/learning_rate_real': 2.466872226292983e-05, 'epoch': 1.01}

 10%|█         | 74/730 [06:39<55:47,  5.10s/it]
                                                
{'debug/num_tok_total': 2661.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2661.0, 'debug/num_lat_loss': 1795.0, 'epoch': 1.01}

 10%|█         | 74/730 [06:41<55:47,  5.10s/it]
                                                
{'train/ce_loss': 5.0, 'train/diffusion_loss': 0.4044923782348633, 'epoch': 1.01}

 10%|█         | 74/730 [06:41<55:47,  5.10s/it]
                                                
{'train/learning_rate_real': 2.466872226292983e-05, 'epoch': 1.01}

 10%|█         | 74/730 [06:41<55:47,  5.10s/it]
                                                
{'debug/num_tok_total': 2644.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2644.0, 'debug/num_lat_loss': 1782.0, 'epoch': 1.01}

 10%|█         | 74/730 [06:42<55:47,  5.10s/it]
                                                
{'train/ce_loss': 5.4375, 'train/diffusion_loss': 0.421188086271286, 'epoch': 1.01}

 10%|█         | 74/730 [06:42<55:47,  5.10s/it]
                                                
{'train/learning_rate_real': 2.466872226292983e-05, 'epoch': 1.01}

 10%|█         | 74/730 [06:42<55:47,  5.10s/it]
 10%|█         | 75/730 [06:42<56:31,  5.18s/it]
                                                
{'debug/num_tok_total': 3166.0, 'debug/num_tok_loss': 1657.0, 'debug/num_lat_total': 3166.0, 'debug/num_lat_loss': 1657.0, 'epoch': 1.03}

 10%|█         | 75/730 [06:44<56:31,  5.18s/it]
                                                
{'train/ce_loss': 4.9375, 'train/diffusion_loss': 0.2653091251850128, 'epoch': 1.03}

 10%|█         | 75/730 [06:44<56:31,  5.18s/it]
                                                
{'train/learning_rate_real': 2.4655917635741472e-05, 'epoch': 1.03}

 10%|█         | 75/730 [06:44<56:31,  5.18s/it]
                                                
{'debug/num_tok_total': 2254.0, 'debug/num_tok_loss': 1806.0, 'debug/num_lat_total': 2254.0, 'debug/num_lat_loss': 1806.0, 'epoch': 1.03}

 10%|█         | 75/730 [06:45<56:31,  5.18s/it]
                                                
{'train/ce_loss': 5.0, 'train/diffusion_loss': 0.48585981130599976, 'epoch': 1.03}

 10%|█         | 75/730 [06:45<56:31,  5.18s/it]
                                                
{'train/learning_rate_real': 2.4655917635741472e-05, 'epoch': 1.03}

 10%|█         | 75/730 [06:45<56:31,  5.18s/it]
                                                
{'debug/num_tok_total': 2675.0, 'debug/num_tok_loss': 1808.0, 'debug/num_lat_total': 2675.0, 'debug/num_lat_loss': 1808.0, 'epoch': 1.03}

 10%|█         | 75/730 [06:46<56:31,  5.18s/it]
                                                
{'train/ce_loss': 4.90625, 'train/diffusion_loss': 0.44348567724227905, 'epoch': 1.03}

 10%|█         | 75/730 [06:46<56:31,  5.18s/it]
                                                
{'train/learning_rate_real': 2.4655917635741472e-05, 'epoch': 1.03}

 10%|█         | 75/730 [06:46<56:31,  5.18s/it]
                                                
{'debug/num_tok_total': 2207.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2207.0, 'debug/num_lat_loss': 1773.0, 'epoch': 1.03}

 10%|█         | 75/730 [06:47<56:31,  5.18s/it]
                                                
{'train/ce_loss': 5.03125, 'train/diffusion_loss': 0.4878855049610138, 'epoch': 1.03}

 10%|█         | 75/730 [06:47<56:31,  5.18s/it]
                                                
{'train/learning_rate_real': 2.4655917635741472e-05, 'epoch': 1.03}

 10%|█         | 75/730 [06:47<56:31,  5.18s/it]
 10%|█         | 76/730 [06:48<56:22,  5.17s/it]
                                                
{'debug/num_tok_total': 2690.0, 'debug/num_tok_loss': 1804.0, 'debug/num_lat_total': 2690.0, 'debug/num_lat_loss': 1804.0, 'epoch': 1.04}

 10%|█         | 76/730 [06:49<56:22,  5.17s/it]
                                                
{'train/ce_loss': 5.0, 'train/diffusion_loss': 0.4145924746990204, 'epoch': 1.04}

 10%|█         | 76/730 [06:49<56:22,  5.17s/it]
                                                
{'train/learning_rate_real': 2.4642873665808873e-05, 'epoch': 1.04}

 10%|█         | 76/730 [06:49<56:22,  5.17s/it]
                                                
{'debug/num_tok_total': 2383.0, 'debug/num_tok_loss': 1747.0, 'debug/num_lat_total': 2383.0, 'debug/num_lat_loss': 1747.0, 'epoch': 1.04}

 10%|█         | 76/730 [06:50<56:22,  5.17s/it]
                                                
{'train/ce_loss': 5.0, 'train/diffusion_loss': 0.49800702929496765, 'epoch': 1.04}

 10%|█         | 76/730 [06:50<56:22,  5.17s/it]
                                                
{'train/learning_rate_real': 2.4642873665808873e-05, 'epoch': 1.04}

 10%|█         | 76/730 [06:50<56:22,  5.17s/it]
                                                
{'debug/num_tok_total': 2667.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2667.0, 'debug/num_lat_loss': 1799.0, 'epoch': 1.04}

 10%|█         | 76/730 [06:51<56:22,  5.17s/it]
                                                
{'train/ce_loss': 4.84375, 'train/diffusion_loss': 0.4373202919960022, 'epoch': 1.04}

 10%|█         | 76/730 [06:51<56:22,  5.17s/it]
                                                
{'train/learning_rate_real': 2.4642873665808873e-05, 'epoch': 1.04}

 10%|█         | 76/730 [06:51<56:22,  5.17s/it]
                                                
{'debug/num_tok_total': 2857.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2857.0, 'debug/num_lat_loss': 1783.0, 'epoch': 1.04}

 10%|█         | 76/730 [06:52<56:22,  5.17s/it]
                                                
{'train/ce_loss': 4.9375, 'train/diffusion_loss': 0.3920494318008423, 'epoch': 1.04}

 10%|█         | 76/730 [06:52<56:22,  5.17s/it]
                                                
{'train/learning_rate_real': 2.4642873665808873e-05, 'epoch': 1.04}

 10%|█         | 76/730 [06:52<56:22,  5.17s/it]
 11%|█         | 77/730 [06:53<56:55,  5.23s/it]
                                                
{'debug/num_tok_total': 2656.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2656.0, 'debug/num_lat_loss': 1795.0, 'epoch': 1.05}

 11%|█         | 77/730 [06:54<56:55,  5.23s/it]
                                                
{'train/ce_loss': 4.96875, 'train/diffusion_loss': 0.39454302191734314, 'epoch': 1.05}

 11%|█         | 77/730 [06:54<56:55,  5.23s/it]
                                                
{'train/learning_rate_real': 2.462959060996001e-05, 'epoch': 1.05}

 11%|█         | 77/730 [06:54<56:55,  5.23s/it]
                                                
{'debug/num_tok_total': 3066.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 3066.0, 'debug/num_lat_loss': 1774.0, 'epoch': 1.05}

 11%|█         | 77/730 [06:55<56:55,  5.23s/it]
                                                
{'train/ce_loss': 5.21875, 'train/diffusion_loss': 0.3013768494129181, 'epoch': 1.05}

 11%|█         | 77/730 [06:55<56:55,  5.23s/it]
                                                
{'train/learning_rate_real': 2.462959060996001e-05, 'epoch': 1.05}

 11%|█         | 77/730 [06:55<56:55,  5.23s/it]
                                                
{'debug/num_tok_total': 2032.0, 'debug/num_tok_loss': 1811.0, 'debug/num_lat_total': 2032.0, 'debug/num_lat_loss': 1811.0, 'epoch': 1.05}

 11%|█         | 77/730 [06:56<56:55,  5.23s/it]
                                                
{'train/ce_loss': 5.0, 'train/diffusion_loss': 0.5480638146400452, 'epoch': 1.05}

 11%|█         | 77/730 [06:56<56:55,  5.23s/it]
                                                
{'train/learning_rate_real': 2.462959060996001e-05, 'epoch': 1.05}

 11%|█         | 77/730 [06:56<56:55,  5.23s/it]
                                                
{'debug/num_tok_total': 2650.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2650.0, 'debug/num_lat_loss': 1783.0, 'epoch': 1.05}

 11%|█         | 77/730 [06:58<56:55,  5.23s/it]
                                                
{'train/ce_loss': 4.6875, 'train/diffusion_loss': 0.4419059753417969, 'epoch': 1.05}

 11%|█         | 77/730 [06:58<56:55,  5.23s/it]
                                                
{'train/learning_rate_real': 2.462959060996001e-05, 'epoch': 1.05}

 11%|█         | 77/730 [06:58<56:55,  5.23s/it]
 11%|█         | 78/730 [06:58<56:48,  5.23s/it]
                                                
{'debug/num_tok_total': 3096.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 3096.0, 'debug/num_lat_loss': 1790.0, 'epoch': 1.07}

 11%|█         | 78/730 [06:59<56:48,  5.23s/it]
                                                
{'train/ce_loss': 4.84375, 'train/diffusion_loss': 0.3151288628578186, 'epoch': 1.07}

 11%|█         | 78/730 [06:59<56:48,  5.23s/it]
                                                
{'train/learning_rate_real': 2.4616068729730294e-05, 'epoch': 1.07}

 11%|█         | 78/730 [06:59<56:48,  5.23s/it]
                                                
{'debug/num_tok_total': 2860.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2860.0, 'debug/num_lat_loss': 1782.0, 'epoch': 1.07}

 11%|█         | 78/730 [07:01<56:48,  5.23s/it]
                                                
{'train/ce_loss': 4.8125, 'train/diffusion_loss': 0.3643770217895508, 'epoch': 1.07}

 11%|█         | 78/730 [07:01<56:48,  5.23s/it]
                                                
{'train/learning_rate_real': 2.4616068729730294e-05, 'epoch': 1.07}

 11%|█         | 78/730 [07:01<56:48,  5.23s/it]
                                                
{'debug/num_tok_total': 2453.0, 'debug/num_tok_loss': 1610.0, 'debug/num_lat_total': 2453.0, 'debug/num_lat_loss': 1610.0, 'epoch': 1.07}

 11%|█         | 78/730 [07:02<56:48,  5.23s/it]
                                                
{'train/ce_loss': 4.9375, 'train/diffusion_loss': 0.3915770947933197, 'epoch': 1.07}

 11%|█         | 78/730 [07:02<56:48,  5.23s/it]
                                                
{'train/learning_rate_real': 2.4616068729730294e-05, 'epoch': 1.07}

 11%|█         | 78/730 [07:02<56:48,  5.23s/it]
                                                
{'debug/num_tok_total': 2852.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2852.0, 'debug/num_lat_loss': 1776.0, 'epoch': 1.07}

 11%|█         | 78/730 [07:03<56:48,  5.23s/it]
                                                
{'train/ce_loss': 4.96875, 'train/diffusion_loss': 0.39558202028274536, 'epoch': 1.07}

 11%|█         | 78/730 [07:03<56:48,  5.23s/it]
                                                
{'train/learning_rate_real': 2.4616068729730294e-05, 'epoch': 1.07}

 11%|█         | 78/730 [07:03<56:48,  5.23s/it]
 11%|█         | 79/730 [07:04<57:52,  5.33s/it]
                                                
{'debug/num_tok_total': 2137.0, 'debug/num_tok_loss': 1709.0, 'debug/num_lat_total': 2137.0, 'debug/num_lat_loss': 1709.0, 'epoch': 1.08}

 11%|█         | 79/730 [07:05<57:52,  5.33s/it]
                                                
{'train/ce_loss': 4.875, 'train/diffusion_loss': 0.4808519780635834, 'epoch': 1.08}

 11%|█         | 79/730 [07:05<57:52,  5.33s/it]
                                                
{'train/learning_rate_real': 2.460230829135746e-05, 'epoch': 1.08}

 11%|█         | 79/730 [07:05<57:52,  5.33s/it]
                                                
{'debug/num_tok_total': 2234.0, 'debug/num_tok_loss': 1698.0, 'debug/num_lat_total': 2234.0, 'debug/num_lat_loss': 1698.0, 'epoch': 1.08}

 11%|█         | 79/730 [07:06<57:52,  5.33s/it]
                                                
{'train/ce_loss': 4.84375, 'train/diffusion_loss': 0.487990140914917, 'epoch': 1.08}

 11%|█         | 79/730 [07:06<57:52,  5.33s/it]
                                                
{'train/learning_rate_real': 2.460230829135746e-05, 'epoch': 1.08}

 11%|█         | 79/730 [07:06<57:52,  5.33s/it]
                                                
{'debug/num_tok_total': 2467.0, 'debug/num_tok_loss': 1800.0, 'debug/num_lat_total': 2467.0, 'debug/num_lat_loss': 1800.0, 'epoch': 1.08}

 11%|█         | 79/730 [07:07<57:52,  5.33s/it]
                                                
{'train/ce_loss': 4.8125, 'train/diffusion_loss': 0.4461616575717926, 'epoch': 1.08}

 11%|█         | 79/730 [07:07<57:52,  5.33s/it]
                                                
{'train/learning_rate_real': 2.460230829135746e-05, 'epoch': 1.08}

 11%|█         | 79/730 [07:07<57:52,  5.33s/it]
                                                
{'debug/num_tok_total': 2677.0, 'debug/num_tok_loss': 1800.0, 'debug/num_lat_total': 2677.0, 'debug/num_lat_loss': 1800.0, 'epoch': 1.08}

 11%|█         | 79/730 [07:08<57:52,  5.33s/it]
                                                
{'train/ce_loss': 4.78125, 'train/diffusion_loss': 0.42311185598373413, 'epoch': 1.08}

 11%|█         | 79/730 [07:08<57:52,  5.33s/it]
                                                
{'train/learning_rate_real': 2.460230829135746e-05, 'epoch': 1.08}

 11%|█         | 79/730 [07:08<57:52,  5.33s/it]03/16/2026 06:54:41 - INFO - __main__ - LoRA debug step 80: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 11%|█         | 80/730 [07:09<57:07,  5.27s/it]
                                                
{'loss': 3.1839, 'grad_norm': 1.6366914510726929, 'learning_rate': 2.460230829135746e-05, 'epoch': 1.1}

 11%|█         | 80/730 [07:09<57:07,  5.27s/it]
                                                
{'debug/num_tok_total': 2459.0, 'debug/num_tok_loss': 1798.0, 'debug/num_lat_total': 2459.0, 'debug/num_lat_loss': 1798.0, 'epoch': 1.1}

 11%|█         | 80/730 [07:10<57:07,  5.27s/it]
                                                
{'train/ce_loss': 4.8125, 'train/diffusion_loss': 0.4312282204627991, 'epoch': 1.1}

 11%|█         | 80/730 [07:10<57:07,  5.27s/it]
                                                
{'train/learning_rate_real': 2.4588309565776302e-05, 'epoch': 1.1}

 11%|█         | 80/730 [07:10<57:07,  5.27s/it]
                                                
{'debug/num_tok_total': 2454.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2454.0, 'debug/num_lat_loss': 1799.0, 'epoch': 1.1}

 11%|█         | 80/730 [07:11<57:07,  5.27s/it]
                                                
{'train/ce_loss': 4.78125, 'train/diffusion_loss': 0.4743603467941284, 'epoch': 1.1}

 11%|█         | 80/730 [07:11<57:07,  5.27s/it]
                                                
{'train/learning_rate_real': 2.4588309565776302e-05, 'epoch': 1.1}

 11%|█         | 80/730 [07:11<57:07,  5.27s/it]
                                                
{'debug/num_tok_total': 2443.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2443.0, 'debug/num_lat_loss': 1791.0, 'epoch': 1.1}

 11%|█         | 80/730 [07:12<57:07,  5.27s/it]
                                                
{'train/ce_loss': 4.71875, 'train/diffusion_loss': 0.48455047607421875, 'epoch': 1.1}

 11%|█         | 80/730 [07:12<57:07,  5.27s/it]
                                                
{'train/learning_rate_real': 2.4588309565776302e-05, 'epoch': 1.1}

 11%|█         | 80/730 [07:12<57:07,  5.27s/it]
                                                
{'debug/num_tok_total': 2669.0, 'debug/num_tok_loss': 1804.0, 'debug/num_lat_total': 2669.0, 'debug/num_lat_loss': 1804.0, 'epoch': 1.1}

 11%|█         | 80/730 [07:14<57:07,  5.27s/it]
                                                
{'train/ce_loss': 4.8125, 'train/diffusion_loss': 0.4093216359615326, 'epoch': 1.1}

 11%|█         | 80/730 [07:14<57:07,  5.27s/it]
                                                
{'train/learning_rate_real': 2.4588309565776302e-05, 'epoch': 1.1}

 11%|█         | 80/730 [07:14<57:07,  5.27s/it]
 11%|█         | 81/730 [07:14<56:53,  5.26s/it]
                                                
{'debug/num_tok_total': 2668.0, 'debug/num_tok_loss': 1798.0, 'debug/num_lat_total': 2668.0, 'debug/num_lat_loss': 1798.0, 'epoch': 1.11}

 11%|█         | 81/730 [07:15<56:53,  5.26s/it]
                                                
{'train/ce_loss': 4.84375, 'train/diffusion_loss': 0.4128156304359436, 'epoch': 1.11}

 11%|█         | 81/730 [07:15<56:53,  5.26s/it]
                                                
{'train/learning_rate_real': 2.4574072828613354e-05, 'epoch': 1.11}

 11%|█         | 81/730 [07:15<56:53,  5.26s/it]
                                                
{'debug/num_tok_total': 2876.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2876.0, 'debug/num_lat_loss': 1791.0, 'epoch': 1.11}

 11%|█         | 81/730 [07:16<56:53,  5.26s/it]
                                                
{'train/ce_loss': 5.0625, 'train/diffusion_loss': 0.35143572092056274, 'epoch': 1.11}

 11%|█         | 81/730 [07:16<56:53,  5.26s/it]
                                                
{'train/learning_rate_real': 2.4574072828613354e-05, 'epoch': 1.11}

 11%|█         | 81/730 [07:16<56:53,  5.26s/it]
                                                
{'debug/num_tok_total': 2652.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2652.0, 'debug/num_lat_loss': 1796.0, 'epoch': 1.11}

 11%|█         | 81/730 [07:18<56:53,  5.26s/it]
                                                
{'train/ce_loss': 4.84375, 'train/diffusion_loss': 0.4372323751449585, 'epoch': 1.11}

 11%|█         | 81/730 [07:18<56:53,  5.26s/it]
                                                
{'train/learning_rate_real': 2.4574072828613354e-05, 'epoch': 1.11}

 11%|█         | 81/730 [07:18<56:53,  5.26s/it]
                                                
{'debug/num_tok_total': 3108.0, 'debug/num_tok_loss': 1803.0, 'debug/num_lat_total': 3108.0, 'debug/num_lat_loss': 1803.0, 'epoch': 1.11}

 11%|█         | 81/730 [07:19<56:53,  5.26s/it]
                                                
{'train/ce_loss': 4.84375, 'train/diffusion_loss': 0.3405665457248688, 'epoch': 1.11}

 11%|█         | 81/730 [07:19<56:53,  5.26s/it]
                                                
{'train/learning_rate_real': 2.4574072828613354e-05, 'epoch': 1.11}

 11%|█         | 81/730 [07:19<56:53,  5.26s/it]
 11%|█         | 82/730 [07:19<57:17,  5.30s/it]
                                                
{'debug/num_tok_total': 2637.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2637.0, 'debug/num_lat_loss': 1772.0, 'epoch': 1.12}

 11%|█         | 82/730 [07:21<57:17,  5.30s/it]
                                                
{'train/ce_loss': 4.9375, 'train/diffusion_loss': 0.40083348751068115, 'epoch': 1.12}

 11%|█         | 82/730 [07:21<57:17,  5.30s/it]
                                                
{'train/learning_rate_real': 2.455959836018145e-05, 'epoch': 1.12}

 11%|█         | 82/730 [07:21<57:17,  5.30s/it]
                                                
{'debug/num_tok_total': 3062.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 3062.0, 'debug/num_lat_loss': 1772.0, 'epoch': 1.12}

 11%|█         | 82/730 [07:22<57:17,  5.30s/it]
                                                
{'train/ce_loss': 4.875, 'train/diffusion_loss': 0.2902744710445404, 'epoch': 1.12}

 11%|█         | 82/730 [07:22<57:17,  5.30s/it]
                                                
{'train/learning_rate_real': 2.455959836018145e-05, 'epoch': 1.12}

 11%|█         | 82/730 [07:22<57:17,  5.30s/it]
                                                
{'debug/num_tok_total': 2280.0, 'debug/num_tok_loss': 1699.0, 'debug/num_lat_total': 2280.0, 'debug/num_lat_loss': 1699.0, 'epoch': 1.12}

 11%|█         | 82/730 [07:23<57:17,  5.30s/it]
                                                
{'train/ce_loss': 4.84375, 'train/diffusion_loss': 0.48062434792518616, 'epoch': 1.12}

 11%|█         | 82/730 [07:23<57:17,  5.30s/it]
                                                
{'train/learning_rate_real': 2.455959836018145e-05, 'epoch': 1.12}

 11%|█         | 82/730 [07:23<57:17,  5.30s/it]
                                                
{'debug/num_tok_total': 2765.0, 'debug/num_tok_loss': 1633.0, 'debug/num_lat_total': 2765.0, 'debug/num_lat_loss': 1633.0, 'epoch': 1.12}

 11%|█         | 82/730 [07:24<57:17,  5.30s/it]
                                                
{'train/ce_loss': 4.9375, 'train/diffusion_loss': 0.32621175050735474, 'epoch': 1.12}

 11%|█         | 82/730 [07:24<57:17,  5.30s/it]
                                                
{'train/learning_rate_real': 2.455959836018145e-05, 'epoch': 1.12}

 11%|█         | 82/730 [07:24<57:17,  5.30s/it]
 11%|█▏        | 83/730 [07:25<57:04,  5.29s/it]
                                                
{'debug/num_tok_total': 2657.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2657.0, 'debug/num_lat_loss': 1794.0, 'epoch': 1.14}

 11%|█▏        | 83/730 [07:26<57:04,  5.29s/it]
                                                
{'train/ce_loss': 4.75, 'train/diffusion_loss': 0.41724833846092224, 'epoch': 1.14}

 11%|█▏        | 83/730 [07:26<57:04,  5.29s/it]
                                                
{'train/learning_rate_real': 2.4544886445474215e-05, 'epoch': 1.14}

 11%|█▏        | 83/730 [07:26<57:04,  5.29s/it]
                                                
{'debug/num_tok_total': 3088.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 3088.0, 'debug/num_lat_loss': 1783.0, 'epoch': 1.14}

 11%|█▏        | 83/730 [07:27<57:04,  5.29s/it]
                                                
{'train/ce_loss': 4.8125, 'train/diffusion_loss': 0.3163226842880249, 'epoch': 1.14}

 11%|█▏        | 83/730 [07:27<57:04,  5.29s/it]
                                                
{'train/learning_rate_real': 2.4544886445474215e-05, 'epoch': 1.14}

 11%|█▏        | 83/730 [07:27<57:04,  5.29s/it]
                                                
{'debug/num_tok_total': 2423.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2423.0, 'debug/num_lat_loss': 1779.0, 'epoch': 1.14}

 11%|█▏        | 83/730 [07:28<57:04,  5.29s/it]
                                                
{'train/ce_loss': 4.78125, 'train/diffusion_loss': 0.45068359375, 'epoch': 1.14}

 11%|█▏        | 83/730 [07:28<57:04,  5.29s/it]
                                                
{'train/learning_rate_real': 2.4544886445474215e-05, 'epoch': 1.14}

 11%|█▏        | 83/730 [07:28<57:04,  5.29s/it]
                                                
{'debug/num_tok_total': 3037.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 3037.0, 'debug/num_lat_loss': 1764.0, 'epoch': 1.14}

 11%|█▏        | 83/730 [07:30<57:04,  5.29s/it]
                                                
{'train/ce_loss': 4.75, 'train/diffusion_loss': 0.3273552358150482, 'epoch': 1.14}

 11%|█▏        | 83/730 [07:30<57:04,  5.29s/it]
                                                
{'train/learning_rate_real': 2.4544886445474215e-05, 'epoch': 1.14}

 11%|█▏        | 83/730 [07:30<57:04,  5.29s/it]
 12%|█▏        | 84/730 [07:30<57:49,  5.37s/it]
                                                
{'debug/num_tok_total': 2643.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2643.0, 'debug/num_lat_loss': 1784.0, 'epoch': 1.15}

 12%|█▏        | 84/730 [07:31<57:49,  5.37s/it]
                                                
{'train/ce_loss': 5.125, 'train/diffusion_loss': 0.41567105054855347, 'epoch': 1.15}

 12%|█▏        | 84/730 [07:31<57:49,  5.37s/it]
                                                
{'train/learning_rate_real': 2.4529937374160438e-05, 'epoch': 1.15}

 12%|█▏        | 84/730 [07:31<57:49,  5.37s/it]
                                                
{'debug/num_tok_total': 2448.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2448.0, 'debug/num_lat_loss': 1799.0, 'epoch': 1.15}

 12%|█▏        | 84/730 [07:33<57:49,  5.37s/it]
                                                
{'train/ce_loss': 4.90625, 'train/diffusion_loss': 0.4494771957397461, 'epoch': 1.15}

 12%|█▏        | 84/730 [07:33<57:49,  5.37s/it]
                                                
{'train/learning_rate_real': 2.4529937374160438e-05, 'epoch': 1.15}

 12%|█▏        | 84/730 [07:33<57:49,  5.37s/it]
                                                
{'debug/num_tok_total': 2386.0, 'debug/num_tok_loss': 1751.0, 'debug/num_lat_total': 2386.0, 'debug/num_lat_loss': 1751.0, 'epoch': 1.15}

 12%|█▏        | 84/730 [07:34<57:49,  5.37s/it]
                                                
{'train/ce_loss': 4.71875, 'train/diffusion_loss': 0.45726028084754944, 'epoch': 1.15}

 12%|█▏        | 84/730 [07:34<57:49,  5.37s/it]
                                                
{'train/learning_rate_real': 2.4529937374160438e-05, 'epoch': 1.15}

 12%|█▏        | 84/730 [07:34<57:49,  5.37s/it]
                                                
{'debug/num_tok_total': 2621.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2621.0, 'debug/num_lat_loss': 1766.0, 'epoch': 1.15}

 12%|█▏        | 84/730 [07:35<57:49,  5.37s/it]
                                                
{'train/ce_loss': 5.03125, 'train/diffusion_loss': 0.4082919955253601, 'epoch': 1.15}

 12%|█▏        | 84/730 [07:35<57:49,  5.37s/it]
                                                
{'train/learning_rate_real': 2.4529937374160438e-05, 'epoch': 1.15}

 12%|█▏        | 84/730 [07:35<57:49,  5.37s/it]
 12%|█▏        | 85/730 [07:35<56:49,  5.29s/it]
                                                
{'debug/num_tok_total': 2445.0, 'debug/num_tok_loss': 1798.0, 'debug/num_lat_total': 2445.0, 'debug/num_lat_loss': 1798.0, 'epoch': 1.16}

 12%|█▏        | 85/730 [07:36<56:49,  5.29s/it]
                                                
{'train/ce_loss': 4.65625, 'train/diffusion_loss': 0.45018747448921204, 'epoch': 1.16}

 12%|█▏        | 85/730 [07:36<56:49,  5.29s/it]
                                                
{'train/learning_rate_real': 2.4514751440578394e-05, 'epoch': 1.16}

 12%|█▏        | 85/730 [07:36<56:49,  5.29s/it]
                                                
{'debug/num_tok_total': 2833.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2833.0, 'debug/num_lat_loss': 1764.0, 'epoch': 1.16}

 12%|█▏        | 85/730 [07:38<56:49,  5.29s/it]
                                                
{'train/ce_loss': 4.78125, 'train/diffusion_loss': 0.36236572265625, 'epoch': 1.16}

 12%|█▏        | 85/730 [07:38<56:49,  5.29s/it]
                                                
{'train/learning_rate_real': 2.4514751440578394e-05, 'epoch': 1.16}

 12%|█▏        | 85/730 [07:38<56:49,  5.29s/it]
                                                
{'debug/num_tok_total': 2428.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2428.0, 'debug/num_lat_loss': 1792.0, 'epoch': 1.16}

 12%|█▏        | 85/730 [07:39<56:49,  5.29s/it]
                                                
{'train/ce_loss': 4.8125, 'train/diffusion_loss': 0.4757889211177826, 'epoch': 1.16}

 12%|█▏        | 85/730 [07:39<56:49,  5.29s/it]
                                                
{'train/learning_rate_real': 2.4514751440578394e-05, 'epoch': 1.16}

 12%|█▏        | 85/730 [07:39<56:49,  5.29s/it]
                                                
{'debug/num_tok_total': 2668.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2668.0, 'debug/num_lat_loss': 1795.0, 'epoch': 1.16}

 12%|█▏        | 85/730 [07:40<56:49,  5.29s/it]
                                                
{'train/ce_loss': 4.8125, 'train/diffusion_loss': 0.40141940116882324, 'epoch': 1.16}

 12%|█▏        | 85/730 [07:40<56:49,  5.29s/it]
                                                
{'train/learning_rate_real': 2.4514751440578394e-05, 'epoch': 1.16}

 12%|█▏        | 85/730 [07:40<56:49,  5.29s/it]
 12%|█▏        | 86/730 [07:41<56:22,  5.25s/it]
                                                
{'debug/num_tok_total': 2393.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 2393.0, 'debug/num_lat_loss': 1767.0, 'epoch': 1.18}

 12%|█▏        | 86/730 [07:42<56:22,  5.25s/it]
                                                
{'train/ce_loss': 4.6875, 'train/diffusion_loss': 0.48776713013648987, 'epoch': 1.18}

 12%|█▏        | 86/730 [07:42<56:22,  5.25s/it]
                                                
{'train/learning_rate_real': 2.4499328943730026e-05, 'epoch': 1.18}

 12%|█▏        | 86/730 [07:42<56:22,  5.25s/it]
                                                
{'debug/num_tok_total': 2692.0, 'debug/num_tok_loss': 1699.0, 'debug/num_lat_total': 2692.0, 'debug/num_lat_loss': 1699.0, 'epoch': 1.18}

 12%|█▏        | 86/730 [07:43<56:22,  5.25s/it]
                                                
{'train/ce_loss': 4.84375, 'train/diffusion_loss': 0.3635094165802002, 'epoch': 1.18}

 12%|█▏        | 86/730 [07:43<56:22,  5.25s/it]
                                                
{'train/learning_rate_real': 2.4499328943730026e-05, 'epoch': 1.18}

 12%|█▏        | 86/730 [07:43<56:22,  5.25s/it]
                                                
{'debug/num_tok_total': 2856.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2856.0, 'debug/num_lat_loss': 1774.0, 'epoch': 1.18}

 12%|█▏        | 86/730 [07:44<56:22,  5.25s/it]
                                                
{'train/ce_loss': 4.75, 'train/diffusion_loss': 0.38390570878982544, 'epoch': 1.18}

 12%|█▏        | 86/730 [07:44<56:22,  5.25s/it]
                                                
{'train/learning_rate_real': 2.4499328943730026e-05, 'epoch': 1.18}

 12%|█▏        | 86/730 [07:44<56:22,  5.25s/it]
                                                
{'debug/num_tok_total': 2422.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2422.0, 'debug/num_lat_loss': 1790.0, 'epoch': 1.18}

 12%|█▏        | 86/730 [07:45<56:22,  5.25s/it]
                                                
{'train/ce_loss': 4.75, 'train/diffusion_loss': 0.4719712436199188, 'epoch': 1.18}

 12%|█▏        | 86/730 [07:45<56:22,  5.25s/it]
                                                
{'train/learning_rate_real': 2.4499328943730026e-05, 'epoch': 1.18}

 12%|█▏        | 86/730 [07:45<56:22,  5.25s/it]
 12%|█▏        | 87/730 [07:46<56:33,  5.28s/it]
                                                
{'debug/num_tok_total': 2637.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2637.0, 'debug/num_lat_loss': 1779.0, 'epoch': 1.19}

 12%|█▏        | 87/730 [07:47<56:33,  5.28s/it]
                                                
{'train/ce_loss': 4.46875, 'train/diffusion_loss': 0.4493198096752167, 'epoch': 1.19}

 12%|█▏        | 87/730 [07:47<56:33,  5.28s/it]
                                                
{'train/learning_rate_real': 2.448367018727507e-05, 'epoch': 1.19}

 12%|█▏        | 87/730 [07:47<56:33,  5.28s/it]
                                                
{'debug/num_tok_total': 2464.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2464.0, 'debug/num_lat_loss': 1802.0, 'epoch': 1.19}

 12%|█▏        | 87/730 [07:48<56:33,  5.28s/it]
                                                
{'train/ce_loss': 4.75, 'train/diffusion_loss': 0.44079920649528503, 'epoch': 1.19}

 12%|█▏        | 87/730 [07:48<56:33,  5.28s/it]
                                                
{'train/learning_rate_real': 2.448367018727507e-05, 'epoch': 1.19}

 12%|█▏        | 87/730 [07:48<56:33,  5.28s/it]
                                                
{'debug/num_tok_total': 2218.0, 'debug/num_tok_loss': 1803.0, 'debug/num_lat_total': 2218.0, 'debug/num_lat_loss': 1803.0, 'epoch': 1.19}

 12%|█▏        | 87/730 [07:49<56:33,  5.28s/it]
                                                
{'train/ce_loss': 4.8125, 'train/diffusion_loss': 0.5176950097084045, 'epoch': 1.19}

 12%|█▏        | 87/730 [07:49<56:33,  5.28s/it]
                                                
{'train/learning_rate_real': 2.448367018727507e-05, 'epoch': 1.19}

 12%|█▏        | 87/730 [07:49<56:33,  5.28s/it]
                                                
{'debug/num_tok_total': 2677.0, 'debug/num_tok_loss': 1804.0, 'debug/num_lat_total': 2677.0, 'debug/num_lat_loss': 1804.0, 'epoch': 1.19}

 12%|█▏        | 87/730 [07:51<56:33,  5.28s/it]
                                                
{'train/ce_loss': 4.6875, 'train/diffusion_loss': 0.4029632806777954, 'epoch': 1.19}

 12%|█▏        | 87/730 [07:51<56:33,  5.28s/it]
                                                
{'train/learning_rate_real': 2.448367018727507e-05, 'epoch': 1.19}

 12%|█▏        | 87/730 [07:51<56:33,  5.28s/it]
 12%|█▏        | 88/730 [07:51<55:58,  5.23s/it]
                                                
{'debug/num_tok_total': 2404.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2404.0, 'debug/num_lat_loss': 1770.0, 'epoch': 1.21}

 12%|█▏        | 88/730 [07:52<55:58,  5.23s/it]
                                                
{'train/ce_loss': 4.5, 'train/diffusion_loss': 0.4859849512577057, 'epoch': 1.21}

 12%|█▏        | 88/730 [07:52<55:58,  5.23s/it]
                                                
{'train/learning_rate_real': 2.446777547952507e-05, 'epoch': 1.21}

 12%|█▏        | 88/730 [07:52<55:58,  5.23s/it]
                                                
{'debug/num_tok_total': 2067.0, 'debug/num_tok_loss': 1625.0, 'debug/num_lat_total': 2067.0, 'debug/num_lat_loss': 1625.0, 'epoch': 1.21}

 12%|█▏        | 88/730 [07:53<55:58,  5.23s/it]
                                                
{'train/ce_loss': 4.78125, 'train/diffusion_loss': 0.5068560838699341, 'epoch': 1.21}

 12%|█▏        | 88/730 [07:53<55:58,  5.23s/it]
                                                
{'train/learning_rate_real': 2.446777547952507e-05, 'epoch': 1.21}

 12%|█▏        | 88/730 [07:53<55:58,  5.23s/it]
                                                
{'debug/num_tok_total': 3051.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 3051.0, 'debug/num_lat_loss': 1780.0, 'epoch': 1.21}

 12%|█▏        | 88/730 [07:55<55:58,  5.23s/it]
                                                
{'train/ce_loss': 4.6875, 'train/diffusion_loss': 0.3534332513809204, 'epoch': 1.21}

 12%|█▏        | 88/730 [07:55<55:58,  5.23s/it]
                                                
{'train/learning_rate_real': 2.446777547952507e-05, 'epoch': 1.21}

 12%|█▏        | 88/730 [07:55<55:58,  5.23s/it]
                                                
{'debug/num_tok_total': 2410.0, 'debug/num_tok_loss': 1761.0, 'debug/num_lat_total': 2410.0, 'debug/num_lat_loss': 1761.0, 'epoch': 1.21}

 12%|█▏        | 88/730 [07:56<55:58,  5.23s/it]
                                                
{'train/ce_loss': 4.6875, 'train/diffusion_loss': 0.45841628313064575, 'epoch': 1.21}

 12%|█▏        | 88/730 [07:56<55:58,  5.23s/it]
                                                
{'train/learning_rate_real': 2.446777547952507e-05, 'epoch': 1.21}

 12%|█▏        | 88/730 [07:56<55:58,  5.23s/it]
 12%|█▏        | 89/730 [07:56<56:03,  5.25s/it]
                                                
{'debug/num_tok_total': 2399.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2399.0, 'debug/num_lat_loss': 1772.0, 'epoch': 1.22}

 12%|█▏        | 89/730 [07:57<56:03,  5.25s/it]
                                                
{'train/ce_loss': 4.625, 'train/diffusion_loss': 0.4496181309223175, 'epoch': 1.22}

 12%|█▏        | 89/730 [07:57<56:03,  5.25s/it]
                                                
{'train/learning_rate_real': 2.445164513343731e-05, 'epoch': 1.22}

 12%|█▏        | 89/730 [07:57<56:03,  5.25s/it]
                                                
{'debug/num_tok_total': 2210.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2210.0, 'debug/num_lat_loss': 1782.0, 'epoch': 1.22}

 12%|█▏        | 89/730 [07:58<56:03,  5.25s/it]
                                                
{'train/ce_loss': 4.65625, 'train/diffusion_loss': 0.5095124840736389, 'epoch': 1.22}

 12%|█▏        | 89/730 [07:59<56:03,  5.25s/it]
                                                
{'train/learning_rate_real': 2.445164513343731e-05, 'epoch': 1.22}

 12%|█▏        | 89/730 [07:59<56:03,  5.25s/it]
                                                
{'debug/num_tok_total': 2244.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 2244.0, 'debug/num_lat_loss': 1801.0, 'epoch': 1.22}

 12%|█▏        | 89/730 [08:00<56:03,  5.25s/it]
                                                
{'train/ce_loss': 4.84375, 'train/diffusion_loss': 0.5054518580436707, 'epoch': 1.22}

 12%|█▏        | 89/730 [08:00<56:03,  5.25s/it]
                                                
{'train/learning_rate_real': 2.445164513343731e-05, 'epoch': 1.22}

 12%|█▏        | 89/730 [08:00<56:03,  5.25s/it]
                                                
{'debug/num_tok_total': 3260.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 3260.0, 'debug/num_lat_loss': 1766.0, 'epoch': 1.22}

 12%|█▏        | 89/730 [08:01<56:03,  5.25s/it]
                                                
{'train/ce_loss': 4.65625, 'train/diffusion_loss': 0.2607570290565491, 'epoch': 1.22}

 12%|█▏        | 89/730 [08:01<56:03,  5.25s/it]
                                                
{'train/learning_rate_real': 2.445164513343731e-05, 'epoch': 1.22}

 12%|█▏        | 89/730 [08:01<56:03,  5.25s/it]03/16/2026 06:55:33 - INFO - __main__ - LoRA debug step 90: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 12%|█▏        | 90/730 [08:02<55:47,  5.23s/it]
                                                
{'loss': 3.1204, 'grad_norm': 1.9624649286270142, 'learning_rate': 2.445164513343731e-05, 'epoch': 1.23}

 12%|█▏        | 90/730 [08:02<55:47,  5.23s/it]
                                                
{'debug/num_tok_total': 2453.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2453.0, 'debug/num_lat_loss': 1782.0, 'epoch': 1.23}

 12%|█▏        | 90/730 [08:03<55:47,  5.23s/it]
                                                
{'train/ce_loss': 4.6875, 'train/diffusion_loss': 0.4628406763076782, 'epoch': 1.23}

 12%|█▏        | 90/730 [08:03<55:47,  5.23s/it]
                                                
{'train/learning_rate_real': 2.443527946660865e-05, 'epoch': 1.23}

 12%|█▏        | 90/730 [08:03<55:47,  5.23s/it]
                                                
{'debug/num_tok_total': 3073.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 3073.0, 'debug/num_lat_loss': 1781.0, 'epoch': 1.23}

 12%|█▏        | 90/730 [08:04<55:47,  5.23s/it]
                                                
{'train/ce_loss': 4.71875, 'train/diffusion_loss': 0.2992222309112549, 'epoch': 1.23}

 12%|█▏        | 90/730 [08:04<55:47,  5.23s/it]
                                                
{'train/learning_rate_real': 2.443527946660865e-05, 'epoch': 1.23}

 12%|█▏        | 90/730 [08:04<55:47,  5.23s/it]
                                                
{'debug/num_tok_total': 2897.0, 'debug/num_tok_loss': 1806.0, 'debug/num_lat_total': 2897.0, 'debug/num_lat_loss': 1806.0, 'epoch': 1.23}

 12%|█▏        | 90/730 [08:05<55:47,  5.23s/it]
                                                
{'train/ce_loss': 4.71875, 'train/diffusion_loss': 0.36971840262413025, 'epoch': 1.23}

 12%|█▏        | 90/730 [08:05<55:47,  5.23s/it]
                                                
{'train/learning_rate_real': 2.443527946660865e-05, 'epoch': 1.23}

 12%|█▏        | 90/730 [08:05<55:47,  5.23s/it]
                                                
{'debug/num_tok_total': 2889.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2889.0, 'debug/num_lat_loss': 1793.0, 'epoch': 1.23}

 12%|█▏        | 90/730 [08:07<55:47,  5.23s/it]
                                                
{'train/ce_loss': 4.5625, 'train/diffusion_loss': 0.3495144248008728, 'epoch': 1.23}

 12%|█▏        | 90/730 [08:07<55:47,  5.23s/it]
                                                
{'train/learning_rate_real': 2.443527946660865e-05, 'epoch': 1.23}

 12%|█▏        | 90/730 [08:07<55:47,  5.23s/it]
 12%|█▏        | 91/730 [08:07<56:57,  5.35s/it]
                                                
{'debug/num_tok_total': 3326.0, 'debug/num_tok_loss': 1806.0, 'debug/num_lat_total': 3326.0, 'debug/num_lat_loss': 1806.0, 'epoch': 1.25}

 12%|█▏        | 91/730 [08:08<56:57,  5.35s/it]
                                                
{'train/ce_loss': 4.5625, 'train/diffusion_loss': 0.27441421151161194, 'epoch': 1.25}

 12%|█▏        | 91/730 [08:09<56:57,  5.35s/it]
                                                
{'train/learning_rate_real': 2.441867880126928e-05, 'epoch': 1.25}

 12%|█▏        | 91/730 [08:09<56:57,  5.35s/it]
                                                
{'debug/num_tok_total': 2192.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2192.0, 'debug/num_lat_loss': 1770.0, 'epoch': 1.25}

 12%|█▏        | 91/730 [08:10<56:57,  5.35s/it]
                                                
{'train/ce_loss': 4.75, 'train/diffusion_loss': 0.5229299068450928, 'epoch': 1.25}

 12%|█▏        | 91/730 [08:10<56:57,  5.35s/it]
                                                
{'train/learning_rate_real': 2.441867880126928e-05, 'epoch': 1.25}

 12%|█▏        | 91/730 [08:10<56:57,  5.35s/it]
                                                
{'debug/num_tok_total': 2612.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 2612.0, 'debug/num_lat_loss': 1767.0, 'epoch': 1.25}

 12%|█▏        | 91/730 [08:11<56:57,  5.35s/it]
                                                
{'train/ce_loss': 4.6875, 'train/diffusion_loss': 0.4148879945278168, 'epoch': 1.25}

 12%|█▏        | 91/730 [08:11<56:57,  5.35s/it]
                                                
{'train/learning_rate_real': 2.441867880126928e-05, 'epoch': 1.25}

 12%|█▏        | 91/730 [08:11<56:57,  5.35s/it]
                                                
{'debug/num_tok_total': 3067.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 3067.0, 'debug/num_lat_loss': 1780.0, 'epoch': 1.25}

 12%|█▏        | 91/730 [08:12<56:57,  5.35s/it]
                                                
{'train/ce_loss': 4.71875, 'train/diffusion_loss': 0.30536091327667236, 'epoch': 1.25}

 12%|█▏        | 91/730 [08:12<56:57,  5.35s/it]
                                                
{'train/learning_rate_real': 2.441867880126928e-05, 'epoch': 1.25}

 12%|█▏        | 91/730 [08:12<56:57,  5.35s/it]
 13%|█▎        | 92/730 [08:13<57:37,  5.42s/it]
                                                
{'debug/num_tok_total': 2094.0, 'debug/num_tok_loss': 1677.0, 'debug/num_lat_total': 2094.0, 'debug/num_lat_loss': 1677.0, 'epoch': 1.26}

 13%|█▎        | 92/730 [08:14<57:37,  5.42s/it]
                                                
{'train/ce_loss': 4.5, 'train/diffusion_loss': 0.4853772521018982, 'epoch': 1.26}

 13%|█▎        | 92/730 [08:14<57:37,  5.42s/it]
                                                
{'train/learning_rate_real': 2.4401843464276368e-05, 'epoch': 1.26}

 13%|█▎        | 92/730 [08:14<57:37,  5.42s/it]
                                                
{'debug/num_tok_total': 3220.0, 'debug/num_tok_loss': 1750.0, 'debug/num_lat_total': 3220.0, 'debug/num_lat_loss': 1750.0, 'epoch': 1.26}

 13%|█▎        | 92/730 [08:15<57:37,  5.42s/it]
                                                
{'train/ce_loss': 4.625, 'train/diffusion_loss': 0.2949419319629669, 'epoch': 1.26}

 13%|█▎        | 92/730 [08:15<57:37,  5.42s/it]
                                                
{'train/learning_rate_real': 2.4401843464276368e-05, 'epoch': 1.26}

 13%|█▎        | 92/730 [08:15<57:37,  5.42s/it]
                                                
{'debug/num_tok_total': 2483.0, 'debug/num_tok_loss': 1653.0, 'debug/num_lat_total': 2483.0, 'debug/num_lat_loss': 1653.0, 'epoch': 1.26}

 13%|█▎        | 92/730 [08:16<57:37,  5.42s/it]
                                                
{'train/ce_loss': 4.71875, 'train/diffusion_loss': 0.3998708128929138, 'epoch': 1.26}

 13%|█▎        | 92/730 [08:16<57:37,  5.42s/it]
                                                
{'train/learning_rate_real': 2.4401843464276368e-05, 'epoch': 1.26}

 13%|█▎        | 92/730 [08:16<57:37,  5.42s/it]
                                                
{'debug/num_tok_total': 3071.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 3071.0, 'debug/num_lat_loss': 1775.0, 'epoch': 1.26}

 13%|█▎        | 92/730 [08:18<57:37,  5.42s/it]
                                                
{'train/ce_loss': 4.875, 'train/diffusion_loss': 0.28526005148887634, 'epoch': 1.26}

 13%|█▎        | 92/730 [08:18<57:37,  5.42s/it]
                                                
{'train/learning_rate_real': 2.4401843464276368e-05, 'epoch': 1.26}

 13%|█▎        | 92/730 [08:18<57:37,  5.42s/it]
 13%|█▎        | 93/730 [08:18<57:14,  5.39s/it]
                                                
{'debug/num_tok_total': 2826.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2826.0, 'debug/num_lat_loss': 1772.0, 'epoch': 1.27}

 13%|█▎        | 93/730 [08:19<57:14,  5.39s/it]
                                                
{'train/ce_loss': 4.53125, 'train/diffusion_loss': 0.36494237184524536, 'epoch': 1.27}

 13%|█▎        | 93/730 [08:19<57:14,  5.39s/it]
                                                
{'train/learning_rate_real': 2.4384773787107616e-05, 'epoch': 1.27}

 13%|█▎        | 93/730 [08:19<57:14,  5.39s/it]
                                                
{'debug/num_tok_total': 2644.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2644.0, 'debug/num_lat_loss': 1770.0, 'epoch': 1.27}

 13%|█▎        | 93/730 [08:20<57:14,  5.39s/it]
                                                
{'train/ce_loss': 4.59375, 'train/diffusion_loss': 0.4222411513328552, 'epoch': 1.27}

 13%|█▎        | 93/730 [08:20<57:14,  5.39s/it]
                                                
{'train/learning_rate_real': 2.4384773787107616e-05, 'epoch': 1.27}

 13%|█▎        | 93/730 [08:20<57:14,  5.39s/it]
                                                
{'debug/num_tok_total': 2240.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2240.0, 'debug/num_lat_loss': 1802.0, 'epoch': 1.27}

 13%|█▎        | 93/730 [08:22<57:14,  5.39s/it]
                                                
{'train/ce_loss': 4.4375, 'train/diffusion_loss': 0.4870891571044922, 'epoch': 1.27}

 13%|█▎        | 93/730 [08:22<57:14,  5.39s/it]
                                                
{'train/learning_rate_real': 2.4384773787107616e-05, 'epoch': 1.27}

 13%|█▎        | 93/730 [08:22<57:14,  5.39s/it]
                                                
{'debug/num_tok_total': 2636.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2636.0, 'debug/num_lat_loss': 1778.0, 'epoch': 1.27}

 13%|█▎        | 93/730 [08:23<57:14,  5.39s/it]
                                                
{'train/ce_loss': 4.53125, 'train/diffusion_loss': 0.3951069116592407, 'epoch': 1.27}

 13%|█▎        | 93/730 [08:23<57:14,  5.39s/it]
                                                
{'train/learning_rate_real': 2.4384773787107616e-05, 'epoch': 1.27}

 13%|█▎        | 93/730 [08:23<57:14,  5.39s/it]
 13%|█▎        | 94/730 [08:23<57:06,  5.39s/it]
                                                
{'debug/num_tok_total': 2271.0, 'debug/num_tok_loss': 1698.0, 'debug/num_lat_total': 2271.0, 'debug/num_lat_loss': 1698.0, 'epoch': 1.29}

 13%|█▎        | 94/730 [08:25<57:06,  5.39s/it]
                                                
{'train/ce_loss': 4.53125, 'train/diffusion_loss': 0.4701569974422455, 'epoch': 1.29}

 13%|█▎        | 94/730 [08:25<57:06,  5.39s/it]
                                                
{'train/learning_rate_real': 2.4367470105854766e-05, 'epoch': 1.29}

 13%|█▎        | 94/730 [08:25<57:06,  5.39s/it]
                                                
{'debug/num_tok_total': 2868.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2868.0, 'debug/num_lat_loss': 1794.0, 'epoch': 1.29}

 13%|█▎        | 94/730 [08:26<57:06,  5.39s/it]
                                                
{'train/ce_loss': 4.53125, 'train/diffusion_loss': 0.34901419281959534, 'epoch': 1.29}

 13%|█▎        | 94/730 [08:26<57:06,  5.39s/it]
                                                
{'train/learning_rate_real': 2.4367470105854766e-05, 'epoch': 1.29}

 13%|█▎        | 94/730 [08:26<57:06,  5.39s/it]
                                                
{'debug/num_tok_total': 2200.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 2200.0, 'debug/num_lat_loss': 1765.0, 'epoch': 1.29}

 13%|█▎        | 94/730 [08:27<57:06,  5.39s/it]
                                                
{'train/ce_loss': 4.59375, 'train/diffusion_loss': 0.5008375644683838, 'epoch': 1.29}

 13%|█▎        | 94/730 [08:27<57:06,  5.39s/it]
                                                
{'train/learning_rate_real': 2.4367470105854766e-05, 'epoch': 1.29}

 13%|█▎        | 94/730 [08:27<57:06,  5.39s/it]
                                                
{'debug/num_tok_total': 2420.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2420.0, 'debug/num_lat_loss': 1777.0, 'epoch': 1.29}

 13%|█▎        | 94/730 [08:28<57:06,  5.39s/it]
                                                
{'train/ce_loss': 4.6875, 'train/diffusion_loss': 0.4832378625869751, 'epoch': 1.29}

 13%|█▎        | 94/730 [08:28<57:06,  5.39s/it]
                                                
{'train/learning_rate_real': 2.4367470105854766e-05, 'epoch': 1.29}

 13%|█▎        | 94/730 [08:28<57:06,  5.39s/it]
 13%|█▎        | 95/730 [08:28<55:47,  5.27s/it]
                                                
{'debug/num_tok_total': 2578.0, 'debug/num_tok_loss': 1728.0, 'debug/num_lat_total': 2578.0, 'debug/num_lat_loss': 1728.0, 'epoch': 1.3}

 13%|█▎        | 95/730 [08:30<55:47,  5.27s/it]
                                                
{'train/ce_loss': 4.4375, 'train/diffusion_loss': 0.4089829623699188, 'epoch': 1.3}

 13%|█▎        | 95/730 [08:30<55:47,  5.27s/it]
                                                
{'train/learning_rate_real': 2.4349932761216942e-05, 'epoch': 1.3}

 13%|█▎        | 95/730 [08:30<55:47,  5.27s/it]
                                                
{'debug/num_tok_total': 2026.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2026.0, 'debug/num_lat_loss': 1802.0, 'epoch': 1.3}

 13%|█▎        | 95/730 [08:31<55:47,  5.27s/it]
                                                
{'train/ce_loss': 4.59375, 'train/diffusion_loss': 0.5384713411331177, 'epoch': 1.3}

 13%|█▎        | 95/730 [08:31<55:47,  5.27s/it]
                                                
{'train/learning_rate_real': 2.4349932761216942e-05, 'epoch': 1.3}

 13%|█▎        | 95/730 [08:31<55:47,  5.27s/it]
                                                
{'debug/num_tok_total': 2137.0, 'debug/num_tok_loss': 1482.0, 'debug/num_lat_total': 2137.0, 'debug/num_lat_loss': 1482.0, 'epoch': 1.3}

 13%|█▎        | 95/730 [08:32<55:47,  5.27s/it]
                                                
{'train/ce_loss': 4.5, 'train/diffusion_loss': 0.4377511143684387, 'epoch': 1.3}

 13%|█▎        | 95/730 [08:32<55:47,  5.27s/it]
                                                
{'train/learning_rate_real': 2.4349932761216942e-05, 'epoch': 1.3}

 13%|█▎        | 95/730 [08:32<55:47,  5.27s/it]
                                                
{'debug/num_tok_total': 2183.0, 'debug/num_tok_loss': 1757.0, 'debug/num_lat_total': 2183.0, 'debug/num_lat_loss': 1757.0, 'epoch': 1.3}

 13%|█▎        | 95/730 [08:33<55:47,  5.27s/it]
                                                
{'train/ce_loss': 4.65625, 'train/diffusion_loss': 0.5004580020904541, 'epoch': 1.3}

 13%|█▎        | 95/730 [08:33<55:47,  5.27s/it]
                                                
{'train/learning_rate_real': 2.4349932761216942e-05, 'epoch': 1.3}

 13%|█▎        | 95/730 [08:33<55:47,  5.27s/it]
 13%|█▎        | 96/730 [08:33<54:53,  5.19s/it]
                                                
{'debug/num_tok_total': 2826.0, 'debug/num_tok_loss': 1659.0, 'debug/num_lat_total': 2826.0, 'debug/num_lat_loss': 1659.0, 'epoch': 1.32}

 13%|█▎        | 96/730 [08:35<54:53,  5.19s/it]
                                                
{'train/ce_loss': 4.46875, 'train/diffusion_loss': 0.3254546821117401, 'epoch': 1.32}

 13%|█▎        | 96/730 [08:35<54:53,  5.19s/it]
                                                
{'train/learning_rate_real': 2.433216209849396e-05, 'epoch': 1.32}

 13%|█▎        | 96/730 [08:35<54:53,  5.19s/it]
                                                
{'debug/num_tok_total': 2857.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2857.0, 'debug/num_lat_loss': 1772.0, 'epoch': 1.32}

 13%|█▎        | 96/730 [08:36<54:53,  5.19s/it]
                                                
{'train/ce_loss': 4.46875, 'train/diffusion_loss': 0.33290404081344604, 'epoch': 1.32}

 13%|█▎        | 96/730 [08:36<54:53,  5.19s/it]
                                                
{'train/learning_rate_real': 2.433216209849396e-05, 'epoch': 1.32}

 13%|█▎        | 96/730 [08:36<54:53,  5.19s/it]
                                                
{'debug/num_tok_total': 3070.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 3070.0, 'debug/num_lat_loss': 1787.0, 'epoch': 1.32}

 13%|█▎        | 96/730 [08:37<54:53,  5.19s/it]
                                                
{'train/ce_loss': 5.03125, 'train/diffusion_loss': 0.3361597955226898, 'epoch': 1.32}

 13%|█▎        | 96/730 [08:37<54:53,  5.19s/it]
                                                
{'train/learning_rate_real': 2.433216209849396e-05, 'epoch': 1.32}

 13%|█▎        | 96/730 [08:37<54:53,  5.19s/it]
                                                
{'debug/num_tok_total': 2858.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2858.0, 'debug/num_lat_loss': 1781.0, 'epoch': 1.32}

 13%|█▎        | 96/730 [08:39<54:53,  5.19s/it]
                                                
{'train/ce_loss': 4.96875, 'train/diffusion_loss': 0.39105093479156494, 'epoch': 1.32}

 13%|█▎        | 96/730 [08:39<54:53,  5.19s/it]
                                                
{'train/learning_rate_real': 2.433216209849396e-05, 'epoch': 1.32}

 13%|█▎        | 96/730 [08:39<54:53,  5.19s/it]
 13%|█▎        | 97/730 [08:39<56:14,  5.33s/it]
                                                
{'debug/num_tok_total': 2608.0, 'debug/num_tok_loss': 1760.0, 'debug/num_lat_total': 2608.0, 'debug/num_lat_loss': 1760.0, 'epoch': 1.33}

 13%|█▎        | 97/730 [08:40<56:14,  5.33s/it]
                                                
{'train/ce_loss': 4.375, 'train/diffusion_loss': 0.4111551344394684, 'epoch': 1.33}

 13%|█▎        | 97/730 [08:40<56:14,  5.33s/it]
                                                
{'train/learning_rate_real': 2.4314158467579544e-05, 'epoch': 1.33}

 13%|█▎        | 97/730 [08:40<56:14,  5.33s/it]
                                                
{'debug/num_tok_total': 2663.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2663.0, 'debug/num_lat_loss': 1795.0, 'epoch': 1.33}

 13%|█▎        | 97/730 [08:41<56:14,  5.33s/it]
                                                
{'train/ce_loss': 4.625, 'train/diffusion_loss': 0.4159858524799347, 'epoch': 1.33}

 13%|█▎        | 97/730 [08:41<56:14,  5.33s/it]
                                                
{'train/learning_rate_real': 2.4314158467579544e-05, 'epoch': 1.33}

 13%|█▎        | 97/730 [08:41<56:14,  5.33s/it]
                                                
{'debug/num_tok_total': 2891.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2891.0, 'debug/num_lat_loss': 1790.0, 'epoch': 1.33}

 13%|█▎        | 97/730 [08:43<56:14,  5.33s/it]
                                                
{'train/ce_loss': 4.59375, 'train/diffusion_loss': 0.3672471344470978, 'epoch': 1.33}

 13%|█▎        | 97/730 [08:43<56:14,  5.33s/it]
                                                
{'train/learning_rate_real': 2.4314158467579544e-05, 'epoch': 1.33}

 13%|█▎        | 97/730 [08:43<56:14,  5.33s/it]
                                                
{'debug/num_tok_total': 2221.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2221.0, 'debug/num_lat_loss': 1795.0, 'epoch': 1.33}

 13%|█▎        | 97/730 [08:44<56:14,  5.33s/it]
                                                
{'train/ce_loss': 4.40625, 'train/diffusion_loss': 0.5292556881904602, 'epoch': 1.33}

 13%|█▎        | 97/730 [08:44<56:14,  5.33s/it]
                                                
{'train/learning_rate_real': 2.4314158467579544e-05, 'epoch': 1.33}

 13%|█▎        | 97/730 [08:44<56:14,  5.33s/it]
 13%|█▎        | 98/730 [08:44<55:35,  5.28s/it]
                                                
{'debug/num_tok_total': 2425.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2425.0, 'debug/num_lat_loss': 1783.0, 'epoch': 1.34}

 13%|█▎        | 98/730 [08:45<55:35,  5.28s/it]
                                                
{'train/ce_loss': 4.5, 'train/diffusion_loss': 0.43493154644966125, 'epoch': 1.34}

 13%|█▎        | 98/730 [08:45<55:35,  5.28s/it]
                                                
{'train/learning_rate_real': 2.429592222295441e-05, 'epoch': 1.34}

 13%|█▎        | 98/730 [08:45<55:35,  5.28s/it]
                                                
{'debug/num_tok_total': 2475.0, 'debug/num_tok_loss': 1622.0, 'debug/num_lat_total': 2475.0, 'debug/num_lat_loss': 1622.0, 'epoch': 1.34}

 13%|█▎        | 98/730 [08:47<55:35,  5.28s/it]
                                                
{'train/ce_loss': 4.4375, 'train/diffusion_loss': 0.37283438444137573, 'epoch': 1.34}

 13%|█▎        | 98/730 [08:47<55:35,  5.28s/it]
                                                
{'train/learning_rate_real': 2.429592222295441e-05, 'epoch': 1.34}

 13%|█▎        | 98/730 [08:47<55:35,  5.28s/it]
                                                
{'debug/num_tok_total': 2631.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2631.0, 'debug/num_lat_loss': 1778.0, 'epoch': 1.34}

 13%|█▎        | 98/730 [08:48<55:35,  5.28s/it]
                                                
{'train/ce_loss': 4.4375, 'train/diffusion_loss': 0.4417295753955841, 'epoch': 1.34}

 13%|█▎        | 98/730 [08:48<55:35,  5.28s/it]
                                                
{'train/learning_rate_real': 2.429592222295441e-05, 'epoch': 1.34}

 13%|█▎        | 98/730 [08:48<55:35,  5.28s/it]
                                                
{'debug/num_tok_total': 2727.0, 'debug/num_tok_loss': 1644.0, 'debug/num_lat_total': 2727.0, 'debug/num_lat_loss': 1644.0, 'epoch': 1.34}

 13%|█▎        | 98/730 [08:49<55:35,  5.28s/it]
                                                
{'train/ce_loss': 4.40625, 'train/diffusion_loss': 0.3511442244052887, 'epoch': 1.34}

 13%|█▎        | 98/730 [08:49<55:35,  5.28s/it]
                                                
{'train/learning_rate_real': 2.429592222295441e-05, 'epoch': 1.34}

 13%|█▎        | 98/730 [08:49<55:35,  5.28s/it]
 14%|█▎        | 99/730 [08:50<55:49,  5.31s/it]
                                                
{'debug/num_tok_total': 2841.0, 'debug/num_tok_loss': 1737.0, 'debug/num_lat_total': 2841.0, 'debug/num_lat_loss': 1737.0, 'epoch': 1.36}

 14%|█▎        | 99/730 [08:51<55:49,  5.31s/it]
                                                
{'train/ce_loss': 4.59375, 'train/diffusion_loss': 0.36495158076286316, 'epoch': 1.36}

 14%|█▎        | 99/730 [08:51<55:49,  5.31s/it]
                                                
{'train/learning_rate_real': 2.42774537236793e-05, 'epoch': 1.36}

 14%|█▎        | 99/730 [08:51<55:49,  5.31s/it]
                                                
{'debug/num_tok_total': 2005.0, 'debug/num_tok_loss': 1436.0, 'debug/num_lat_total': 2005.0, 'debug/num_lat_loss': 1436.0, 'epoch': 1.36}

 14%|█▎        | 99/730 [08:52<55:49,  5.31s/it]
                                                
{'train/ce_loss': 4.625, 'train/diffusion_loss': 0.4353146255016327, 'epoch': 1.36}

 14%|█▎        | 99/730 [08:52<55:49,  5.31s/it]
                                                
{'train/learning_rate_real': 2.42774537236793e-05, 'epoch': 1.36}

 14%|█▎        | 99/730 [08:52<55:49,  5.31s/it]
                                                
{'debug/num_tok_total': 2501.0, 'debug/num_tok_loss': 1633.0, 'debug/num_lat_total': 2501.0, 'debug/num_lat_loss': 1633.0, 'epoch': 1.36}

 14%|█▎        | 99/730 [08:53<55:49,  5.31s/it]
                                                
{'train/ce_loss': 4.34375, 'train/diffusion_loss': 0.3748423457145691, 'epoch': 1.36}

 14%|█▎        | 99/730 [08:53<55:49,  5.31s/it]
                                                
{'train/learning_rate_real': 2.42774537236793e-05, 'epoch': 1.36}

 14%|█▎        | 99/730 [08:53<55:49,  5.31s/it]
                                                
{'debug/num_tok_total': 2891.0, 'debug/num_tok_loss': 1800.0, 'debug/num_lat_total': 2891.0, 'debug/num_lat_loss': 1800.0, 'epoch': 1.36}

 14%|█▎        | 99/730 [08:55<55:49,  5.31s/it]
                                                
{'train/ce_loss': 4.375, 'train/diffusion_loss': 0.3602873682975769, 'epoch': 1.36}

 14%|█▎        | 99/730 [08:55<55:49,  5.31s/it]
                                                
{'train/learning_rate_real': 2.42774537236793e-05, 'epoch': 1.36}

 14%|█▎        | 99/730 [08:55<55:49,  5.31s/it]03/16/2026 06:56:27 - INFO - __main__ - LoRA debug step 100: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 14%|█▎        | 100/730 [08:55<55:59,  5.33s/it]
                                                 
{'loss': 2.983, 'grad_norm': 2.010390281677246, 'learning_rate': 2.42774537236793e-05, 'epoch': 1.37}

 14%|█▎        | 100/730 [08:55<55:59,  5.33s/it]
                                                 
{'debug/num_tok_total': 2860.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2860.0, 'debug/num_lat_loss': 1789.0, 'epoch': 1.37}

 14%|█▎        | 100/730 [08:56<55:59,  5.33s/it]
                                                 
{'train/ce_loss': 4.59375, 'train/diffusion_loss': 0.3802020847797394, 'epoch': 1.37}

 14%|█▎        | 100/730 [08:56<55:59,  5.33s/it]
                                                 
{'train/learning_rate_real': 2.425875333338792e-05, 'epoch': 1.37}

 14%|█▎        | 100/730 [08:56<55:59,  5.33s/it]
                                                 
{'debug/num_tok_total': 2640.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2640.0, 'debug/num_lat_loss': 1785.0, 'epoch': 1.37}

 14%|█▎        | 100/730 [08:57<55:59,  5.33s/it]
                                                 
{'train/ce_loss': 4.6875, 'train/diffusion_loss': 0.4197426736354828, 'epoch': 1.37}

 14%|█▎        | 100/730 [08:57<55:59,  5.33s/it]
                                                 
{'train/learning_rate_real': 2.425875333338792e-05, 'epoch': 1.37}

 14%|█▎        | 100/730 [08:57<55:59,  5.33s/it]
                                                 
{'debug/num_tok_total': 2512.0, 'debug/num_tok_loss': 1716.0, 'debug/num_lat_total': 2512.0, 'debug/num_lat_loss': 1716.0, 'epoch': 1.37}

 14%|█▎        | 100/730 [08:59<55:59,  5.33s/it]
                                                 
{'train/ce_loss': 4.53125, 'train/diffusion_loss': 0.4053674042224884, 'epoch': 1.37}

 14%|█▎        | 100/730 [08:59<55:59,  5.33s/it]
                                                 
{'train/learning_rate_real': 2.425875333338792e-05, 'epoch': 1.37}

 14%|█▎        | 100/730 [08:59<55:59,  5.33s/it]
                                                 
{'debug/num_tok_total': 2184.0, 'debug/num_tok_loss': 1746.0, 'debug/num_lat_total': 2184.0, 'debug/num_lat_loss': 1746.0, 'epoch': 1.37}

 14%|█▎        | 100/730 [09:00<55:59,  5.33s/it]
                                                 
{'train/ce_loss': 4.3125, 'train/diffusion_loss': 0.4971557855606079, 'epoch': 1.37}

 14%|█▎        | 100/730 [09:00<55:59,  5.33s/it]
                                                 
{'train/learning_rate_real': 2.425875333338792e-05, 'epoch': 1.37}

 14%|█▎        | 100/730 [09:00<55:59,  5.33s/it]
 14%|█▍        | 101/730 [09:00<55:19,  5.28s/it]
                                                 
{'debug/num_tok_total': 2669.0, 'debug/num_tok_loss': 1599.0, 'debug/num_lat_total': 2669.0, 'debug/num_lat_loss': 1599.0, 'epoch': 1.38}

 14%|█▍        | 101/730 [09:01<55:19,  5.28s/it]
                                                 
{'train/ce_loss': 4.53125, 'train/diffusion_loss': 0.34701186418533325, 'epoch': 1.38}

 14%|█▍        | 101/730 [09:01<55:19,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.4239821420279753e-05, 'epoch': 1.38}

 14%|█▍        | 101/730 [09:01<55:19,  5.28s/it]
                                                 
{'debug/num_tok_total': 2862.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2862.0, 'debug/num_lat_loss': 1776.0, 'epoch': 1.38}

 14%|█▍        | 101/730 [09:03<55:19,  5.28s/it]
                                                 
{'train/ce_loss': 4.78125, 'train/diffusion_loss': 0.3729672431945801, 'epoch': 1.38}

 14%|█▍        | 101/730 [09:03<55:19,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.4239821420279753e-05, 'epoch': 1.38}

 14%|█▍        | 101/730 [09:03<55:19,  5.28s/it]
                                                 
{'debug/num_tok_total': 2978.0, 'debug/num_tok_loss': 1629.0, 'debug/num_lat_total': 2978.0, 'debug/num_lat_loss': 1629.0, 'epoch': 1.38}

 14%|█▍        | 101/730 [09:04<55:19,  5.28s/it]
                                                 
{'train/ce_loss': 4.4375, 'train/diffusion_loss': 0.2844712734222412, 'epoch': 1.38}

 14%|█▍        | 101/730 [09:04<55:19,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.4239821420279753e-05, 'epoch': 1.38}

 14%|█▍        | 101/730 [09:04<55:19,  5.28s/it]
                                                 
{'debug/num_tok_total': 2636.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2636.0, 'debug/num_lat_loss': 1776.0, 'epoch': 1.38}

 14%|█▍        | 101/730 [09:05<55:19,  5.28s/it]
                                                 
{'train/ce_loss': 4.375, 'train/diffusion_loss': 0.40753600001335144, 'epoch': 1.38}

 14%|█▍        | 101/730 [09:05<55:19,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.4239821420279753e-05, 'epoch': 1.38}

 14%|█▍        | 101/730 [09:05<55:19,  5.28s/it]
 14%|█▍        | 102/730 [09:06<56:26,  5.39s/it]
                                                 
{'debug/num_tok_total': 3038.0, 'debug/num_tok_loss': 1762.0, 'debug/num_lat_total': 3038.0, 'debug/num_lat_loss': 1762.0, 'epoch': 1.4}

 14%|█▍        | 102/730 [09:07<56:26,  5.39s/it]
                                                 
{'train/ce_loss': 4.53125, 'train/diffusion_loss': 0.3289642035961151, 'epoch': 1.4}

 14%|█▍        | 102/730 [09:07<56:26,  5.39s/it]
                                                 
{'train/learning_rate_real': 2.422065835711285e-05, 'epoch': 1.4}

 14%|█▍        | 102/730 [09:07<56:26,  5.39s/it]
                                                 
{'debug/num_tok_total': 2665.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2665.0, 'debug/num_lat_loss': 1795.0, 'epoch': 1.4}

 14%|█▍        | 102/730 [09:08<56:26,  5.39s/it]
                                                 
{'train/ce_loss': 4.40625, 'train/diffusion_loss': 0.3820474445819855, 'epoch': 1.4}

 14%|█▍        | 102/730 [09:08<56:26,  5.39s/it]
                                                 
{'train/learning_rate_real': 2.422065835711285e-05, 'epoch': 1.4}

 14%|█▍        | 102/730 [09:08<56:26,  5.39s/it]
                                                 
{'debug/num_tok_total': 2671.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2671.0, 'debug/num_lat_loss': 1795.0, 'epoch': 1.4}

 14%|█▍        | 102/730 [09:10<56:26,  5.39s/it]
                                                 
{'train/ce_loss': 4.46875, 'train/diffusion_loss': 0.4218079447746277, 'epoch': 1.4}

 14%|█▍        | 102/730 [09:10<56:26,  5.39s/it]
                                                 
{'train/learning_rate_real': 2.422065835711285e-05, 'epoch': 1.4}

 14%|█▍        | 102/730 [09:10<56:26,  5.39s/it]
                                                 
{'debug/num_tok_total': 2416.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2416.0, 'debug/num_lat_loss': 1772.0, 'epoch': 1.4}

 14%|█▍        | 102/730 [09:11<56:26,  5.39s/it]
                                                 
{'train/ce_loss': 4.3125, 'train/diffusion_loss': 0.4562211334705353, 'epoch': 1.4}

 14%|█▍        | 102/730 [09:11<56:26,  5.39s/it]
                                                 
{'train/learning_rate_real': 2.422065835711285e-05, 'epoch': 1.4}

 14%|█▍        | 102/730 [09:11<56:26,  5.39s/it]
 14%|█▍        | 103/730 [09:11<56:06,  5.37s/it]
                                                 
{'debug/num_tok_total': 2874.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2874.0, 'debug/num_lat_loss': 1792.0, 'epoch': 1.41}

 14%|█▍        | 103/730 [09:12<56:06,  5.37s/it]
                                                 
{'train/ce_loss': 4.5625, 'train/diffusion_loss': 0.3860173523426056, 'epoch': 1.41}

 14%|█▍        | 103/730 [09:12<56:06,  5.37s/it]
                                                 
{'train/learning_rate_real': 2.4201264521196455e-05, 'epoch': 1.41}

 14%|█▍        | 103/730 [09:12<56:06,  5.37s/it]
                                                 
{'debug/num_tok_total': 2442.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2442.0, 'debug/num_lat_loss': 1792.0, 'epoch': 1.41}

 14%|█▍        | 103/730 [09:13<56:06,  5.37s/it]
                                                 
{'train/ce_loss': 4.21875, 'train/diffusion_loss': 0.45044654607772827, 'epoch': 1.41}

 14%|█▍        | 103/730 [09:13<56:06,  5.37s/it]
                                                 
{'train/learning_rate_real': 2.4201264521196455e-05, 'epoch': 1.41}

 14%|█▍        | 103/730 [09:13<56:06,  5.37s/it]
                                                 
{'debug/num_tok_total': 2619.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2619.0, 'debug/num_lat_loss': 1770.0, 'epoch': 1.41}

 14%|█▍        | 103/730 [09:15<56:06,  5.37s/it]
                                                 
{'train/ce_loss': 4.46875, 'train/diffusion_loss': 0.4299888014793396, 'epoch': 1.41}

 14%|█▍        | 103/730 [09:15<56:06,  5.37s/it]
                                                 
{'train/learning_rate_real': 2.4201264521196455e-05, 'epoch': 1.41}

 14%|█▍        | 103/730 [09:15<56:06,  5.37s/it]
                                                 
{'debug/num_tok_total': 2638.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2638.0, 'debug/num_lat_loss': 1782.0, 'epoch': 1.41}

 14%|█▍        | 103/730 [09:16<56:06,  5.37s/it]
                                                 
{'train/ce_loss': 4.4375, 'train/diffusion_loss': 0.3971606194972992, 'epoch': 1.41}

 14%|█▍        | 103/730 [09:16<56:06,  5.37s/it]
                                                 
{'train/learning_rate_real': 2.4201264521196455e-05, 'epoch': 1.41}

 14%|█▍        | 103/730 [09:16<56:06,  5.37s/it]
 14%|█▍        | 104/730 [09:16<55:21,  5.31s/it]
                                                 
{'debug/num_tok_total': 2223.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2223.0, 'debug/num_lat_loss': 1792.0, 'epoch': 1.42}

 14%|█▍        | 104/730 [09:17<55:21,  5.31s/it]
                                                 
{'train/ce_loss': 4.15625, 'train/diffusion_loss': 0.48428550362586975, 'epoch': 1.42}

 14%|█▍        | 104/730 [09:17<55:21,  5.31s/it]
                                                 
{'train/learning_rate_real': 2.4181640294383585e-05, 'epoch': 1.42}

 14%|█▍        | 104/730 [09:17<55:21,  5.31s/it]
                                                 
{'debug/num_tok_total': 2539.0, 'debug/num_tok_loss': 1674.0, 'debug/num_lat_total': 2539.0, 'debug/num_lat_loss': 1674.0, 'epoch': 1.42}

 14%|█▍        | 104/730 [09:19<55:21,  5.31s/it]
                                                 
{'train/ce_loss': 4.40625, 'train/diffusion_loss': 0.3819606602191925, 'epoch': 1.42}

 14%|█▍        | 104/730 [09:19<55:21,  5.31s/it]
                                                 
{'train/learning_rate_real': 2.4181640294383585e-05, 'epoch': 1.42}

 14%|█▍        | 104/730 [09:19<55:21,  5.31s/it]
                                                 
{'debug/num_tok_total': 2253.0, 'debug/num_tok_loss': 1690.0, 'debug/num_lat_total': 2253.0, 'debug/num_lat_loss': 1690.0, 'epoch': 1.42}

 14%|█▍        | 104/730 [09:20<55:21,  5.31s/it]
                                                 
{'train/ce_loss': 4.3125, 'train/diffusion_loss': 0.4630298614501953, 'epoch': 1.42}

 14%|█▍        | 104/730 [09:20<55:21,  5.31s/it]
                                                 
{'train/learning_rate_real': 2.4181640294383585e-05, 'epoch': 1.42}

 14%|█▍        | 104/730 [09:20<55:21,  5.31s/it]
                                                 
{'debug/num_tok_total': 2633.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2633.0, 'debug/num_lat_loss': 1771.0, 'epoch': 1.42}

 14%|█▍        | 104/730 [09:21<55:21,  5.31s/it]
                                                 
{'train/ce_loss': 4.34375, 'train/diffusion_loss': 0.3922845125198364, 'epoch': 1.42}

 14%|█▍        | 104/730 [09:21<55:21,  5.31s/it]
                                                 
{'train/learning_rate_real': 2.4181640294383585e-05, 'epoch': 1.42}

 14%|█▍        | 104/730 [09:21<55:21,  5.31s/it]
 14%|█▍        | 105/730 [09:22<55:00,  5.28s/it]
                                                 
{'debug/num_tok_total': 2897.0, 'debug/num_tok_loss': 1804.0, 'debug/num_lat_total': 2897.0, 'debug/num_lat_loss': 1804.0, 'epoch': 1.44}

 14%|█▍        | 105/730 [09:23<55:00,  5.28s/it]
                                                 
{'train/ce_loss': 4.34375, 'train/diffusion_loss': 0.37782275676727295, 'epoch': 1.44}

 14%|█▍        | 105/730 [09:23<55:00,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.4161786063063536e-05, 'epoch': 1.44}

 14%|█▍        | 105/730 [09:23<55:00,  5.28s/it]
                                                 
{'debug/num_tok_total': 2358.0, 'debug/num_tok_loss': 1712.0, 'debug/num_lat_total': 2358.0, 'debug/num_lat_loss': 1712.0, 'epoch': 1.44}

 14%|█▍        | 105/730 [09:24<55:00,  5.28s/it]
                                                 
{'train/ce_loss': 4.1875, 'train/diffusion_loss': 0.4383047819137573, 'epoch': 1.44}

 14%|█▍        | 105/730 [09:24<55:00,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.4161786063063536e-05, 'epoch': 1.44}

 14%|█▍        | 105/730 [09:24<55:00,  5.28s/it]
                                                 
{'debug/num_tok_total': 2836.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2836.0, 'debug/num_lat_loss': 1776.0, 'epoch': 1.44}

 14%|█▍        | 105/730 [09:25<55:00,  5.28s/it]
                                                 
{'train/ce_loss': 4.21875, 'train/diffusion_loss': 0.3715423047542572, 'epoch': 1.44}

 14%|█▍        | 105/730 [09:25<55:00,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.4161786063063536e-05, 'epoch': 1.44}

 14%|█▍        | 105/730 [09:25<55:00,  5.28s/it]
                                                 
{'debug/num_tok_total': 2647.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2647.0, 'debug/num_lat_loss': 1795.0, 'epoch': 1.44}

 14%|█▍        | 105/730 [09:27<55:00,  5.28s/it]
                                                 
{'train/ce_loss': 4.3125, 'train/diffusion_loss': 0.4179925322532654, 'epoch': 1.44}

 14%|█▍        | 105/730 [09:27<55:00,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.4161786063063536e-05, 'epoch': 1.44}

 14%|█▍        | 105/730 [09:27<55:00,  5.28s/it]
 15%|█▍        | 106/730 [09:27<55:31,  5.34s/it]
                                                 
{'debug/num_tok_total': 2265.0, 'debug/num_tok_loss': 1625.0, 'debug/num_lat_total': 2265.0, 'debug/num_lat_loss': 1625.0, 'epoch': 1.45}

 15%|█▍        | 106/730 [09:28<55:31,  5.34s/it]
                                                 
{'train/ce_loss': 4.28125, 'train/diffusion_loss': 0.4229045808315277, 'epoch': 1.45}

 15%|█▍        | 106/730 [09:28<55:31,  5.34s/it]
                                                 
{'train/learning_rate_real': 2.4141702218154232e-05, 'epoch': 1.45}

 15%|█▍        | 106/730 [09:28<55:31,  5.34s/it]
                                                 
{'debug/num_tok_total': 2211.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 2211.0, 'debug/num_lat_loss': 1768.0, 'epoch': 1.45}

 15%|█▍        | 106/730 [09:29<55:31,  5.34s/it]
                                                 
{'train/ce_loss': 4.25, 'train/diffusion_loss': 0.49752187728881836, 'epoch': 1.45}

 15%|█▍        | 106/730 [09:29<55:31,  5.34s/it]
                                                 
{'train/learning_rate_real': 2.4141702218154232e-05, 'epoch': 1.45}

 15%|█▍        | 106/730 [09:29<55:31,  5.34s/it]
                                                 
{'debug/num_tok_total': 2611.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 2611.0, 'debug/num_lat_loss': 1768.0, 'epoch': 1.45}

 15%|█▍        | 106/730 [09:30<55:31,  5.34s/it]
                                                 
{'train/ce_loss': 4.1875, 'train/diffusion_loss': 0.4122409224510193, 'epoch': 1.45}

 15%|█▍        | 106/730 [09:30<55:31,  5.34s/it]
                                                 
{'train/learning_rate_real': 2.4141702218154232e-05, 'epoch': 1.45}

 15%|█▍        | 106/730 [09:30<55:31,  5.34s/it]
                                                 
{'debug/num_tok_total': 2434.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2434.0, 'debug/num_lat_loss': 1790.0, 'epoch': 1.45}

 15%|█▍        | 106/730 [09:32<55:31,  5.34s/it]
                                                 
{'train/ce_loss': 4.25, 'train/diffusion_loss': 0.48291388154029846, 'epoch': 1.45}

 15%|█▍        | 106/730 [09:32<55:31,  5.34s/it]
                                                 
{'train/learning_rate_real': 2.4141702218154232e-05, 'epoch': 1.45}

 15%|█▍        | 106/730 [09:32<55:31,  5.34s/it]
 15%|█▍        | 107/730 [09:32<54:35,  5.26s/it]
                                                 
{'debug/num_tok_total': 2204.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2204.0, 'debug/num_lat_loss': 1766.0, 'epoch': 1.47}

 15%|█▍        | 107/730 [09:33<54:35,  5.26s/it]
                                                 
{'train/ce_loss': 4.28125, 'train/diffusion_loss': 0.48381802439689636, 'epoch': 1.47}

 15%|█▍        | 107/730 [09:33<54:35,  5.26s/it]
                                                 
{'train/learning_rate_real': 2.4121389155094564e-05, 'epoch': 1.47}

 15%|█▍        | 107/730 [09:33<54:35,  5.26s/it]
                                                 
{'debug/num_tok_total': 2371.0, 'debug/num_tok_loss': 1645.0, 'debug/num_lat_total': 2371.0, 'debug/num_lat_loss': 1645.0, 'epoch': 1.47}

 15%|█▍        | 107/730 [09:34<54:35,  5.26s/it]
                                                 
{'train/ce_loss': 4.53125, 'train/diffusion_loss': 0.4406833052635193, 'epoch': 1.47}

 15%|█▍        | 107/730 [09:34<54:35,  5.26s/it]
                                                 
{'train/learning_rate_real': 2.4121389155094564e-05, 'epoch': 1.47}

 15%|█▍        | 107/730 [09:34<54:35,  5.26s/it]
                                                 
{'debug/num_tok_total': 2371.0, 'debug/num_tok_loss': 1723.0, 'debug/num_lat_total': 2371.0, 'debug/num_lat_loss': 1723.0, 'epoch': 1.47}

 15%|█▍        | 107/730 [09:35<54:35,  5.26s/it]
                                                 
{'train/ce_loss': 4.25, 'train/diffusion_loss': 0.4781853258609772, 'epoch': 1.47}

 15%|█▍        | 107/730 [09:35<54:35,  5.26s/it]
                                                 
{'train/learning_rate_real': 2.4121389155094564e-05, 'epoch': 1.47}

 15%|█▍        | 107/730 [09:35<54:35,  5.26s/it]
                                                 
{'debug/num_tok_total': 2843.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 2843.0, 'debug/num_lat_loss': 1769.0, 'epoch': 1.47}

 15%|█▍        | 107/730 [09:37<54:35,  5.26s/it]
                                                 
{'train/ce_loss': 4.21875, 'train/diffusion_loss': 0.3677610754966736, 'epoch': 1.47}

 15%|█▍        | 107/730 [09:37<54:35,  5.26s/it]
                                                 
{'train/learning_rate_real': 2.4121389155094564e-05, 'epoch': 1.47}

 15%|█▍        | 107/730 [09:37<54:35,  5.26s/it]
 15%|█▍        | 108/730 [09:37<54:02,  5.21s/it]
                                                 
{'debug/num_tok_total': 2864.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2864.0, 'debug/num_lat_loss': 1788.0, 'epoch': 1.48}

 15%|█▍        | 108/730 [09:38<54:02,  5.21s/it]
                                                 
{'train/ce_loss': 4.25, 'train/diffusion_loss': 0.35337358713150024, 'epoch': 1.48}

 15%|█▍        | 108/730 [09:38<54:02,  5.21s/it]
                                                 
{'train/learning_rate_real': 2.410084727383659e-05, 'epoch': 1.48}

 15%|█▍        | 108/730 [09:38<54:02,  5.21s/it]
                                                 
{'debug/num_tok_total': 2456.0, 'debug/num_tok_loss': 1804.0, 'debug/num_lat_total': 2456.0, 'debug/num_lat_loss': 1804.0, 'epoch': 1.48}

 15%|█▍        | 108/730 [09:40<54:02,  5.21s/it]
                                                 
{'train/ce_loss': 4.125, 'train/diffusion_loss': 0.47408199310302734, 'epoch': 1.48}

 15%|█▍        | 108/730 [09:40<54:02,  5.21s/it]
                                                 
{'train/learning_rate_real': 2.410084727383659e-05, 'epoch': 1.48}

 15%|█▍        | 108/730 [09:40<54:02,  5.21s/it]
                                                 
{'debug/num_tok_total': 2409.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2409.0, 'debug/num_lat_loss': 1784.0, 'epoch': 1.48}

 15%|█▍        | 108/730 [09:41<54:02,  5.21s/it]
                                                 
{'train/ce_loss': 4.25, 'train/diffusion_loss': 0.460326611995697, 'epoch': 1.48}

 15%|█▍        | 108/730 [09:41<54:02,  5.21s/it]
                                                 
{'train/learning_rate_real': 2.410084727383659e-05, 'epoch': 1.48}

 15%|█▍        | 108/730 [09:41<54:02,  5.21s/it]
                                                 
{'debug/num_tok_total': 3324.0, 'debug/num_tok_loss': 1800.0, 'debug/num_lat_total': 3324.0, 'debug/num_lat_loss': 1800.0, 'epoch': 1.48}

 15%|█▍        | 108/730 [09:42<54:02,  5.21s/it]
                                                 
{'train/ce_loss': 4.3125, 'train/diffusion_loss': 0.26926982402801514, 'epoch': 1.48}

 15%|█▍        | 108/730 [09:42<54:02,  5.21s/it]
                                                 
{'train/learning_rate_real': 2.410084727383659e-05, 'epoch': 1.48}

 15%|█▍        | 108/730 [09:42<54:02,  5.21s/it]
 15%|█▍        | 109/730 [09:43<54:29,  5.26s/it]
                                                 
{'debug/num_tok_total': 2652.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2652.0, 'debug/num_lat_loss': 1791.0, 'epoch': 1.49}

 15%|█▍        | 109/730 [09:44<54:29,  5.26s/it]
                                                 
{'train/ce_loss': 4.1875, 'train/diffusion_loss': 0.44318896532058716, 'epoch': 1.49}

 15%|█▍        | 109/730 [09:44<54:29,  5.26s/it]
                                                 
{'train/learning_rate_real': 2.4080076978837658e-05, 'epoch': 1.49}

 15%|█▍        | 109/730 [09:44<54:29,  5.26s/it]
                                                 
{'debug/num_tok_total': 2879.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2879.0, 'debug/num_lat_loss': 1795.0, 'epoch': 1.49}

 15%|█▍        | 109/730 [09:45<54:29,  5.26s/it]
                                                 
{'train/ce_loss': 4.15625, 'train/diffusion_loss': 0.3627161979675293, 'epoch': 1.49}

 15%|█▍        | 109/730 [09:45<54:29,  5.26s/it]
                                                 
{'train/learning_rate_real': 2.4080076978837658e-05, 'epoch': 1.49}

 15%|█▍        | 109/730 [09:45<54:29,  5.26s/it]
                                                 
{'debug/num_tok_total': 2070.0, 'debug/num_tok_loss': 1633.0, 'debug/num_lat_total': 2070.0, 'debug/num_lat_loss': 1633.0, 'epoch': 1.49}

 15%|█▍        | 109/730 [09:46<54:29,  5.26s/it]
                                                 
{'train/ce_loss': 4.25, 'train/diffusion_loss': 0.47383245825767517, 'epoch': 1.49}

 15%|█▍        | 109/730 [09:46<54:29,  5.26s/it]
                                                 
{'train/learning_rate_real': 2.4080076978837658e-05, 'epoch': 1.49}

 15%|█▍        | 109/730 [09:46<54:29,  5.26s/it]
                                                 
{'debug/num_tok_total': 2443.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2443.0, 'debug/num_lat_loss': 1792.0, 'epoch': 1.49}

 15%|█▍        | 109/730 [09:47<54:29,  5.26s/it]
                                                 
{'train/ce_loss': 4.46875, 'train/diffusion_loss': 0.4685288071632385, 'epoch': 1.49}

 15%|█▍        | 109/730 [09:47<54:29,  5.26s/it]
                                                 
{'train/learning_rate_real': 2.4080076978837658e-05, 'epoch': 1.49}

 15%|█▍        | 109/730 [09:47<54:29,  5.26s/it]03/16/2026 06:57:20 - INFO - __main__ - LoRA debug step 110: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 15%|█▌        | 110/730 [09:48<54:27,  5.27s/it]
                                                 
{'loss': 3.0191, 'grad_norm': 1.5252450704574585, 'learning_rate': 2.4080076978837658e-05, 'epoch': 1.51}

 15%|█▌        | 110/730 [09:48<54:27,  5.27s/it]
                                                 
{'debug/num_tok_total': 2442.0, 'debug/num_tok_loss': 1806.0, 'debug/num_lat_total': 2442.0, 'debug/num_lat_loss': 1806.0, 'epoch': 1.51}

 15%|█▌        | 110/730 [09:49<54:27,  5.27s/it]
                                                 
{'train/ce_loss': 4.5, 'train/diffusion_loss': 0.45642104744911194, 'epoch': 1.51}

 15%|█▌        | 110/730 [09:49<54:27,  5.27s/it]
                                                 
{'train/learning_rate_real': 2.405907867905245e-05, 'epoch': 1.51}

 15%|█▌        | 110/730 [09:49<54:27,  5.27s/it]
                                                 
{'debug/num_tok_total': 2425.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2425.0, 'debug/num_lat_loss': 1785.0, 'epoch': 1.51}

 15%|█▌        | 110/730 [09:50<54:27,  5.27s/it]
                                                 
{'train/ce_loss': 4.1875, 'train/diffusion_loss': 0.48576435446739197, 'epoch': 1.51}

 15%|█▌        | 110/730 [09:50<54:27,  5.27s/it]
                                                 
{'train/learning_rate_real': 2.405907867905245e-05, 'epoch': 1.51}

 15%|█▌        | 110/730 [09:50<54:27,  5.27s/it]
                                                 
{'debug/num_tok_total': 2746.0, 'debug/num_tok_loss': 1679.0, 'debug/num_lat_total': 2746.0, 'debug/num_lat_loss': 1679.0, 'epoch': 1.51}

 15%|█▌        | 110/730 [09:51<54:27,  5.27s/it]
                                                 
{'train/ce_loss': 4.125, 'train/diffusion_loss': 0.3542007803916931, 'epoch': 1.51}

 15%|█▌        | 110/730 [09:51<54:27,  5.27s/it]
                                                 
{'train/learning_rate_real': 2.405907867905245e-05, 'epoch': 1.51}

 15%|█▌        | 110/730 [09:51<54:27,  5.27s/it]
                                                 
{'debug/num_tok_total': 2609.0, 'debug/num_tok_loss': 1758.0, 'debug/num_lat_total': 2609.0, 'debug/num_lat_loss': 1758.0, 'epoch': 1.51}

 15%|█▌        | 110/730 [09:53<54:27,  5.27s/it]
                                                 
{'train/ce_loss': 4.0, 'train/diffusion_loss': 0.4068741798400879, 'epoch': 1.51}

 15%|█▌        | 110/730 [09:53<54:27,  5.27s/it]
                                                 
{'train/learning_rate_real': 2.405907867905245e-05, 'epoch': 1.51}

 15%|█▌        | 110/730 [09:53<54:27,  5.27s/it]
 15%|█▌        | 111/730 [09:53<54:41,  5.30s/it]
                                                 
{'debug/num_tok_total': 2577.0, 'debug/num_tok_loss': 1738.0, 'debug/num_lat_total': 2577.0, 'debug/num_lat_loss': 1738.0, 'epoch': 1.52}

 15%|█▌        | 111/730 [09:54<54:41,  5.30s/it]
                                                 
{'train/ce_loss': 4.0, 'train/diffusion_loss': 0.41163885593414307, 'epoch': 1.52}

 15%|█▌        | 111/730 [09:54<54:41,  5.30s/it]
                                                 
{'train/learning_rate_real': 2.403785278792492e-05, 'epoch': 1.52}

 15%|█▌        | 111/730 [09:54<54:41,  5.30s/it]
                                                 
{'debug/num_tok_total': 2848.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2848.0, 'debug/num_lat_loss': 1777.0, 'epoch': 1.52}

 15%|█▌        | 111/730 [09:56<54:41,  5.30s/it]
                                                 
{'train/ce_loss': 4.21875, 'train/diffusion_loss': 0.36420491337776184, 'epoch': 1.52}

 15%|█▌        | 111/730 [09:56<54:41,  5.30s/it]
                                                 
{'train/learning_rate_real': 2.403785278792492e-05, 'epoch': 1.52}

 15%|█▌        | 111/730 [09:56<54:41,  5.30s/it]
                                                 
{'debug/num_tok_total': 2596.0, 'debug/num_tok_loss': 1747.0, 'debug/num_lat_total': 2596.0, 'debug/num_lat_loss': 1747.0, 'epoch': 1.52}

 15%|█▌        | 111/730 [09:57<54:41,  5.30s/it]
                                                 
{'train/ce_loss': 4.46875, 'train/diffusion_loss': 0.3811115622520447, 'epoch': 1.52}

 15%|█▌        | 111/730 [09:57<54:41,  5.30s/it]
                                                 
{'train/learning_rate_real': 2.403785278792492e-05, 'epoch': 1.52}

 15%|█▌        | 111/730 [09:57<54:41,  5.30s/it]
                                                 
{'debug/num_tok_total': 3021.0, 'debug/num_tok_loss': 1749.0, 'debug/num_lat_total': 3021.0, 'debug/num_lat_loss': 1749.0, 'epoch': 1.52}

 15%|█▌        | 111/730 [09:58<54:41,  5.30s/it]
                                                 
{'train/ce_loss': 4.28125, 'train/diffusion_loss': 0.33210745453834534, 'epoch': 1.52}

 15%|█▌        | 111/730 [09:58<54:41,  5.30s/it]
                                                 
{'train/learning_rate_real': 2.403785278792492e-05, 'epoch': 1.52}

 15%|█▌        | 111/730 [09:58<54:41,  5.30s/it]
 15%|█▌        | 112/730 [09:59<54:56,  5.33s/it]
                                                 
{'debug/num_tok_total': 2830.0, 'debug/num_tok_loss': 1759.0, 'debug/num_lat_total': 2830.0, 'debug/num_lat_loss': 1759.0, 'epoch': 1.53}

 15%|█▌        | 112/730 [10:00<54:56,  5.33s/it]
                                                 
{'train/ce_loss': 4.0625, 'train/diffusion_loss': 0.35382625460624695, 'epoch': 1.53}

 15%|█▌        | 112/730 [10:00<54:56,  5.33s/it]
                                                 
{'train/learning_rate_real': 2.401639972338016e-05, 'epoch': 1.53}

 15%|█▌        | 112/730 [10:00<54:56,  5.33s/it]
                                                 
{'debug/num_tok_total': 2603.0, 'debug/num_tok_loss': 1756.0, 'debug/num_lat_total': 2603.0, 'debug/num_lat_loss': 1756.0, 'epoch': 1.53}

 15%|█▌        | 112/730 [10:01<54:56,  5.33s/it]
                                                 
{'train/ce_loss': 4.0, 'train/diffusion_loss': 0.4346335232257843, 'epoch': 1.53}

 15%|█▌        | 112/730 [10:01<54:56,  5.33s/it]
                                                 
{'train/learning_rate_real': 2.401639972338016e-05, 'epoch': 1.53}

 15%|█▌        | 112/730 [10:01<54:56,  5.33s/it]
                                                 
{'debug/num_tok_total': 3095.0, 'debug/num_tok_loss': 1800.0, 'debug/num_lat_total': 3095.0, 'debug/num_lat_loss': 1800.0, 'epoch': 1.53}

 15%|█▌        | 112/730 [10:03<54:56,  5.33s/it]
                                                 
{'train/ce_loss': 4.0, 'train/diffusion_loss': 0.331741064786911, 'epoch': 1.53}

 15%|█▌        | 112/730 [10:03<54:56,  5.33s/it]
                                                 
{'train/learning_rate_real': 2.401639972338016e-05, 'epoch': 1.53}

 15%|█▌        | 112/730 [10:03<54:56,  5.33s/it]
                                                 
{'debug/num_tok_total': 3063.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 3063.0, 'debug/num_lat_loss': 1773.0, 'epoch': 1.53}

 15%|█▌        | 112/730 [10:04<54:56,  5.33s/it]
                                                 
{'train/ce_loss': 4.21875, 'train/diffusion_loss': 0.3286246955394745, 'epoch': 1.53}

 15%|█▌        | 112/730 [10:04<54:56,  5.33s/it]
                                                 
{'train/learning_rate_real': 2.401639972338016e-05, 'epoch': 1.53}

 15%|█▌        | 112/730 [10:04<54:56,  5.33s/it]
 15%|█▌        | 113/730 [10:04<55:49,  5.43s/it]
                                                 
{'debug/num_tok_total': 2411.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 2411.0, 'debug/num_lat_loss': 1768.0, 'epoch': 1.55}

 15%|█▌        | 113/730 [10:05<55:49,  5.43s/it]
                                                 
{'train/ce_loss': 3.84375, 'train/diffusion_loss': 0.44130316376686096, 'epoch': 1.55}

 15%|█▌        | 113/730 [10:05<55:49,  5.43s/it]
                                                 
{'train/learning_rate_real': 2.399471990781619e-05, 'epoch': 1.55}

 15%|█▌        | 113/730 [10:05<55:49,  5.43s/it]
                                                 
{'debug/num_tok_total': 2801.0, 'debug/num_tok_loss': 1744.0, 'debug/num_lat_total': 2801.0, 'debug/num_lat_loss': 1744.0, 'epoch': 1.55}

 15%|█▌        | 113/730 [10:07<55:49,  5.43s/it]
                                                 
{'train/ce_loss': 4.03125, 'train/diffusion_loss': 0.3459753096103668, 'epoch': 1.55}

 15%|█▌        | 113/730 [10:07<55:49,  5.43s/it]
                                                 
{'train/learning_rate_real': 2.399471990781619e-05, 'epoch': 1.55}

 15%|█▌        | 113/730 [10:07<55:49,  5.43s/it]
                                                 
{'debug/num_tok_total': 2598.0, 'debug/num_tok_loss': 1744.0, 'debug/num_lat_total': 2598.0, 'debug/num_lat_loss': 1744.0, 'epoch': 1.55}

 15%|█▌        | 113/730 [10:08<55:49,  5.43s/it]
                                                 
{'train/ce_loss': 4.21875, 'train/diffusion_loss': 0.4222409129142761, 'epoch': 1.55}

 15%|█▌        | 113/730 [10:08<55:49,  5.43s/it]
                                                 
{'train/learning_rate_real': 2.399471990781619e-05, 'epoch': 1.55}

 15%|█▌        | 113/730 [10:08<55:49,  5.43s/it]
                                                 
{'debug/num_tok_total': 2874.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2874.0, 'debug/num_lat_loss': 1785.0, 'epoch': 1.55}

 15%|█▌        | 113/730 [10:09<55:49,  5.43s/it]
                                                 
{'train/ce_loss': 4.0625, 'train/diffusion_loss': 0.35831475257873535, 'epoch': 1.55}

 15%|█▌        | 113/730 [10:09<55:49,  5.43s/it]
                                                 
{'train/learning_rate_real': 2.399471990781619e-05, 'epoch': 1.55}

 15%|█▌        | 113/730 [10:09<55:49,  5.43s/it]
 16%|█▌        | 114/730 [10:10<55:44,  5.43s/it]
                                                 
{'debug/num_tok_total': 3056.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 3056.0, 'debug/num_lat_loss': 1764.0, 'epoch': 1.56}

 16%|█▌        | 114/730 [10:11<55:44,  5.43s/it]
                                                 
{'train/ce_loss': 4.15625, 'train/diffusion_loss': 0.2948022186756134, 'epoch': 1.56}

 16%|█▌        | 114/730 [10:11<55:44,  5.43s/it]
                                                 
{'train/learning_rate_real': 2.3972813768095583e-05, 'epoch': 1.56}

 16%|█▌        | 114/730 [10:11<55:44,  5.43s/it]
                                                 
{'debug/num_tok_total': 2659.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2659.0, 'debug/num_lat_loss': 1792.0, 'epoch': 1.56}

 16%|█▌        | 114/730 [10:12<55:44,  5.43s/it]
                                                 
{'train/ce_loss': 4.21875, 'train/diffusion_loss': 0.4711344838142395, 'epoch': 1.56}

 16%|█▌        | 114/730 [10:12<55:44,  5.43s/it]
                                                 
{'train/learning_rate_real': 2.3972813768095583e-05, 'epoch': 1.56}

 16%|█▌        | 114/730 [10:12<55:44,  5.43s/it]
                                                 
{'debug/num_tok_total': 3543.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 3543.0, 'debug/num_lat_loss': 1801.0, 'epoch': 1.56}

 16%|█▌        | 114/730 [10:14<55:44,  5.43s/it]
                                                 
{'train/ce_loss': 4.0625, 'train/diffusion_loss': 0.23170864582061768, 'epoch': 1.56}

 16%|█▌        | 114/730 [10:14<55:44,  5.43s/it]
                                                 
{'train/learning_rate_real': 2.3972813768095583e-05, 'epoch': 1.56}

 16%|█▌        | 114/730 [10:14<55:44,  5.43s/it]
                                                 
{'debug/num_tok_total': 2682.0, 'debug/num_tok_loss': 1805.0, 'debug/num_lat_total': 2682.0, 'debug/num_lat_loss': 1805.0, 'epoch': 1.56}

 16%|█▌        | 114/730 [10:15<55:44,  5.43s/it]
                                                 
{'train/ce_loss': 4.0625, 'train/diffusion_loss': 0.40863677859306335, 'epoch': 1.56}

 16%|█▌        | 114/730 [10:15<55:44,  5.43s/it]
                                                 
{'train/learning_rate_real': 2.3972813768095583e-05, 'epoch': 1.56}

 16%|█▌        | 114/730 [10:15<55:44,  5.43s/it]
 16%|█▌        | 115/730 [10:15<56:11,  5.48s/it]
                                                 
{'debug/num_tok_total': 2399.0, 'debug/num_tok_loss': 1762.0, 'debug/num_lat_total': 2399.0, 'debug/num_lat_loss': 1762.0, 'epoch': 1.58}

 16%|█▌        | 115/730 [10:16<56:11,  5.48s/it]
                                                 
{'train/ce_loss': 3.71875, 'train/diffusion_loss': 0.4591188132762909, 'epoch': 1.58}

 16%|█▌        | 115/730 [10:16<56:11,  5.48s/it]
                                                 
{'train/learning_rate_real': 2.3950681735537135e-05, 'epoch': 1.58}

 16%|█▌        | 115/730 [10:16<56:11,  5.48s/it]
                                                 
{'debug/num_tok_total': 2499.0, 'debug/num_tok_loss': 1639.0, 'debug/num_lat_total': 2499.0, 'debug/num_lat_loss': 1639.0, 'epoch': 1.58}

 16%|█▌        | 115/730 [10:18<56:11,  5.48s/it]
                                                 
{'train/ce_loss': 3.953125, 'train/diffusion_loss': 0.39648905396461487, 'epoch': 1.58}

 16%|█▌        | 115/730 [10:18<56:11,  5.48s/it]
                                                 
{'train/learning_rate_real': 2.3950681735537135e-05, 'epoch': 1.58}

 16%|█▌        | 115/730 [10:18<56:11,  5.48s/it]
                                                 
{'debug/num_tok_total': 2213.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2213.0, 'debug/num_lat_loss': 1793.0, 'epoch': 1.58}

 16%|█▌        | 115/730 [10:19<56:11,  5.48s/it]
                                                 
{'train/ce_loss': 3.890625, 'train/diffusion_loss': 0.49079185724258423, 'epoch': 1.58}

 16%|█▌        | 115/730 [10:19<56:11,  5.48s/it]
                                                 
{'train/learning_rate_real': 2.3950681735537135e-05, 'epoch': 1.58}

 16%|█▌        | 115/730 [10:19<56:11,  5.48s/it]
                                                 
{'debug/num_tok_total': 2230.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2230.0, 'debug/num_lat_loss': 1793.0, 'epoch': 1.58}

 16%|█▌        | 115/730 [10:20<56:11,  5.48s/it]
                                                 
{'train/ce_loss': 4.03125, 'train/diffusion_loss': 0.49304887652397156, 'epoch': 1.58}

 16%|█▌        | 115/730 [10:20<56:11,  5.48s/it]
                                                 
{'train/learning_rate_real': 2.3950681735537135e-05, 'epoch': 1.58}

 16%|█▌        | 115/730 [10:20<56:11,  5.48s/it]
 16%|█▌        | 116/730 [10:20<54:41,  5.34s/it]
                                                 
{'debug/num_tok_total': 3094.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 3094.0, 'debug/num_lat_loss': 1793.0, 'epoch': 1.59}

 16%|█▌        | 116/730 [10:22<54:41,  5.34s/it]
                                                 
{'train/ce_loss': 3.859375, 'train/diffusion_loss': 0.3350035548210144, 'epoch': 1.59}

 16%|█▌        | 116/730 [10:22<54:41,  5.34s/it]
                                                 
{'train/learning_rate_real': 2.3928324245907324e-05, 'epoch': 1.59}

 16%|█▌        | 116/730 [10:22<54:41,  5.34s/it]
                                                 
{'debug/num_tok_total': 3081.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 3081.0, 'debug/num_lat_loss': 1783.0, 'epoch': 1.59}

 16%|█▌        | 116/730 [10:23<54:41,  5.34s/it]
                                                 
{'train/ce_loss': 3.921875, 'train/diffusion_loss': 0.30515187978744507, 'epoch': 1.59}

 16%|█▌        | 116/730 [10:23<54:41,  5.34s/it]
                                                 
{'train/learning_rate_real': 2.3928324245907324e-05, 'epoch': 1.59}

 16%|█▌        | 116/730 [10:23<54:41,  5.34s/it]
                                                 
{'debug/num_tok_total': 2870.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2870.0, 'debug/num_lat_loss': 1791.0, 'epoch': 1.59}

 16%|█▌        | 116/730 [10:24<54:41,  5.34s/it]
                                                 
{'train/ce_loss': 3.953125, 'train/diffusion_loss': 0.3786502778530121, 'epoch': 1.59}

 16%|█▌        | 116/730 [10:24<54:41,  5.34s/it]
                                                 
{'train/learning_rate_real': 2.3928324245907324e-05, 'epoch': 1.59}

 16%|█▌        | 116/730 [10:24<54:41,  5.34s/it]
                                                 
{'debug/num_tok_total': 2645.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2645.0, 'debug/num_lat_loss': 1785.0, 'epoch': 1.59}

 16%|█▌        | 116/730 [10:26<54:41,  5.34s/it]
                                                 
{'train/ce_loss': 3.8125, 'train/diffusion_loss': 0.415124773979187, 'epoch': 1.59}

 16%|█▌        | 116/730 [10:26<54:41,  5.34s/it]
                                                 
{'train/learning_rate_real': 2.3928324245907324e-05, 'epoch': 1.59}

 16%|█▌        | 116/730 [10:26<54:41,  5.34s/it]
 16%|█▌        | 117/730 [10:26<55:38,  5.45s/it]
                                                 
{'debug/num_tok_total': 2645.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2645.0, 'debug/num_lat_loss': 1785.0, 'epoch': 1.6}

 16%|█▌        | 117/730 [10:27<55:38,  5.45s/it]
                                                 
{'train/ce_loss': 3.90625, 'train/diffusion_loss': 0.4104914665222168, 'epoch': 1.6}

 16%|█▌        | 117/730 [10:27<55:38,  5.45s/it]
                                                 
{'train/learning_rate_real': 2.3905741739411735e-05, 'epoch': 1.6}

 16%|█▌        | 117/730 [10:27<55:38,  5.45s/it]
                                                 
{'debug/num_tok_total': 3078.0, 'debug/num_tok_loss': 1679.0, 'debug/num_lat_total': 3078.0, 'debug/num_lat_loss': 1679.0, 'epoch': 1.6}

 16%|█▌        | 117/730 [10:29<55:38,  5.45s/it]
                                                 
{'train/ce_loss': 3.921875, 'train/diffusion_loss': 0.26384884119033813, 'epoch': 1.6}

 16%|█▌        | 117/730 [10:29<55:38,  5.45s/it]
                                                 
{'train/learning_rate_real': 2.3905741739411735e-05, 'epoch': 1.6}

 16%|█▌        | 117/730 [10:29<55:38,  5.45s/it]
                                                 
{'debug/num_tok_total': 2408.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2408.0, 'debug/num_lat_loss': 1777.0, 'epoch': 1.6}

 16%|█▌        | 117/730 [10:30<55:38,  5.45s/it]
                                                 
{'train/ce_loss': 3.75, 'train/diffusion_loss': 0.4527377486228943, 'epoch': 1.6}

 16%|█▌        | 117/730 [10:30<55:38,  5.45s/it]
                                                 
{'train/learning_rate_real': 2.3905741739411735e-05, 'epoch': 1.6}

 16%|█▌        | 117/730 [10:30<55:38,  5.45s/it]
                                                 
{'debug/num_tok_total': 2517.0, 'debug/num_tok_loss': 1643.0, 'debug/num_lat_total': 2517.0, 'debug/num_lat_loss': 1643.0, 'epoch': 1.6}

 16%|█▌        | 117/730 [10:31<55:38,  5.45s/it]
                                                 
{'train/ce_loss': 4.03125, 'train/diffusion_loss': 0.4090951979160309, 'epoch': 1.6}

 16%|█▌        | 117/730 [10:31<55:38,  5.45s/it]
                                                 
{'train/learning_rate_real': 2.3905741739411735e-05, 'epoch': 1.6}

 16%|█▌        | 117/730 [10:31<55:38,  5.45s/it]
 16%|█▌        | 118/730 [10:31<55:24,  5.43s/it]
                                                 
{'debug/num_tok_total': 3067.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 3067.0, 'debug/num_lat_loss': 1780.0, 'epoch': 1.62}

 16%|█▌        | 118/730 [10:33<55:24,  5.43s/it]
                                                 
{'train/ce_loss': 3.796875, 'train/diffusion_loss': 0.30969923734664917, 'epoch': 1.62}

 16%|█▌        | 118/730 [10:33<55:24,  5.43s/it]
                                                 
{'train/learning_rate_real': 2.3882934660686418e-05, 'epoch': 1.62}

 16%|█▌        | 118/730 [10:33<55:24,  5.43s/it]
                                                 
{'debug/num_tok_total': 2414.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2414.0, 'debug/num_lat_loss': 1781.0, 'epoch': 1.62}

 16%|█▌        | 118/730 [10:34<55:24,  5.43s/it]
                                                 
{'train/ce_loss': 3.796875, 'train/diffusion_loss': 0.46044453978538513, 'epoch': 1.62}

 16%|█▌        | 118/730 [10:34<55:24,  5.43s/it]
                                                 
{'train/learning_rate_real': 2.3882934660686418e-05, 'epoch': 1.62}

 16%|█▌        | 118/730 [10:34<55:24,  5.43s/it]
                                                 
{'debug/num_tok_total': 2839.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2839.0, 'debug/num_lat_loss': 1775.0, 'epoch': 1.62}

 16%|█▌        | 118/730 [10:35<55:24,  5.43s/it]
                                                 
{'train/ce_loss': 3.90625, 'train/diffusion_loss': 0.4008499085903168, 'epoch': 1.62}

 16%|█▌        | 118/730 [10:35<55:24,  5.43s/it]
                                                 
{'train/learning_rate_real': 2.3882934660686418e-05, 'epoch': 1.62}

 16%|█▌        | 118/730 [10:35<55:24,  5.43s/it]
                                                 
{'debug/num_tok_total': 2868.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2868.0, 'debug/num_lat_loss': 1772.0, 'epoch': 1.62}

 16%|█▌        | 118/730 [10:36<55:24,  5.43s/it]
                                                 
{'train/ce_loss': 4.0, 'train/diffusion_loss': 0.33701223134994507, 'epoch': 1.62}

 16%|█▌        | 118/730 [10:36<55:24,  5.43s/it]
                                                 
{'train/learning_rate_real': 2.3882934660686418e-05, 'epoch': 1.62}

 16%|█▌        | 118/730 [10:36<55:24,  5.43s/it]
 16%|█▋        | 119/730 [10:37<55:36,  5.46s/it]
                                                 
{'debug/num_tok_total': 2748.0, 'debug/num_tok_loss': 1727.0, 'debug/num_lat_total': 2748.0, 'debug/num_lat_loss': 1727.0, 'epoch': 1.63}

 16%|█▋        | 119/730 [10:38<55:36,  5.46s/it]
                                                 
{'train/ce_loss': 3.84375, 'train/diffusion_loss': 0.36649128794670105, 'epoch': 1.63}

 16%|█▋        | 119/730 [10:38<55:36,  5.46s/it]
                                                 
{'train/learning_rate_real': 2.3859903458789094e-05, 'epoch': 1.63}

 16%|█▋        | 119/730 [10:38<55:36,  5.46s/it]
                                                 
{'debug/num_tok_total': 2493.0, 'debug/num_tok_loss': 1707.0, 'debug/num_lat_total': 2493.0, 'debug/num_lat_loss': 1707.0, 'epoch': 1.63}

 16%|█▋        | 119/730 [10:39<55:36,  5.46s/it]
                                                 
{'train/ce_loss': 3.75, 'train/diffusion_loss': 0.4311346709728241, 'epoch': 1.63}

 16%|█▋        | 119/730 [10:39<55:36,  5.46s/it]
                                                 
{'train/learning_rate_real': 2.3859903458789094e-05, 'epoch': 1.63}

 16%|█▋        | 119/730 [10:39<55:36,  5.46s/it]
                                                 
{'debug/num_tok_total': 2782.0, 'debug/num_tok_loss': 1697.0, 'debug/num_lat_total': 2782.0, 'debug/num_lat_loss': 1697.0, 'epoch': 1.63}

 16%|█▋        | 119/730 [10:41<55:36,  5.46s/it]
                                                 
{'train/ce_loss': 3.90625, 'train/diffusion_loss': 0.3630007207393646, 'epoch': 1.63}

 16%|█▋        | 119/730 [10:41<55:36,  5.46s/it]
                                                 
{'train/learning_rate_real': 2.3859903458789094e-05, 'epoch': 1.63}

 16%|█▋        | 119/730 [10:41<55:36,  5.46s/it]
                                                 
{'debug/num_tok_total': 2421.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2421.0, 'debug/num_lat_loss': 1780.0, 'epoch': 1.63}

 16%|█▋        | 119/730 [10:42<55:36,  5.46s/it]
                                                 
{'train/ce_loss': 3.8125, 'train/diffusion_loss': 0.432765394449234, 'epoch': 1.63}

 16%|█▋        | 119/730 [10:42<55:36,  5.46s/it]
                                                 
{'train/learning_rate_real': 2.3859903458789094e-05, 'epoch': 1.63}

 16%|█▋        | 119/730 [10:42<55:36,  5.46s/it]03/16/2026 06:58:14 - INFO - __main__ - LoRA debug step 120: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 16%|█▋        | 120/730 [10:42<54:53,  5.40s/it]
                                                 
{'loss': 2.8148, 'grad_norm': 1.4388368129730225, 'learning_rate': 2.3859903458789094e-05, 'epoch': 1.64}

 16%|█▋        | 120/730 [10:42<54:53,  5.40s/it]
                                                 
{'debug/num_tok_total': 2617.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2617.0, 'debug/num_lat_loss': 1764.0, 'epoch': 1.64}

 16%|█▋        | 120/730 [10:43<54:53,  5.40s/it]
                                                 
{'train/ce_loss': 3.953125, 'train/diffusion_loss': 0.4270760118961334, 'epoch': 1.64}

 16%|█▋        | 120/730 [10:43<54:53,  5.40s/it]
                                                 
{'train/learning_rate_real': 2.3836648587190354e-05, 'epoch': 1.64}

 16%|█▋        | 120/730 [10:43<54:53,  5.40s/it]
                                                 
{'debug/num_tok_total': 2220.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2220.0, 'debug/num_lat_loss': 1789.0, 'epoch': 1.64}

 16%|█▋        | 120/730 [10:45<54:53,  5.40s/it]
                                                 
{'train/ce_loss': 3.796875, 'train/diffusion_loss': 0.5018504858016968, 'epoch': 1.64}

 16%|█▋        | 120/730 [10:45<54:53,  5.40s/it]
                                                 
{'train/learning_rate_real': 2.3836648587190354e-05, 'epoch': 1.64}

 16%|█▋        | 120/730 [10:45<54:53,  5.40s/it]
                                                 
{'debug/num_tok_total': 2422.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2422.0, 'debug/num_lat_loss': 1777.0, 'epoch': 1.64}

 16%|█▋        | 120/730 [10:46<54:53,  5.40s/it]
                                                 
{'train/ce_loss': 3.796875, 'train/diffusion_loss': 0.47332513332366943, 'epoch': 1.64}

 16%|█▋        | 120/730 [10:46<54:53,  5.40s/it]
                                                 
{'train/learning_rate_real': 2.3836648587190354e-05, 'epoch': 1.64}

 16%|█▋        | 120/730 [10:46<54:53,  5.40s/it]
                                                 
{'debug/num_tok_total': 2864.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2864.0, 'debug/num_lat_loss': 1785.0, 'epoch': 1.64}

 16%|█▋        | 120/730 [10:47<54:53,  5.40s/it]
                                                 
{'train/ce_loss': 3.75, 'train/diffusion_loss': 0.3780655264854431, 'epoch': 1.64}

 16%|█▋        | 120/730 [10:47<54:53,  5.40s/it]
                                                 
{'train/learning_rate_real': 2.3836648587190354e-05, 'epoch': 1.64}

 16%|█▋        | 120/730 [10:47<54:53,  5.40s/it]
 17%|█▋        | 121/730 [10:47<54:13,  5.34s/it]
                                                 
{'debug/num_tok_total': 2640.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2640.0, 'debug/num_lat_loss': 1774.0, 'epoch': 1.66}

 17%|█▋        | 121/730 [10:49<54:13,  5.34s/it]
                                                 
{'train/ce_loss': 4.125, 'train/diffusion_loss': 0.44194358587265015, 'epoch': 1.66}

 17%|█▋        | 121/730 [10:49<54:13,  5.34s/it]
                                                 
{'train/learning_rate_real': 2.38131705037647e-05, 'epoch': 1.66}

 17%|█▋        | 121/730 [10:49<54:13,  5.34s/it]
                                                 
{'debug/num_tok_total': 2855.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2855.0, 'debug/num_lat_loss': 1781.0, 'epoch': 1.66}

 17%|█▋        | 121/730 [10:50<54:13,  5.34s/it]
                                                 
{'train/ce_loss': 3.703125, 'train/diffusion_loss': 0.3793834149837494, 'epoch': 1.66}

 17%|█▋        | 121/730 [10:50<54:13,  5.34s/it]
                                                 
{'train/learning_rate_real': 2.38131705037647e-05, 'epoch': 1.66}

 17%|█▋        | 121/730 [10:50<54:13,  5.34s/it]
                                                 
{'debug/num_tok_total': 2801.0, 'debug/num_tok_loss': 1745.0, 'debug/num_lat_total': 2801.0, 'debug/num_lat_loss': 1745.0, 'epoch': 1.66}

 17%|█▋        | 121/730 [10:51<54:13,  5.34s/it]
                                                 
{'train/ce_loss': 3.90625, 'train/diffusion_loss': 0.36281681060791016, 'epoch': 1.66}

 17%|█▋        | 121/730 [10:51<54:13,  5.34s/it]
                                                 
{'train/learning_rate_real': 2.38131705037647e-05, 'epoch': 1.66}

 17%|█▋        | 121/730 [10:51<54:13,  5.34s/it]
                                                 
{'debug/num_tok_total': 2887.0, 'debug/num_tok_loss': 1803.0, 'debug/num_lat_total': 2887.0, 'debug/num_lat_loss': 1803.0, 'epoch': 1.66}

 17%|█▋        | 121/730 [10:52<54:13,  5.34s/it]
                                                 
{'train/ce_loss': 3.84375, 'train/diffusion_loss': 0.3547899127006531, 'epoch': 1.66}

 17%|█▋        | 121/730 [10:52<54:13,  5.34s/it]
                                                 
{'train/learning_rate_real': 2.38131705037647e-05, 'epoch': 1.66}

 17%|█▋        | 121/730 [10:52<54:13,  5.34s/it]
 17%|█▋        | 122/730 [10:53<54:28,  5.38s/it]
                                                 
{'debug/num_tok_total': 2153.0, 'debug/num_tok_loss': 1712.0, 'debug/num_lat_total': 2153.0, 'debug/num_lat_loss': 1712.0, 'epoch': 1.67}

 17%|█▋        | 122/730 [10:54<54:28,  5.38s/it]
                                                 
{'train/ce_loss': 3.828125, 'train/diffusion_loss': 0.49153730273246765, 'epoch': 1.67}

 17%|█▋        | 122/730 [10:54<54:28,  5.38s/it]
                                                 
{'train/learning_rate_real': 2.3789469670781547e-05, 'epoch': 1.67}

 17%|█▋        | 122/730 [10:54<54:28,  5.38s/it]
                                                 
{'debug/num_tok_total': 2218.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2218.0, 'debug/num_lat_loss': 1787.0, 'epoch': 1.67}

 17%|█▋        | 122/730 [10:55<54:28,  5.38s/it]
                                                 
{'train/ce_loss': 3.65625, 'train/diffusion_loss': 0.5030884742736816, 'epoch': 1.67}

 17%|█▋        | 122/730 [10:55<54:28,  5.38s/it]
                                                 
{'train/learning_rate_real': 2.3789469670781547e-05, 'epoch': 1.67}

 17%|█▋        | 122/730 [10:55<54:28,  5.38s/it]
                                                 
{'debug/num_tok_total': 2865.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2865.0, 'debug/num_lat_loss': 1787.0, 'epoch': 1.67}

 17%|█▋        | 122/730 [10:56<54:28,  5.38s/it]
                                                 
{'train/ce_loss': 4.03125, 'train/diffusion_loss': 0.3352532982826233, 'epoch': 1.67}

 17%|█▋        | 122/730 [10:56<54:28,  5.38s/it]
                                                 
{'train/learning_rate_real': 2.3789469670781547e-05, 'epoch': 1.67}

 17%|█▋        | 122/730 [10:56<54:28,  5.38s/it]
                                                 
{'debug/num_tok_total': 3069.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 3069.0, 'debug/num_lat_loss': 1779.0, 'epoch': 1.67}

 17%|█▋        | 122/730 [10:58<54:28,  5.38s/it]
                                                 
{'train/ce_loss': 3.828125, 'train/diffusion_loss': 0.32163891196250916, 'epoch': 1.67}

 17%|█▋        | 122/730 [10:58<54:28,  5.38s/it]
                                                 
{'train/learning_rate_real': 2.3789469670781547e-05, 'epoch': 1.67}

 17%|█▋        | 122/730 [10:58<54:28,  5.38s/it]
 17%|█▋        | 123/730 [10:58<53:55,  5.33s/it]
                                                 
{'debug/num_tok_total': 2383.0, 'debug/num_tok_loss': 1750.0, 'debug/num_lat_total': 2383.0, 'debug/num_lat_loss': 1750.0, 'epoch': 1.68}

 17%|█▋        | 123/730 [10:59<53:55,  5.33s/it]
                                                 
{'train/ce_loss': 3.734375, 'train/diffusion_loss': 0.4190615117549896, 'epoch': 1.68}

 17%|█▋        | 123/730 [10:59<53:55,  5.33s/it]
                                                 
{'train/learning_rate_real': 2.376554655489612e-05, 'epoch': 1.68}

 17%|█▋        | 123/730 [10:59<53:55,  5.33s/it]
                                                 
{'debug/num_tok_total': 2214.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2214.0, 'debug/num_lat_loss': 1785.0, 'epoch': 1.68}

 17%|█▋        | 123/730 [11:00<53:55,  5.33s/it]
                                                 
{'train/ce_loss': 3.609375, 'train/diffusion_loss': 0.5074918866157532, 'epoch': 1.68}

 17%|█▋        | 123/730 [11:00<53:55,  5.33s/it]
                                                 
{'train/learning_rate_real': 2.376554655489612e-05, 'epoch': 1.68}

 17%|█▋        | 123/730 [11:00<53:55,  5.33s/it]
                                                 
{'debug/num_tok_total': 2459.0, 'debug/num_tok_loss': 1803.0, 'debug/num_lat_total': 2459.0, 'debug/num_lat_loss': 1803.0, 'epoch': 1.68}

 17%|█▋        | 123/730 [11:01<53:55,  5.33s/it]
                                                 
{'train/ce_loss': 3.671875, 'train/diffusion_loss': 0.4351557791233063, 'epoch': 1.68}

 17%|█▋        | 123/730 [11:01<53:55,  5.33s/it]
                                                 
{'train/learning_rate_real': 2.376554655489612e-05, 'epoch': 1.68}

 17%|█▋        | 123/730 [11:01<53:55,  5.33s/it]
                                                 
{'debug/num_tok_total': 2306.0, 'debug/num_tok_loss': 1642.0, 'debug/num_lat_total': 2306.0, 'debug/num_lat_loss': 1642.0, 'epoch': 1.68}

 17%|█▋        | 123/730 [11:03<53:55,  5.33s/it]
                                                 
{'train/ce_loss': 3.703125, 'train/diffusion_loss': 0.4449380934238434, 'epoch': 1.68}

 17%|█▋        | 123/730 [11:03<53:55,  5.33s/it]
                                                 
{'train/learning_rate_real': 2.376554655489612e-05, 'epoch': 1.68}

 17%|█▋        | 123/730 [11:03<53:55,  5.33s/it]
 17%|█▋        | 124/730 [11:03<52:57,  5.24s/it]
                                                 
{'debug/num_tok_total': 2881.0, 'debug/num_tok_loss': 1798.0, 'debug/num_lat_total': 2881.0, 'debug/num_lat_loss': 1798.0, 'epoch': 1.7}

 17%|█▋        | 124/730 [11:04<52:57,  5.24s/it]
                                                 
{'train/ce_loss': 3.703125, 'train/diffusion_loss': 0.36390984058380127, 'epoch': 1.7}

 17%|█▋        | 124/730 [11:04<52:57,  5.24s/it]
                                                 
{'train/learning_rate_real': 2.3741401627140242e-05, 'epoch': 1.7}

 17%|█▋        | 124/730 [11:04<52:57,  5.24s/it]
                                                 
{'debug/num_tok_total': 2810.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2810.0, 'debug/num_lat_loss': 1763.0, 'epoch': 1.7}

 17%|█▋        | 124/730 [11:06<52:57,  5.24s/it]
                                                 
{'train/ce_loss': 3.734375, 'train/diffusion_loss': 0.3646058440208435, 'epoch': 1.7}

 17%|█▋        | 124/730 [11:06<52:57,  5.24s/it]
                                                 
{'train/learning_rate_real': 2.3741401627140242e-05, 'epoch': 1.7}

 17%|█▋        | 124/730 [11:06<52:57,  5.24s/it]
                                                 
{'debug/num_tok_total': 2642.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 2642.0, 'debug/num_lat_loss': 1769.0, 'epoch': 1.7}

 17%|█▋        | 124/730 [11:07<52:57,  5.24s/it]
                                                 
{'train/ce_loss': 3.796875, 'train/diffusion_loss': 0.40764036774635315, 'epoch': 1.7}

 17%|█▋        | 124/730 [11:07<52:57,  5.24s/it]
                                                 
{'train/learning_rate_real': 2.3741401627140242e-05, 'epoch': 1.7}

 17%|█▋        | 124/730 [11:07<52:57,  5.24s/it]
                                                 
{'debug/num_tok_total': 2195.0, 'debug/num_tok_loss': 1761.0, 'debug/num_lat_total': 2195.0, 'debug/num_lat_loss': 1761.0, 'epoch': 1.7}

 17%|█▋        | 124/730 [11:08<52:57,  5.24s/it]
                                                 
{'train/ce_loss': 4.09375, 'train/diffusion_loss': 0.47080162167549133, 'epoch': 1.7}

 17%|█▋        | 124/730 [11:08<52:57,  5.24s/it]
                                                 
{'train/learning_rate_real': 2.3741401627140242e-05, 'epoch': 1.7}

 17%|█▋        | 124/730 [11:08<52:57,  5.24s/it]
 17%|█▋        | 125/730 [11:08<52:54,  5.25s/it]
                                                 
{'debug/num_tok_total': 2225.0, 'debug/num_tok_loss': 1679.0, 'debug/num_lat_total': 2225.0, 'debug/num_lat_loss': 1679.0, 'epoch': 1.71}

 17%|█▋        | 125/730 [11:09<52:54,  5.25s/it]
                                                 
{'train/ce_loss': 3.546875, 'train/diffusion_loss': 0.464213103055954, 'epoch': 1.71}

 17%|█▋        | 125/730 [11:09<52:54,  5.25s/it]
                                                 
{'train/learning_rate_real': 2.3717035362913105e-05, 'epoch': 1.71}

 17%|█▋        | 125/730 [11:09<52:54,  5.25s/it]
                                                 
{'debug/num_tok_total': 2696.0, 'debug/num_tok_loss': 1592.0, 'debug/num_lat_total': 2696.0, 'debug/num_lat_loss': 1592.0, 'epoch': 1.71}

 17%|█▋        | 125/730 [11:11<52:54,  5.25s/it]
                                                 
{'train/ce_loss': 3.90625, 'train/diffusion_loss': 0.3316189646720886, 'epoch': 1.71}

 17%|█▋        | 125/730 [11:11<52:54,  5.25s/it]
                                                 
{'train/learning_rate_real': 2.3717035362913105e-05, 'epoch': 1.71}

 17%|█▋        | 125/730 [11:11<52:54,  5.25s/it]
                                                 
{'debug/num_tok_total': 2460.0, 'debug/num_tok_loss': 1578.0, 'debug/num_lat_total': 2460.0, 'debug/num_lat_loss': 1578.0, 'epoch': 1.71}

 17%|█▋        | 125/730 [11:12<52:54,  5.25s/it]
                                                 
{'train/ce_loss': 3.796875, 'train/diffusion_loss': 0.39441150426864624, 'epoch': 1.71}

 17%|█▋        | 125/730 [11:12<52:54,  5.25s/it]
                                                 
{'train/learning_rate_real': 2.3717035362913105e-05, 'epoch': 1.71}

 17%|█▋        | 125/730 [11:12<52:54,  5.25s/it]
                                                 
{'debug/num_tok_total': 2464.0, 'debug/num_tok_loss': 1800.0, 'debug/num_lat_total': 2464.0, 'debug/num_lat_loss': 1800.0, 'epoch': 1.71}

 17%|█▋        | 125/730 [11:13<52:54,  5.25s/it]
                                                 
{'train/ce_loss': 3.625, 'train/diffusion_loss': 0.4529780447483063, 'epoch': 1.71}

 17%|█▋        | 125/730 [11:13<52:54,  5.25s/it]
                                                 
{'train/learning_rate_real': 2.3717035362913105e-05, 'epoch': 1.71}

 17%|█▋        | 125/730 [11:13<52:54,  5.25s/it]
 17%|█▋        | 126/730 [11:14<53:09,  5.28s/it]
                                                 
{'debug/num_tok_total': 2644.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2644.0, 'debug/num_lat_loss': 1780.0, 'epoch': 1.73}

 17%|█▋        | 126/730 [11:15<53:09,  5.28s/it]
                                                 
{'train/ce_loss': 3.71875, 'train/diffusion_loss': 0.41818350553512573, 'epoch': 1.73}

 17%|█▋        | 126/730 [11:15<53:09,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.3692448241971866e-05, 'epoch': 1.73}

 17%|█▋        | 126/730 [11:15<53:09,  5.28s/it]
                                                 
{'debug/num_tok_total': 2631.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2631.0, 'debug/num_lat_loss': 1773.0, 'epoch': 1.73}

 17%|█▋        | 126/730 [11:16<53:09,  5.28s/it]
                                                 
{'train/ce_loss': 3.828125, 'train/diffusion_loss': 0.4160309433937073, 'epoch': 1.73}

 17%|█▋        | 126/730 [11:16<53:09,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.3692448241971866e-05, 'epoch': 1.73}

 17%|█▋        | 126/730 [11:16<53:09,  5.28s/it]
                                                 
{'debug/num_tok_total': 2413.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2413.0, 'debug/num_lat_loss': 1773.0, 'epoch': 1.73}

 17%|█▋        | 126/730 [11:17<53:09,  5.28s/it]
                                                 
{'train/ce_loss': 3.75, 'train/diffusion_loss': 0.4794480800628662, 'epoch': 1.73}

 17%|█▋        | 126/730 [11:17<53:09,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.3692448241971866e-05, 'epoch': 1.73}

 17%|█▋        | 126/730 [11:17<53:09,  5.28s/it]
                                                 
{'debug/num_tok_total': 2695.0, 'debug/num_tok_loss': 1812.0, 'debug/num_lat_total': 2695.0, 'debug/num_lat_loss': 1812.0, 'epoch': 1.73}

 17%|█▋        | 126/730 [11:19<53:09,  5.28s/it]
                                                 
{'train/ce_loss': 3.765625, 'train/diffusion_loss': 0.4259827435016632, 'epoch': 1.73}

 17%|█▋        | 126/730 [11:19<53:09,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.3692448241971866e-05, 'epoch': 1.73}

 17%|█▋        | 126/730 [11:19<53:09,  5.28s/it]
 17%|█▋        | 127/730 [11:19<53:10,  5.29s/it]
                                                 
{'debug/num_tok_total': 2427.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2427.0, 'debug/num_lat_loss': 1792.0, 'epoch': 1.74}

 17%|█▋        | 127/730 [11:20<53:10,  5.29s/it]
                                                 
{'train/ce_loss': 3.75, 'train/diffusion_loss': 0.4588565230369568, 'epoch': 1.74}

 17%|█▋        | 127/730 [11:20<53:10,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.366764074842222e-05, 'epoch': 1.74}

 17%|█▋        | 127/730 [11:20<53:10,  5.29s/it]
                                                 
{'debug/num_tok_total': 2430.0, 'debug/num_tok_loss': 1683.0, 'debug/num_lat_total': 2430.0, 'debug/num_lat_loss': 1683.0, 'epoch': 1.74}

 17%|█▋        | 127/730 [11:21<53:10,  5.29s/it]
                                                 
{'train/ce_loss': 3.765625, 'train/diffusion_loss': 0.42775484919548035, 'epoch': 1.74}

 17%|█▋        | 127/730 [11:21<53:10,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.366764074842222e-05, 'epoch': 1.74}

 17%|█▋        | 127/730 [11:21<53:10,  5.29s/it]
                                                 
{'debug/num_tok_total': 2229.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2229.0, 'debug/num_lat_loss': 1792.0, 'epoch': 1.74}

 17%|█▋        | 127/730 [11:22<53:10,  5.29s/it]
                                                 
{'train/ce_loss': 3.65625, 'train/diffusion_loss': 0.4776081442832947, 'epoch': 1.74}

 17%|█▋        | 127/730 [11:22<53:10,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.366764074842222e-05, 'epoch': 1.74}

 17%|█▋        | 127/730 [11:22<53:10,  5.29s/it]
                                                 
{'debug/num_tok_total': 2917.0, 'debug/num_tok_loss': 1598.0, 'debug/num_lat_total': 2917.0, 'debug/num_lat_loss': 1598.0, 'epoch': 1.74}

 17%|█▋        | 127/730 [11:24<53:10,  5.29s/it]
                                                 
{'train/ce_loss': 3.734375, 'train/diffusion_loss': 0.2688691020011902, 'epoch': 1.74}

 17%|█▋        | 127/730 [11:24<53:10,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.366764074842222e-05, 'epoch': 1.74}

 17%|█▋        | 127/730 [11:24<53:10,  5.29s/it]
 18%|█▊        | 128/730 [11:24<53:01,  5.29s/it]
                                                 
{'debug/num_tok_total': 2647.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2647.0, 'debug/num_lat_loss': 1794.0, 'epoch': 1.75}

 18%|█▊        | 128/730 [11:25<53:01,  5.29s/it]
                                                 
{'train/ce_loss': 3.609375, 'train/diffusion_loss': 0.41486573219299316, 'epoch': 1.75}

 18%|█▊        | 128/730 [11:25<53:01,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.364261337070888e-05, 'epoch': 1.75}

 18%|█▊        | 128/730 [11:25<53:01,  5.29s/it]
                                                 
{'debug/num_tok_total': 2188.0, 'debug/num_tok_loss': 1762.0, 'debug/num_lat_total': 2188.0, 'debug/num_lat_loss': 1762.0, 'epoch': 1.75}

 18%|█▊        | 128/730 [11:27<53:01,  5.29s/it]
                                                 
{'train/ce_loss': 3.625, 'train/diffusion_loss': 0.5014437437057495, 'epoch': 1.75}

 18%|█▊        | 128/730 [11:27<53:01,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.364261337070888e-05, 'epoch': 1.75}

 18%|█▊        | 128/730 [11:27<53:01,  5.29s/it]
                                                 
{'debug/num_tok_total': 2525.0, 'debug/num_tok_loss': 1660.0, 'debug/num_lat_total': 2525.0, 'debug/num_lat_loss': 1660.0, 'epoch': 1.75}

 18%|█▊        | 128/730 [11:28<53:01,  5.29s/it]
                                                 
{'train/ce_loss': 3.484375, 'train/diffusion_loss': 0.40325120091438293, 'epoch': 1.75}

 18%|█▊        | 128/730 [11:28<53:01,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.364261337070888e-05, 'epoch': 1.75}

 18%|█▊        | 128/730 [11:28<53:01,  5.29s/it]
                                                 
{'debug/num_tok_total': 2832.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2832.0, 'debug/num_lat_loss': 1764.0, 'epoch': 1.75}

 18%|█▊        | 128/730 [11:29<53:01,  5.29s/it]
                                                 
{'train/ce_loss': 4.09375, 'train/diffusion_loss': 0.3543040454387665, 'epoch': 1.75}

 18%|█▊        | 128/730 [11:29<53:01,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.364261337070888e-05, 'epoch': 1.75}

 18%|█▊        | 128/730 [11:29<53:01,  5.29s/it]
 18%|█▊        | 129/730 [11:30<52:54,  5.28s/it]
                                                 
{'debug/num_tok_total': 2445.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2445.0, 'debug/num_lat_loss': 1790.0, 'epoch': 1.77}

 18%|█▊        | 129/730 [11:31<52:54,  5.28s/it]
                                                 
{'train/ce_loss': 3.859375, 'train/diffusion_loss': 0.472688764333725, 'epoch': 1.77}

 18%|█▊        | 129/730 [11:31<52:54,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.361736660160593e-05, 'epoch': 1.77}

 18%|█▊        | 129/730 [11:31<52:54,  5.28s/it]
                                                 
{'debug/num_tok_total': 3572.0, 'debug/num_tok_loss': 1816.0, 'debug/num_lat_total': 3572.0, 'debug/num_lat_loss': 1816.0, 'epoch': 1.77}

 18%|█▊        | 129/730 [11:32<52:54,  5.28s/it]
                                                 
{'train/ce_loss': 3.734375, 'train/diffusion_loss': 0.22201991081237793, 'epoch': 1.77}

 18%|█▊        | 129/730 [11:32<52:54,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.361736660160593e-05, 'epoch': 1.77}

 18%|█▊        | 129/730 [11:32<52:54,  5.28s/it]
                                                 
{'debug/num_tok_total': 2410.0, 'debug/num_tok_loss': 1761.0, 'debug/num_lat_total': 2410.0, 'debug/num_lat_loss': 1761.0, 'epoch': 1.77}

 18%|█▊        | 129/730 [11:33<52:54,  5.28s/it]
                                                 
{'train/ce_loss': 3.53125, 'train/diffusion_loss': 0.44313621520996094, 'epoch': 1.77}

 18%|█▊        | 129/730 [11:33<52:54,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.361736660160593e-05, 'epoch': 1.77}

 18%|█▊        | 129/730 [11:33<52:54,  5.28s/it]
                                                 
{'debug/num_tok_total': 2825.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2825.0, 'debug/num_lat_loss': 1764.0, 'epoch': 1.77}

 18%|█▊        | 129/730 [11:35<52:54,  5.28s/it]
                                                 
{'train/ce_loss': 3.625, 'train/diffusion_loss': 0.3336658179759979, 'epoch': 1.77}

 18%|█▊        | 129/730 [11:35<52:54,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.361736660160593e-05, 'epoch': 1.77}

 18%|█▊        | 129/730 [11:35<52:54,  5.28s/it]03/16/2026 06:59:07 - INFO - __main__ - LoRA debug step 130: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 18%|█▊        | 130/730 [11:35<53:18,  5.33s/it]
                                                 
{'loss': 2.9194, 'grad_norm': 1.576515793800354, 'learning_rate': 2.361736660160593e-05, 'epoch': 1.78}

 18%|█▊        | 130/730 [11:35<53:18,  5.33s/it]
                                                 
{'debug/num_tok_total': 2308.0, 'debug/num_tok_loss': 1646.0, 'debug/num_lat_total': 2308.0, 'debug/num_lat_loss': 1646.0, 'epoch': 1.78}

 18%|█▊        | 130/730 [11:36<53:18,  5.33s/it]
                                                 
{'train/ce_loss': 3.578125, 'train/diffusion_loss': 0.43932580947875977, 'epoch': 1.78}

 18%|█▊        | 130/730 [11:36<53:18,  5.33s/it]
                                                 
{'train/learning_rate_real': 2.3591900938207147e-05, 'epoch': 1.78}

 18%|█▊        | 130/730 [11:36<53:18,  5.33s/it]
                                                 
{'debug/num_tok_total': 2868.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2868.0, 'debug/num_lat_loss': 1791.0, 'epoch': 1.78}

 18%|█▊        | 130/730 [11:37<53:18,  5.33s/it]
                                                 
{'train/ce_loss': 3.6875, 'train/diffusion_loss': 0.3732472360134125, 'epoch': 1.78}

 18%|█▊        | 130/730 [11:37<53:18,  5.33s/it]
                                                 
{'train/learning_rate_real': 2.3591900938207147e-05, 'epoch': 1.78}

 18%|█▊        | 130/730 [11:37<53:18,  5.33s/it]
                                                 
{'debug/num_tok_total': 2425.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2425.0, 'debug/num_lat_loss': 1772.0, 'epoch': 1.78}

 18%|█▊        | 130/730 [11:39<53:18,  5.33s/it]
                                                 
{'train/ce_loss': 3.515625, 'train/diffusion_loss': 0.45174187421798706, 'epoch': 1.78}

 18%|█▊        | 130/730 [11:39<53:18,  5.33s/it]
                                                 
{'train/learning_rate_real': 2.3591900938207147e-05, 'epoch': 1.78}

 18%|█▊        | 130/730 [11:39<53:18,  5.33s/it]
                                                 
{'debug/num_tok_total': 2887.0, 'debug/num_tok_loss': 1692.0, 'debug/num_lat_total': 2887.0, 'debug/num_lat_loss': 1692.0, 'epoch': 1.78}

 18%|█▊        | 130/730 [11:40<53:18,  5.33s/it]
                                                 
{'train/ce_loss': 3.734375, 'train/diffusion_loss': 0.32698309421539307, 'epoch': 1.78}

 18%|█▊        | 130/730 [11:40<53:18,  5.33s/it]
                                                 
{'train/learning_rate_real': 2.3591900938207147e-05, 'epoch': 1.78}

 18%|█▊        | 130/730 [11:40<53:18,  5.33s/it]
 18%|█▊        | 131/730 [11:40<53:31,  5.36s/it]
                                                 
{'debug/num_tok_total': 2840.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2840.0, 'debug/num_lat_loss': 1766.0, 'epoch': 1.79}

 18%|█▊        | 131/730 [11:42<53:31,  5.36s/it]
                                                 
{'train/ce_loss': 3.5, 'train/diffusion_loss': 0.34751012921333313, 'epoch': 1.79}

 18%|█▊        | 131/730 [11:42<53:31,  5.36s/it]
                                                 
{'train/learning_rate_real': 2.3566216881916202e-05, 'epoch': 1.79}

 18%|█▊        | 131/730 [11:42<53:31,  5.36s/it]
                                                 
{'debug/num_tok_total': 2644.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2644.0, 'debug/num_lat_loss': 1786.0, 'epoch': 1.79}

 18%|█▊        | 131/730 [11:43<53:31,  5.36s/it]
                                                 
{'train/ce_loss': 3.53125, 'train/diffusion_loss': 0.4357163906097412, 'epoch': 1.79}

 18%|█▊        | 131/730 [11:43<53:31,  5.36s/it]
                                                 
{'train/learning_rate_real': 2.3566216881916202e-05, 'epoch': 1.79}

 18%|█▊        | 131/730 [11:43<53:31,  5.36s/it]
                                                 
{'debug/num_tok_total': 2584.0, 'debug/num_tok_loss': 1753.0, 'debug/num_lat_total': 2584.0, 'debug/num_lat_loss': 1753.0, 'epoch': 1.79}

 18%|█▊        | 131/730 [11:44<53:31,  5.36s/it]
                                                 
{'train/ce_loss': 3.546875, 'train/diffusion_loss': 0.41815292835235596, 'epoch': 1.79}

 18%|█▊        | 131/730 [11:44<53:31,  5.36s/it]
                                                 
{'train/learning_rate_real': 2.3566216881916202e-05, 'epoch': 1.79}

 18%|█▊        | 131/730 [11:44<53:31,  5.36s/it]
                                                 
{'debug/num_tok_total': 2648.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2648.0, 'debug/num_lat_loss': 1791.0, 'epoch': 1.79}

 18%|█▊        | 131/730 [11:45<53:31,  5.36s/it]
                                                 
{'train/ce_loss': 3.765625, 'train/diffusion_loss': 0.36779430508613586, 'epoch': 1.79}

 18%|█▊        | 131/730 [11:45<53:31,  5.36s/it]
                                                 
{'train/learning_rate_real': 2.3566216881916202e-05, 'epoch': 1.79}

 18%|█▊        | 131/730 [11:45<53:31,  5.36s/it]
 18%|█▊        | 132/730 [11:46<53:12,  5.34s/it]
                                                 
{'debug/num_tok_total': 2171.0, 'debug/num_tok_loss': 1757.0, 'debug/num_lat_total': 2171.0, 'debug/num_lat_loss': 1757.0, 'epoch': 1.81}

 18%|█▊        | 132/730 [11:47<53:12,  5.34s/it]
                                                 
{'train/ce_loss': 3.578125, 'train/diffusion_loss': 0.47450587153434753, 'epoch': 1.81}

 18%|█▊        | 132/730 [11:47<53:12,  5.34s/it]
                                                 
{'train/learning_rate_real': 2.35403149384368e-05, 'epoch': 1.81}

 18%|█▊        | 132/730 [11:47<53:12,  5.34s/it]
                                                 
{'debug/num_tok_total': 2014.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2014.0, 'debug/num_lat_loss': 1793.0, 'epoch': 1.81}

 18%|█▊        | 132/730 [11:48<53:12,  5.34s/it]
                                                 
{'train/ce_loss': 3.4375, 'train/diffusion_loss': 0.5649107694625854, 'epoch': 1.81}

 18%|█▊        | 132/730 [11:48<53:12,  5.34s/it]
                                                 
{'train/learning_rate_real': 2.35403149384368e-05, 'epoch': 1.81}

 18%|█▊        | 132/730 [11:48<53:12,  5.34s/it]
                                                 
{'debug/num_tok_total': 2633.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2633.0, 'debug/num_lat_loss': 1780.0, 'epoch': 1.81}

 18%|█▊        | 132/730 [11:49<53:12,  5.34s/it]
                                                 
{'train/ce_loss': 3.5625, 'train/diffusion_loss': 0.4057047963142395, 'epoch': 1.81}

 18%|█▊        | 132/730 [11:49<53:12,  5.34s/it]
                                                 
{'train/learning_rate_real': 2.35403149384368e-05, 'epoch': 1.81}

 18%|█▊        | 132/730 [11:49<53:12,  5.34s/it]
                                                 
{'debug/num_tok_total': 2930.0, 'debug/num_tok_loss': 1647.0, 'debug/num_lat_total': 2930.0, 'debug/num_lat_loss': 1647.0, 'epoch': 1.81}

 18%|█▊        | 132/730 [11:50<53:12,  5.34s/it]
                                                 
{'train/ce_loss': 3.625, 'train/diffusion_loss': 0.28367847204208374, 'epoch': 1.81}

 18%|█▊        | 132/730 [11:50<53:12,  5.34s/it]
                                                 
{'train/learning_rate_real': 2.35403149384368e-05, 'epoch': 1.81}

 18%|█▊        | 132/730 [11:50<53:12,  5.34s/it]
 18%|█▊        | 133/730 [11:51<52:25,  5.27s/it]
                                                 
{'debug/num_tok_total': 2431.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2431.0, 'debug/num_lat_loss': 1780.0, 'epoch': 1.82}

 18%|█▊        | 133/730 [11:52<52:25,  5.27s/it]
                                                 
{'train/ce_loss': 3.53125, 'train/diffusion_loss': 0.44793954491615295, 'epoch': 1.82}

 18%|█▊        | 133/730 [11:52<52:25,  5.27s/it]
                                                 
{'train/learning_rate_real': 2.3514195617762706e-05, 'epoch': 1.82}

 18%|█▊        | 133/730 [11:52<52:25,  5.27s/it]
                                                 
{'debug/num_tok_total': 2612.0, 'debug/num_tok_loss': 1759.0, 'debug/num_lat_total': 2612.0, 'debug/num_lat_loss': 1759.0, 'epoch': 1.82}

 18%|█▊        | 133/730 [11:53<52:25,  5.27s/it]
                                                 
{'train/ce_loss': 3.46875, 'train/diffusion_loss': 0.4012427031993866, 'epoch': 1.82}

 18%|█▊        | 133/730 [11:53<52:25,  5.27s/it]
                                                 
{'train/learning_rate_real': 2.3514195617762706e-05, 'epoch': 1.82}

 18%|█▊        | 133/730 [11:53<52:25,  5.27s/it]
                                                 
{'debug/num_tok_total': 2788.0, 'debug/num_tok_loss': 1748.0, 'debug/num_lat_total': 2788.0, 'debug/num_lat_loss': 1748.0, 'epoch': 1.82}

 18%|█▊        | 133/730 [11:54<52:25,  5.27s/it]
                                                 
{'train/ce_loss': 3.484375, 'train/diffusion_loss': 0.3418940603733063, 'epoch': 1.82}

 18%|█▊        | 133/730 [11:54<52:25,  5.27s/it]
                                                 
{'train/learning_rate_real': 2.3514195617762706e-05, 'epoch': 1.82}

 18%|█▊        | 133/730 [11:54<52:25,  5.27s/it]
                                                 
{'debug/num_tok_total': 2204.0, 'debug/num_tok_loss': 1572.0, 'debug/num_lat_total': 2204.0, 'debug/num_lat_loss': 1572.0, 'epoch': 1.82}

 18%|█▊        | 133/730 [11:56<52:25,  5.27s/it]
                                                 
{'train/ce_loss': 3.453125, 'train/diffusion_loss': 0.4106413424015045, 'epoch': 1.82}

 18%|█▊        | 133/730 [11:56<52:25,  5.27s/it]
                                                 
{'train/learning_rate_real': 2.3514195617762706e-05, 'epoch': 1.82}

 18%|█▊        | 133/730 [11:56<52:25,  5.27s/it]
 18%|█▊        | 134/730 [11:56<52:10,  5.25s/it]
                                                 
{'debug/num_tok_total': 2190.0, 'debug/num_tok_loss': 1760.0, 'debug/num_lat_total': 2190.0, 'debug/num_lat_loss': 1760.0, 'epoch': 1.84}

 18%|█▊        | 134/730 [11:57<52:10,  5.25s/it]
                                                 
{'train/ce_loss': 3.9375, 'train/diffusion_loss': 0.5045044422149658, 'epoch': 1.84}

 18%|█▊        | 134/730 [11:57<52:10,  5.25s/it]
                                                 
{'train/learning_rate_real': 2.348785943416772e-05, 'epoch': 1.84}

 18%|█▊        | 134/730 [11:57<52:10,  5.25s/it]
                                                 
{'debug/num_tok_total': 2641.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2641.0, 'debug/num_lat_loss': 1784.0, 'epoch': 1.84}

 18%|█▊        | 134/730 [11:58<52:10,  5.25s/it]
                                                 
{'train/ce_loss': 3.578125, 'train/diffusion_loss': 0.38860100507736206, 'epoch': 1.84}

 18%|█▊        | 134/730 [11:58<52:10,  5.25s/it]
                                                 
{'train/learning_rate_real': 2.348785943416772e-05, 'epoch': 1.84}

 18%|█▊        | 134/730 [11:58<52:10,  5.25s/it]
                                                 
{'debug/num_tok_total': 2631.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2631.0, 'debug/num_lat_loss': 1777.0, 'epoch': 1.84}

 18%|█▊        | 134/730 [12:00<52:10,  5.25s/it]
                                                 
{'train/ce_loss': 3.578125, 'train/diffusion_loss': 0.393936425447464, 'epoch': 1.84}

 18%|█▊        | 134/730 [12:00<52:10,  5.25s/it]
                                                 
{'train/learning_rate_real': 2.348785943416772e-05, 'epoch': 1.84}

 18%|█▊        | 134/730 [12:00<52:10,  5.25s/it]
                                                 
{'debug/num_tok_total': 2408.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2408.0, 'debug/num_lat_loss': 1770.0, 'epoch': 1.84}

 18%|█▊        | 134/730 [12:01<52:10,  5.25s/it]
                                                 
{'train/ce_loss': 3.5, 'train/diffusion_loss': 0.44742295145988464, 'epoch': 1.84}

 18%|█▊        | 134/730 [12:01<52:10,  5.25s/it]
                                                 
{'train/learning_rate_real': 2.348785943416772e-05, 'epoch': 1.84}

 18%|█▊        | 134/730 [12:01<52:10,  5.25s/it]
 18%|█▊        | 135/730 [12:01<51:36,  5.20s/it]
                                                 
{'debug/num_tok_total': 2402.0, 'debug/num_tok_loss': 1759.0, 'debug/num_lat_total': 2402.0, 'debug/num_lat_loss': 1759.0, 'epoch': 1.85}

 18%|█▊        | 135/730 [12:02<51:36,  5.20s/it]
                                                 
{'train/ce_loss': 3.390625, 'train/diffusion_loss': 0.43890005350112915, 'epoch': 1.85}

 18%|█▊        | 135/730 [12:02<51:36,  5.20s/it]
                                                 
{'train/learning_rate_real': 2.3461306906195524e-05, 'epoch': 1.85}

 18%|█▊        | 135/730 [12:02<51:36,  5.20s/it]
                                                 
{'debug/num_tok_total': 2634.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2634.0, 'debug/num_lat_loss': 1776.0, 'epoch': 1.85}

 18%|█▊        | 135/730 [12:03<51:36,  5.20s/it]
                                                 
{'train/ce_loss': 3.53125, 'train/diffusion_loss': 0.4111637473106384, 'epoch': 1.85}

 18%|█▊        | 135/730 [12:03<51:36,  5.20s/it]
                                                 
{'train/learning_rate_real': 2.3461306906195524e-05, 'epoch': 1.85}

 18%|█▊        | 135/730 [12:03<51:36,  5.20s/it]
                                                 
{'debug/num_tok_total': 2712.0, 'debug/num_tok_loss': 1638.0, 'debug/num_lat_total': 2712.0, 'debug/num_lat_loss': 1638.0, 'epoch': 1.85}

 18%|█▊        | 135/730 [12:05<51:36,  5.20s/it]
                                                 
{'train/ce_loss': 3.484375, 'train/diffusion_loss': 0.3159174621105194, 'epoch': 1.85}

 18%|█▊        | 135/730 [12:05<51:36,  5.20s/it]
                                                 
{'train/learning_rate_real': 2.3461306906195524e-05, 'epoch': 1.85}

 18%|█▊        | 135/730 [12:05<51:36,  5.20s/it]
                                                 
{'debug/num_tok_total': 3102.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 3102.0, 'debug/num_lat_loss': 1794.0, 'epoch': 1.85}

 18%|█▊        | 135/730 [12:06<51:36,  5.20s/it]
                                                 
{'train/ce_loss': 3.4375, 'train/diffusion_loss': 0.30301010608673096, 'epoch': 1.85}

 18%|█▊        | 135/730 [12:06<51:36,  5.20s/it]
                                                 
{'train/learning_rate_real': 2.3461306906195524e-05, 'epoch': 1.85}

 18%|█▊        | 135/730 [12:06<51:36,  5.20s/it]
 19%|█▊        | 136/730 [12:07<51:54,  5.24s/it]
                                                 
{'debug/num_tok_total': 2190.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2190.0, 'debug/num_lat_loss': 1772.0, 'epoch': 1.86}

 19%|█▊        | 136/730 [12:08<51:54,  5.24s/it]
                                                 
{'train/ce_loss': 3.515625, 'train/diffusion_loss': 0.5018863081932068, 'epoch': 1.86}

 19%|█▊        | 136/730 [12:08<51:54,  5.24s/it]
                                                 
{'train/learning_rate_real': 2.3434538556649515e-05, 'epoch': 1.86}

 19%|█▊        | 136/730 [12:08<51:54,  5.24s/it]
                                                 
{'debug/num_tok_total': 2203.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2203.0, 'debug/num_lat_loss': 1785.0, 'epoch': 1.86}

 19%|█▊        | 136/730 [12:09<51:54,  5.24s/it]
                                                 
{'train/ce_loss': 3.421875, 'train/diffusion_loss': 0.5082259774208069, 'epoch': 1.86}

 19%|█▊        | 136/730 [12:09<51:54,  5.24s/it]
                                                 
{'train/learning_rate_real': 2.3434538556649515e-05, 'epoch': 1.86}

 19%|█▊        | 136/730 [12:09<51:54,  5.24s/it]
                                                 
{'debug/num_tok_total': 2184.0, 'debug/num_tok_loss': 1749.0, 'debug/num_lat_total': 2184.0, 'debug/num_lat_loss': 1749.0, 'epoch': 1.86}

 19%|█▊        | 136/730 [12:10<51:54,  5.24s/it]
                                                 
{'train/ce_loss': 3.640625, 'train/diffusion_loss': 0.4796072542667389, 'epoch': 1.86}

 19%|█▊        | 136/730 [12:10<51:54,  5.24s/it]
                                                 
{'train/learning_rate_real': 2.3434538556649515e-05, 'epoch': 1.86}

 19%|█▊        | 136/730 [12:10<51:54,  5.24s/it]
                                                 
{'debug/num_tok_total': 2459.0, 'debug/num_tok_loss': 1800.0, 'debug/num_lat_total': 2459.0, 'debug/num_lat_loss': 1800.0, 'epoch': 1.86}

 19%|█▊        | 136/730 [12:11<51:54,  5.24s/it]
                                                 
{'train/ce_loss': 3.671875, 'train/diffusion_loss': 0.4622892439365387, 'epoch': 1.86}

 19%|█▊        | 136/730 [12:11<51:54,  5.24s/it]
                                                 
{'train/learning_rate_real': 2.3434538556649515e-05, 'epoch': 1.86}

 19%|█▊        | 136/730 [12:11<51:54,  5.24s/it]
 19%|█▉        | 137/730 [12:11<50:58,  5.16s/it]
                                                 
{'debug/num_tok_total': 2692.0, 'debug/num_tok_loss': 1810.0, 'debug/num_lat_total': 2692.0, 'debug/num_lat_loss': 1810.0, 'epoch': 1.88}

 19%|█▉        | 137/730 [12:13<50:58,  5.16s/it]
                                                 
{'train/ce_loss': 3.359375, 'train/diffusion_loss': 0.39755386114120483, 'epoch': 1.88}

 19%|█▉        | 137/730 [12:13<50:58,  5.16s/it]
                                                 
{'train/learning_rate_real': 2.340755491258248e-05, 'epoch': 1.88}

 19%|█▉        | 137/730 [12:13<50:58,  5.16s/it]
                                                 
{'debug/num_tok_total': 2460.0, 'debug/num_tok_loss': 1689.0, 'debug/num_lat_total': 2460.0, 'debug/num_lat_loss': 1689.0, 'epoch': 1.88}

 19%|█▉        | 137/730 [12:14<50:58,  5.16s/it]
                                                 
{'train/ce_loss': 3.5, 'train/diffusion_loss': 0.4218902289867401, 'epoch': 1.88}

 19%|█▉        | 137/730 [12:14<50:58,  5.16s/it]
                                                 
{'train/learning_rate_real': 2.340755491258248e-05, 'epoch': 1.88}

 19%|█▉        | 137/730 [12:14<50:58,  5.16s/it]
                                                 
{'debug/num_tok_total': 2195.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2195.0, 'debug/num_lat_loss': 1772.0, 'epoch': 1.88}

 19%|█▉        | 137/730 [12:15<50:58,  5.16s/it]
                                                 
{'train/ce_loss': 3.421875, 'train/diffusion_loss': 0.5013386607170105, 'epoch': 1.88}

 19%|█▉        | 137/730 [12:15<50:58,  5.16s/it]
                                                 
{'train/learning_rate_real': 2.340755491258248e-05, 'epoch': 1.88}

 19%|█▉        | 137/730 [12:15<50:58,  5.16s/it]
                                                 
{'debug/num_tok_total': 2611.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 2611.0, 'debug/num_lat_loss': 1767.0, 'epoch': 1.88}

 19%|█▉        | 137/730 [12:16<50:58,  5.16s/it]
                                                 
{'train/ce_loss': 3.578125, 'train/diffusion_loss': 0.4383437931537628, 'epoch': 1.88}

 19%|█▉        | 137/730 [12:16<50:58,  5.16s/it]
                                                 
{'train/learning_rate_real': 2.340755491258248e-05, 'epoch': 1.88}

 19%|█▉        | 137/730 [12:16<50:58,  5.16s/it]
 19%|█▉        | 138/730 [12:17<51:13,  5.19s/it]
                                                 
{'debug/num_tok_total': 2640.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2640.0, 'debug/num_lat_loss': 1773.0, 'epoch': 1.89}

 19%|█▉        | 138/730 [12:18<51:13,  5.19s/it]
                                                 
{'train/ce_loss': 3.359375, 'train/diffusion_loss': 0.3976190388202667, 'epoch': 1.89}

 19%|█▉        | 138/730 [12:18<51:13,  5.19s/it]
                                                 
{'train/learning_rate_real': 2.3380356505286216e-05, 'epoch': 1.89}

 19%|█▉        | 138/730 [12:18<51:13,  5.19s/it]
                                                 
{'debug/num_tok_total': 2621.0, 'debug/num_tok_loss': 1758.0, 'debug/num_lat_total': 2621.0, 'debug/num_lat_loss': 1758.0, 'epoch': 1.89}

 19%|█▉        | 138/730 [12:19<51:13,  5.19s/it]
                                                 
{'train/ce_loss': 3.71875, 'train/diffusion_loss': 0.40847277641296387, 'epoch': 1.89}

 19%|█▉        | 138/730 [12:19<51:13,  5.19s/it]
                                                 
{'train/learning_rate_real': 2.3380356505286216e-05, 'epoch': 1.89}

 19%|█▉        | 138/730 [12:19<51:13,  5.19s/it]
                                                 
{'debug/num_tok_total': 2660.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2660.0, 'debug/num_lat_loss': 1789.0, 'epoch': 1.89}

 19%|█▉        | 138/730 [12:20<51:13,  5.19s/it]
                                                 
{'train/ce_loss': 3.375, 'train/diffusion_loss': 0.42175373435020447, 'epoch': 1.89}

 19%|█▉        | 138/730 [12:20<51:13,  5.19s/it]
                                                 
{'train/learning_rate_real': 2.3380356505286216e-05, 'epoch': 1.89}

 19%|█▉        | 138/730 [12:20<51:13,  5.19s/it]
                                                 
{'debug/num_tok_total': 2843.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2843.0, 'debug/num_lat_loss': 1778.0, 'epoch': 1.89}

 19%|█▉        | 138/730 [12:22<51:13,  5.19s/it]
                                                 
{'train/ce_loss': 3.671875, 'train/diffusion_loss': 0.38470953702926636, 'epoch': 1.89}

 19%|█▉        | 138/730 [12:22<51:13,  5.19s/it]
                                                 
{'train/learning_rate_real': 2.3380356505286216e-05, 'epoch': 1.89}

 19%|█▉        | 138/730 [12:22<51:13,  5.19s/it]
 19%|█▉        | 139/730 [12:22<51:43,  5.25s/it]
                                                 
{'debug/num_tok_total': 2231.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2231.0, 'debug/num_lat_loss': 1796.0, 'epoch': 1.9}

 19%|█▉        | 139/730 [12:23<51:43,  5.25s/it]
                                                 
{'train/ce_loss': 3.46875, 'train/diffusion_loss': 0.5015687346458435, 'epoch': 1.9}

 19%|█▉        | 139/730 [12:23<51:43,  5.25s/it]
                                                 
{'train/learning_rate_real': 2.335294387028109e-05, 'epoch': 1.9}

 19%|█▉        | 139/730 [12:23<51:43,  5.25s/it]
                                                 
{'debug/num_tok_total': 2364.0, 'debug/num_tok_loss': 1723.0, 'debug/num_lat_total': 2364.0, 'debug/num_lat_loss': 1723.0, 'epoch': 1.9}

 19%|█▉        | 139/730 [12:24<51:43,  5.25s/it]
                                                 
{'train/ce_loss': 3.390625, 'train/diffusion_loss': 0.46811532974243164, 'epoch': 1.9}

 19%|█▉        | 139/730 [12:24<51:43,  5.25s/it]
                                                 
{'train/learning_rate_real': 2.335294387028109e-05, 'epoch': 1.9}

 19%|█▉        | 139/730 [12:24<51:43,  5.25s/it]
                                                 
{'debug/num_tok_total': 2648.0, 'debug/num_tok_loss': 1680.0, 'debug/num_lat_total': 2648.0, 'debug/num_lat_loss': 1680.0, 'epoch': 1.9}

 19%|█▉        | 139/730 [12:26<51:43,  5.25s/it]
                                                 
{'train/ce_loss': 3.453125, 'train/diffusion_loss': 0.3847237229347229, 'epoch': 1.9}

 19%|█▉        | 139/730 [12:26<51:43,  5.25s/it]
                                                 
{'train/learning_rate_real': 2.335294387028109e-05, 'epoch': 1.9}

 19%|█▉        | 139/730 [12:26<51:43,  5.25s/it]
                                                 
{'debug/num_tok_total': 3114.0, 'debug/num_tok_loss': 1806.0, 'debug/num_lat_total': 3114.0, 'debug/num_lat_loss': 1806.0, 'epoch': 1.9}

 19%|█▉        | 139/730 [12:27<51:43,  5.25s/it]
                                                 
{'train/ce_loss': 3.515625, 'train/diffusion_loss': 0.32092833518981934, 'epoch': 1.9}

 19%|█▉        | 139/730 [12:27<51:43,  5.25s/it]
                                                 
{'train/learning_rate_real': 2.335294387028109e-05, 'epoch': 1.9}

 19%|█▉        | 139/730 [12:27<51:43,  5.25s/it]03/16/2026 06:59:59 - INFO - __main__ - LoRA debug step 140: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 19%|█▉        | 140/730 [12:27<52:00,  5.29s/it]
                                                 
{'loss': 2.9031, 'grad_norm': 1.6461002826690674, 'learning_rate': 2.335294387028109e-05, 'epoch': 1.92}

 19%|█▉        | 140/730 [12:27<52:00,  5.29s/it]
                                                 
{'debug/num_tok_total': 2182.0, 'debug/num_tok_loss': 1760.0, 'debug/num_lat_total': 2182.0, 'debug/num_lat_loss': 1760.0, 'epoch': 1.92}

 19%|█▉        | 140/730 [12:29<52:00,  5.29s/it]
                                                 
{'train/ce_loss': 3.390625, 'train/diffusion_loss': 0.5311415195465088, 'epoch': 1.92}

 19%|█▉        | 140/730 [12:29<52:00,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.3325317547305485e-05, 'epoch': 1.92}

 19%|█▉        | 140/730 [12:29<52:00,  5.29s/it]
                                                 
{'debug/num_tok_total': 2197.0, 'debug/num_tok_loss': 1607.0, 'debug/num_lat_total': 2197.0, 'debug/num_lat_loss': 1607.0, 'epoch': 1.92}

 19%|█▉        | 140/730 [12:30<52:00,  5.29s/it]
                                                 
{'train/ce_loss': 3.375, 'train/diffusion_loss': 0.45163846015930176, 'epoch': 1.92}

 19%|█▉        | 140/730 [12:30<52:00,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.3325317547305485e-05, 'epoch': 1.92}

 19%|█▉        | 140/730 [12:30<52:00,  5.29s/it]
                                                 
{'debug/num_tok_total': 2208.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2208.0, 'debug/num_lat_loss': 1782.0, 'epoch': 1.92}

 19%|█▉        | 140/730 [12:31<52:00,  5.29s/it]
                                                 
{'train/ce_loss': 3.53125, 'train/diffusion_loss': 0.4920024275779724, 'epoch': 1.92}

 19%|█▉        | 140/730 [12:31<52:00,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.3325317547305485e-05, 'epoch': 1.92}

 19%|█▉        | 140/730 [12:31<52:00,  5.29s/it]
                                                 
{'debug/num_tok_total': 2617.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 2617.0, 'debug/num_lat_loss': 1768.0, 'epoch': 1.92}

 19%|█▉        | 140/730 [12:32<52:00,  5.29s/it]
                                                 
{'train/ce_loss': 3.40625, 'train/diffusion_loss': 0.43166494369506836, 'epoch': 1.92}

 19%|█▉        | 140/730 [12:32<52:00,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.3325317547305485e-05, 'epoch': 1.92}

 19%|█▉        | 140/730 [12:32<52:00,  5.29s/it]
 19%|█▉        | 141/730 [12:32<51:03,  5.20s/it]
                                                 
{'debug/num_tok_total': 2813.0, 'debug/num_tok_loss': 1753.0, 'debug/num_lat_total': 2813.0, 'debug/num_lat_loss': 1753.0, 'epoch': 1.93}

 19%|█▉        | 141/730 [12:34<51:03,  5.20s/it]
                                                 
{'train/ce_loss': 3.75, 'train/diffusion_loss': 0.3822256922721863, 'epoch': 1.93}

 19%|█▉        | 141/730 [12:34<51:03,  5.20s/it]
                                                 
{'train/learning_rate_real': 2.3297478080305162e-05, 'epoch': 1.93}

 19%|█▉        | 141/730 [12:34<51:03,  5.20s/it]
                                                 
{'debug/num_tok_total': 2649.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2649.0, 'debug/num_lat_loss': 1790.0, 'epoch': 1.93}

 19%|█▉        | 141/730 [12:35<51:03,  5.20s/it]
                                                 
{'train/ce_loss': 4.0, 'train/diffusion_loss': 0.3977663516998291, 'epoch': 1.93}

 19%|█▉        | 141/730 [12:35<51:03,  5.20s/it]
                                                 
{'train/learning_rate_real': 2.3297478080305162e-05, 'epoch': 1.93}

 19%|█▉        | 141/730 [12:35<51:03,  5.20s/it]
                                                 
{'debug/num_tok_total': 2903.0, 'debug/num_tok_loss': 1805.0, 'debug/num_lat_total': 2903.0, 'debug/num_lat_loss': 1805.0, 'epoch': 1.93}

 19%|█▉        | 141/730 [12:36<51:03,  5.20s/it]
                                                 
{'train/ce_loss': 3.421875, 'train/diffusion_loss': 0.3696616291999817, 'epoch': 1.93}

 19%|█▉        | 141/730 [12:36<51:03,  5.20s/it]
                                                 
{'train/learning_rate_real': 2.3297478080305162e-05, 'epoch': 1.93}

 19%|█▉        | 141/730 [12:36<51:03,  5.20s/it]
                                                 
{'debug/num_tok_total': 2076.0, 'debug/num_tok_loss': 1604.0, 'debug/num_lat_total': 2076.0, 'debug/num_lat_loss': 1604.0, 'epoch': 1.93}

 19%|█▉        | 141/730 [12:37<51:03,  5.20s/it]
                                                 
{'train/ce_loss': 3.390625, 'train/diffusion_loss': 0.4954376816749573, 'epoch': 1.93}

 19%|█▉        | 141/730 [12:37<51:03,  5.20s/it]
                                                 
{'train/learning_rate_real': 2.3297478080305162e-05, 'epoch': 1.93}

 19%|█▉        | 141/730 [12:37<51:03,  5.20s/it]
 19%|█▉        | 142/730 [12:38<51:15,  5.23s/it]
                                                 
{'debug/num_tok_total': 2856.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2856.0, 'debug/num_lat_loss': 1781.0, 'epoch': 1.95}

 19%|█▉        | 142/730 [12:39<51:15,  5.23s/it]
                                                 
{'train/ce_loss': 3.40625, 'train/diffusion_loss': 0.3527717590332031, 'epoch': 1.95}

 19%|█▉        | 142/730 [12:39<51:15,  5.23s/it]
                                                 
{'train/learning_rate_real': 2.3269426017422576e-05, 'epoch': 1.95}

 19%|█▉        | 142/730 [12:39<51:15,  5.23s/it]
                                                 
{'debug/num_tok_total': 2470.0, 'debug/num_tok_loss': 1809.0, 'debug/num_lat_total': 2470.0, 'debug/num_lat_loss': 1809.0, 'epoch': 1.95}

 19%|█▉        | 142/730 [12:40<51:15,  5.23s/it]
                                                 
{'train/ce_loss': 3.390625, 'train/diffusion_loss': 0.4418691396713257, 'epoch': 1.95}

 19%|█▉        | 142/730 [12:40<51:15,  5.23s/it]
                                                 
{'train/learning_rate_real': 2.3269426017422576e-05, 'epoch': 1.95}

 19%|█▉        | 142/730 [12:40<51:15,  5.23s/it]
                                                 
{'debug/num_tok_total': 2610.0, 'debug/num_tok_loss': 1747.0, 'debug/num_lat_total': 2610.0, 'debug/num_lat_loss': 1747.0, 'epoch': 1.95}

 19%|█▉        | 142/730 [12:41<51:15,  5.23s/it]
                                                 
{'train/ce_loss': 3.59375, 'train/diffusion_loss': 0.41341838240623474, 'epoch': 1.95}

 19%|█▉        | 142/730 [12:42<51:15,  5.23s/it]
                                                 
{'train/learning_rate_real': 2.3269426017422576e-05, 'epoch': 1.95}

 19%|█▉        | 142/730 [12:42<51:15,  5.23s/it]
                                                 
{'debug/num_tok_total': 2343.0, 'debug/num_tok_loss': 1718.0, 'debug/num_lat_total': 2343.0, 'debug/num_lat_loss': 1718.0, 'epoch': 1.95}

 19%|█▉        | 142/730 [12:43<51:15,  5.23s/it]
                                                 
{'train/ce_loss': 3.5, 'train/diffusion_loss': 0.476648211479187, 'epoch': 1.95}

 19%|█▉        | 142/730 [12:43<51:15,  5.23s/it]
                                                 
{'train/learning_rate_real': 2.3269426017422576e-05, 'epoch': 1.95}

 19%|█▉        | 142/730 [12:43<51:15,  5.23s/it]
 20%|█▉        | 143/730 [12:43<51:18,  5.24s/it]
                                                 
{'debug/num_tok_total': 2221.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2221.0, 'debug/num_lat_loss': 1792.0, 'epoch': 1.96}

 20%|█▉        | 143/730 [12:44<51:18,  5.24s/it]
                                                 
{'train/ce_loss': 3.34375, 'train/diffusion_loss': 0.503341794013977, 'epoch': 1.96}

 20%|█▉        | 143/730 [12:44<51:18,  5.24s/it]
                                                 
{'train/learning_rate_real': 2.3241161910986053e-05, 'epoch': 1.96}

 20%|█▉        | 143/730 [12:44<51:18,  5.24s/it]
                                                 
{'debug/num_tok_total': 2588.0, 'debug/num_tok_loss': 1712.0, 'debug/num_lat_total': 2588.0, 'debug/num_lat_loss': 1712.0, 'epoch': 1.96}

 20%|█▉        | 143/730 [12:45<51:18,  5.24s/it]
                                                 
{'train/ce_loss': 3.453125, 'train/diffusion_loss': 0.39334893226623535, 'epoch': 1.96}

 20%|█▉        | 143/730 [12:45<51:18,  5.24s/it]
                                                 
{'train/learning_rate_real': 2.3241161910986053e-05, 'epoch': 1.96}

 20%|█▉        | 143/730 [12:45<51:18,  5.24s/it]
                                                 
{'debug/num_tok_total': 2928.0, 'debug/num_tok_loss': 1819.0, 'debug/num_lat_total': 2928.0, 'debug/num_lat_loss': 1819.0, 'epoch': 1.96}

 20%|█▉        | 143/730 [12:47<51:18,  5.24s/it]
                                                 
{'train/ce_loss': 3.484375, 'train/diffusion_loss': 0.3769533932209015, 'epoch': 1.96}

 20%|█▉        | 143/730 [12:47<51:18,  5.24s/it]
                                                 
{'train/learning_rate_real': 2.3241161910986053e-05, 'epoch': 1.96}

 20%|█▉        | 143/730 [12:47<51:18,  5.24s/it]
                                                 
{'debug/num_tok_total': 2599.0, 'debug/num_tok_loss': 1741.0, 'debug/num_lat_total': 2599.0, 'debug/num_lat_loss': 1741.0, 'epoch': 1.96}

 20%|█▉        | 143/730 [12:48<51:18,  5.24s/it]
                                                 
{'train/ce_loss': 3.328125, 'train/diffusion_loss': 0.41423624753952026, 'epoch': 1.96}

 20%|█▉        | 143/730 [12:48<51:18,  5.24s/it]
                                                 
{'train/learning_rate_real': 2.3241161910986053e-05, 'epoch': 1.96}

 20%|█▉        | 143/730 [12:48<51:18,  5.24s/it]
 20%|█▉        | 144/730 [12:48<51:07,  5.23s/it]
                                                 
{'debug/num_tok_total': 2174.0, 'debug/num_tok_loss': 1381.0, 'debug/num_lat_total': 2174.0, 'debug/num_lat_loss': 1381.0, 'epoch': 1.97}

 20%|█▉        | 144/730 [12:50<51:07,  5.23s/it]
                                                 
{'train/ce_loss': 3.46875, 'train/diffusion_loss': 0.3866157829761505, 'epoch': 1.97}

 20%|█▉        | 144/730 [12:50<51:07,  5.23s/it]
                                                 
{'train/learning_rate_real': 2.321268631749894e-05, 'epoch': 1.97}

 20%|█▉        | 144/730 [12:50<51:07,  5.23s/it]
                                                 
{'debug/num_tok_total': 2196.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2196.0, 'debug/num_lat_loss': 1763.0, 'epoch': 1.97}

 20%|█▉        | 144/730 [12:51<51:07,  5.23s/it]
                                                 
{'train/ce_loss': 3.40625, 'train/diffusion_loss': 0.5139296054840088, 'epoch': 1.97}

 20%|█▉        | 144/730 [12:51<51:07,  5.23s/it]
                                                 
{'train/learning_rate_real': 2.321268631749894e-05, 'epoch': 1.97}

 20%|█▉        | 144/730 [12:51<51:07,  5.23s/it]
                                                 
{'debug/num_tok_total': 3262.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 3262.0, 'debug/num_lat_loss': 1763.0, 'epoch': 1.97}

 20%|█▉        | 144/730 [12:52<51:07,  5.23s/it]
                                                 
{'train/ce_loss': 3.578125, 'train/diffusion_loss': 0.26967543363571167, 'epoch': 1.97}

 20%|█▉        | 144/730 [12:52<51:07,  5.23s/it]
                                                 
{'train/learning_rate_real': 2.321268631749894e-05, 'epoch': 1.97}

 20%|█▉        | 144/730 [12:52<51:07,  5.23s/it]
                                                 
{'debug/num_tok_total': 2860.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2860.0, 'debug/num_lat_loss': 1786.0, 'epoch': 1.97}

 20%|█▉        | 144/730 [12:53<51:07,  5.23s/it]
                                                 
{'train/ce_loss': 3.390625, 'train/diffusion_loss': 0.37319138646125793, 'epoch': 1.97}

 20%|█▉        | 144/730 [12:53<51:07,  5.23s/it]
                                                 
{'train/learning_rate_real': 2.321268631749894e-05, 'epoch': 1.97}

 20%|█▉        | 144/730 [12:53<51:07,  5.23s/it]
 20%|█▉        | 145/730 [12:54<51:32,  5.29s/it]
                                                 
{'debug/num_tok_total': 2839.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2839.0, 'debug/num_lat_loss': 1781.0, 'epoch': 1.99}

 20%|█▉        | 145/730 [12:55<51:32,  5.29s/it]
                                                 
{'train/ce_loss': 3.265625, 'train/diffusion_loss': 0.3564004898071289, 'epoch': 1.99}

 20%|█▉        | 145/730 [12:55<51:32,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.318399979762863e-05, 'epoch': 1.99}

 20%|█▉        | 145/730 [12:55<51:32,  5.29s/it]
                                                 
{'debug/num_tok_total': 2254.0, 'debug/num_tok_loss': 1600.0, 'debug/num_lat_total': 2254.0, 'debug/num_lat_loss': 1600.0, 'epoch': 1.99}

 20%|█▉        | 145/730 [12:56<51:32,  5.29s/it]
                                                 
{'train/ce_loss': 3.4375, 'train/diffusion_loss': 0.46334365010261536, 'epoch': 1.99}

 20%|█▉        | 145/730 [12:56<51:32,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.318399979762863e-05, 'epoch': 1.99}

 20%|█▉        | 145/730 [12:56<51:32,  5.29s/it]
                                                 
{'debug/num_tok_total': 2466.0, 'debug/num_tok_loss': 1810.0, 'debug/num_lat_total': 2466.0, 'debug/num_lat_loss': 1810.0, 'epoch': 1.99}

 20%|█▉        | 145/730 [12:57<51:32,  5.29s/it]
                                                 
{'train/ce_loss': 3.40625, 'train/diffusion_loss': 0.47243303060531616, 'epoch': 1.99}

 20%|█▉        | 145/730 [12:57<51:32,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.318399979762863e-05, 'epoch': 1.99}

 20%|█▉        | 145/730 [12:57<51:32,  5.29s/it]
                                                 
{'debug/num_tok_total': 451.0, 'debug/num_tok_loss': 451.0, 'debug/num_lat_total': 451.0, 'debug/num_lat_loss': 451.0, 'epoch': 1.99}

 20%|█▉        | 145/730 [12:58<51:32,  5.29s/it]
                                                 
{'train/ce_loss': 3.25, 'train/diffusion_loss': 0.5414097309112549, 'epoch': 1.99}

 20%|█▉        | 145/730 [12:58<51:32,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.318399979762863e-05, 'epoch': 1.99}

 20%|█▉        | 145/730 [12:58<51:32,  5.29s/it]
 20%|██        | 146/730 [12:58<48:33,  4.99s/it]
                                                 
{'debug/num_tok_total': 2300.0, 'debug/num_tok_loss': 1711.0, 'debug/num_lat_total': 2300.0, 'debug/num_lat_loss': 1711.0, 'epoch': 2.0}

 20%|██        | 146/730 [12:59<48:33,  4.99s/it]
                                                 
{'train/ce_loss': 3.34375, 'train/diffusion_loss': 0.4503061771392822, 'epoch': 2.0}

 20%|██        | 146/730 [12:59<48:33,  4.99s/it]
                                                 
{'train/learning_rate_real': 2.3155102916195535e-05, 'epoch': 2.0}

 20%|██        | 146/730 [12:59<48:33,  4.99s/it]
                                                 
{'debug/num_tok_total': 2631.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2631.0, 'debug/num_lat_loss': 1763.0, 'epoch': 2.0}

 20%|██        | 146/730 [13:00<48:33,  4.99s/it]
                                                 
{'train/ce_loss': 3.3125, 'train/diffusion_loss': 0.42279499769210815, 'epoch': 2.0}

 20%|██        | 146/730 [13:00<48:33,  4.99s/it]
                                                 
{'train/learning_rate_real': 2.3155102916195535e-05, 'epoch': 2.0}

 20%|██        | 146/730 [13:00<48:33,  4.99s/it]
                                                 
{'debug/num_tok_total': 2395.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 2395.0, 'debug/num_lat_loss': 1769.0, 'epoch': 2.0}

 20%|██        | 146/730 [13:01<48:33,  4.99s/it]
                                                 
{'train/ce_loss': 3.453125, 'train/diffusion_loss': 0.47049665451049805, 'epoch': 2.0}

 20%|██        | 146/730 [13:01<48:33,  4.99s/it]
                                                 
{'train/learning_rate_real': 2.3155102916195535e-05, 'epoch': 2.0}

 20%|██        | 146/730 [13:01<48:33,  4.99s/it]
                                                 
{'debug/num_tok_total': 2638.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2638.0, 'debug/num_lat_loss': 1775.0, 'epoch': 2.0}

 20%|██        | 146/730 [13:03<48:33,  4.99s/it]
                                                 
{'train/ce_loss': 3.203125, 'train/diffusion_loss': 0.37802591919898987, 'epoch': 2.0}

 20%|██        | 146/730 [13:03<48:33,  4.99s/it]
                                                 
{'train/learning_rate_real': 2.3155102916195535e-05, 'epoch': 2.0}

 20%|██        | 146/730 [13:03<48:33,  4.99s/it]
 20%|██        | 147/730 [13:03<48:55,  5.04s/it]
                                                 
{'debug/num_tok_total': 2408.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 2408.0, 'debug/num_lat_loss': 1768.0, 'epoch': 2.01}

 20%|██        | 147/730 [13:04<48:55,  5.04s/it]
                                                 
{'train/ce_loss': 3.25, 'train/diffusion_loss': 0.45496639609336853, 'epoch': 2.01}

 20%|██        | 147/730 [13:04<48:55,  5.04s/it]
                                                 
{'train/learning_rate_real': 2.3125996242161953e-05, 'epoch': 2.01}

 20%|██        | 147/730 [13:04<48:55,  5.04s/it]
                                                 
{'debug/num_tok_total': 2851.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2851.0, 'debug/num_lat_loss': 1776.0, 'epoch': 2.01}

 20%|██        | 147/730 [13:05<48:55,  5.04s/it]
                                                 
{'train/ce_loss': 3.375, 'train/diffusion_loss': 0.37154674530029297, 'epoch': 2.01}

 20%|██        | 147/730 [13:05<48:55,  5.04s/it]
                                                 
{'train/learning_rate_real': 2.3125996242161953e-05, 'epoch': 2.01}

 20%|██        | 147/730 [13:05<48:55,  5.04s/it]
                                                 
{'debug/num_tok_total': 2051.0, 'debug/num_tok_loss': 1617.0, 'debug/num_lat_total': 2051.0, 'debug/num_lat_loss': 1617.0, 'epoch': 2.01}

 20%|██        | 147/730 [13:07<48:55,  5.04s/it]
                                                 
{'train/ce_loss': 3.453125, 'train/diffusion_loss': 0.46421128511428833, 'epoch': 2.01}

 20%|██        | 147/730 [13:07<48:55,  5.04s/it]
                                                 
{'train/learning_rate_real': 2.3125996242161953e-05, 'epoch': 2.01}

 20%|██        | 147/730 [13:07<48:55,  5.04s/it]
                                                 
{'debug/num_tok_total': 2429.0, 'debug/num_tok_loss': 1585.0, 'debug/num_lat_total': 2429.0, 'debug/num_lat_loss': 1585.0, 'epoch': 2.01}

 20%|██        | 147/730 [13:08<48:55,  5.04s/it]
                                                 
{'train/ce_loss': 3.421875, 'train/diffusion_loss': 0.3745909631252289, 'epoch': 2.01}

 20%|██        | 147/730 [13:08<48:55,  5.04s/it]
                                                 
{'train/learning_rate_real': 2.3125996242161953e-05, 'epoch': 2.01}

 20%|██        | 147/730 [13:08<48:55,  5.04s/it]
 20%|██        | 148/730 [13:08<48:58,  5.05s/it]
                                                 
{'debug/num_tok_total': 2199.0, 'debug/num_tok_loss': 1664.0, 'debug/num_lat_total': 2199.0, 'debug/num_lat_loss': 1664.0, 'epoch': 2.03}

 20%|██        | 148/730 [13:09<48:58,  5.05s/it]
                                                 
{'train/ce_loss': 3.296875, 'train/diffusion_loss': 0.4661211371421814, 'epoch': 2.03}

 20%|██        | 148/730 [13:09<48:58,  5.05s/it]
                                                 
{'train/learning_rate_real': 2.3096680348620886e-05, 'epoch': 2.03}

 20%|██        | 148/730 [13:09<48:58,  5.05s/it]
                                                 
{'debug/num_tok_total': 2827.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 2827.0, 'debug/num_lat_loss': 1768.0, 'epoch': 2.03}

 20%|██        | 148/730 [13:11<48:58,  5.05s/it]
                                                 
{'train/ce_loss': 3.265625, 'train/diffusion_loss': 0.3533343970775604, 'epoch': 2.03}

 20%|██        | 148/730 [13:11<48:58,  5.05s/it]
                                                 
{'train/learning_rate_real': 2.3096680348620886e-05, 'epoch': 2.03}

 20%|██        | 148/730 [13:11<48:58,  5.05s/it]
                                                 
{'debug/num_tok_total': 2686.0, 'debug/num_tok_loss': 1817.0, 'debug/num_lat_total': 2686.0, 'debug/num_lat_loss': 1817.0, 'epoch': 2.03}

 20%|██        | 148/730 [13:12<48:58,  5.05s/it]
                                                 
{'train/ce_loss': 3.453125, 'train/diffusion_loss': 0.41130325198173523, 'epoch': 2.03}

 20%|██        | 148/730 [13:12<48:58,  5.05s/it]
                                                 
{'train/learning_rate_real': 2.3096680348620886e-05, 'epoch': 2.03}

 20%|██        | 148/730 [13:12<48:58,  5.05s/it]
                                                 
{'debug/num_tok_total': 2227.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2227.0, 'debug/num_lat_loss': 1782.0, 'epoch': 2.03}

 20%|██        | 148/730 [13:13<48:58,  5.05s/it]
                                                 
{'train/ce_loss': 3.265625, 'train/diffusion_loss': 0.5083057880401611, 'epoch': 2.03}

 20%|██        | 148/730 [13:13<48:58,  5.05s/it]
                                                 
{'train/learning_rate_real': 2.3096680348620886e-05, 'epoch': 2.03}

 20%|██        | 148/730 [13:13<48:58,  5.05s/it]
 20%|██        | 149/730 [13:13<49:06,  5.07s/it]
                                                 
{'debug/num_tok_total': 2877.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2877.0, 'debug/num_lat_loss': 1802.0, 'epoch': 2.04}

 20%|██        | 149/730 [13:15<49:06,  5.07s/it]
                                                 
{'train/ce_loss': 3.28125, 'train/diffusion_loss': 0.37254709005355835, 'epoch': 2.04}

 20%|██        | 149/730 [13:15<49:06,  5.07s/it]
                                                 
{'train/learning_rate_real': 2.3067155812784734e-05, 'epoch': 2.04}

 20%|██        | 149/730 [13:15<49:06,  5.07s/it]
                                                 
{'debug/num_tok_total': 2709.0, 'debug/num_tok_loss': 1629.0, 'debug/num_lat_total': 2709.0, 'debug/num_lat_loss': 1629.0, 'epoch': 2.04}

 20%|██        | 149/730 [13:16<49:06,  5.07s/it]
                                                 
{'train/ce_loss': 3.46875, 'train/diffusion_loss': 0.3475020229816437, 'epoch': 2.04}

 20%|██        | 149/730 [13:16<49:06,  5.07s/it]
                                                 
{'train/learning_rate_real': 2.3067155812784734e-05, 'epoch': 2.04}

 20%|██        | 149/730 [13:16<49:06,  5.07s/it]
                                                 
{'debug/num_tok_total': 2671.0, 'debug/num_tok_loss': 1807.0, 'debug/num_lat_total': 2671.0, 'debug/num_lat_loss': 1807.0, 'epoch': 2.04}

 20%|██        | 149/730 [13:17<49:06,  5.07s/it]
                                                 
{'train/ce_loss': 3.21875, 'train/diffusion_loss': 0.4053395688533783, 'epoch': 2.04}

 20%|██        | 149/730 [13:17<49:06,  5.07s/it]
                                                 
{'train/learning_rate_real': 2.3067155812784734e-05, 'epoch': 2.04}

 20%|██        | 149/730 [13:17<49:06,  5.07s/it]
                                                 
{'debug/num_tok_total': 2659.0, 'debug/num_tok_loss': 1804.0, 'debug/num_lat_total': 2659.0, 'debug/num_lat_loss': 1804.0, 'epoch': 2.04}

 20%|██        | 149/730 [13:18<49:06,  5.07s/it]
                                                 
{'train/ce_loss': 3.53125, 'train/diffusion_loss': 0.39869940280914307, 'epoch': 2.04}

 20%|██        | 149/730 [13:18<49:06,  5.07s/it]
                                                 
{'train/learning_rate_real': 2.3067155812784734e-05, 'epoch': 2.04}

 20%|██        | 149/730 [13:18<49:06,  5.07s/it]03/16/2026 07:00:51 - INFO - __main__ - LoRA debug step 150: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 21%|██        | 150/730 [13:19<50:29,  5.22s/it]
                                                 
{'loss': 2.9196, 'grad_norm': 1.5437519550323486, 'learning_rate': 2.3067155812784734e-05, 'epoch': 2.05}

 21%|██        | 150/730 [13:19<50:29,  5.22s/it]
                                                 
{'debug/num_tok_total': 3083.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 3083.0, 'debug/num_lat_loss': 1795.0, 'epoch': 2.05}

 21%|██        | 150/730 [13:20<50:29,  5.22s/it]
                                                 
{'train/ce_loss': 3.21875, 'train/diffusion_loss': 0.3080335855484009, 'epoch': 2.05}

 21%|██        | 150/730 [13:20<50:29,  5.22s/it]
                                                 
{'train/learning_rate_real': 2.303742321597394e-05, 'epoch': 2.05}

 21%|██        | 150/730 [13:20<50:29,  5.22s/it]
                                                 
{'debug/num_tok_total': 2548.0, 'debug/num_tok_loss': 1731.0, 'debug/num_lat_total': 2548.0, 'debug/num_lat_loss': 1731.0, 'epoch': 2.05}

 21%|██        | 150/730 [13:21<50:29,  5.22s/it]
                                                 
{'train/ce_loss': 3.21875, 'train/diffusion_loss': 0.4144803285598755, 'epoch': 2.05}

 21%|██        | 150/730 [13:21<50:29,  5.22s/it]
                                                 
{'train/learning_rate_real': 2.303742321597394e-05, 'epoch': 2.05}

 21%|██        | 150/730 [13:21<50:29,  5.22s/it]
                                                 
{'debug/num_tok_total': 2803.0, 'debug/num_tok_loss': 1750.0, 'debug/num_lat_total': 2803.0, 'debug/num_lat_loss': 1750.0, 'epoch': 2.05}

 21%|██        | 150/730 [13:23<50:29,  5.22s/it]
                                                 
{'train/ce_loss': 3.34375, 'train/diffusion_loss': 0.39247575402259827, 'epoch': 2.05}

 21%|██        | 150/730 [13:23<50:29,  5.22s/it]
                                                 
{'train/learning_rate_real': 2.303742321597394e-05, 'epoch': 2.05}

 21%|██        | 150/730 [13:23<50:29,  5.22s/it]
                                                 
{'debug/num_tok_total': 2817.0, 'debug/num_tok_loss': 1755.0, 'debug/num_lat_total': 2817.0, 'debug/num_lat_loss': 1755.0, 'epoch': 2.05}

 21%|██        | 150/730 [13:24<50:29,  5.22s/it]
                                                 
{'train/ce_loss': 3.1875, 'train/diffusion_loss': 0.3341597020626068, 'epoch': 2.05}

 21%|██        | 150/730 [13:24<50:29,  5.22s/it]
                                                 
{'train/learning_rate_real': 2.303742321597394e-05, 'epoch': 2.05}

 21%|██        | 150/730 [13:24<50:29,  5.22s/it]
 21%|██        | 151/730 [13:24<51:25,  5.33s/it]
                                                 
{'debug/num_tok_total': 2384.0, 'debug/num_tok_loss': 1751.0, 'debug/num_lat_total': 2384.0, 'debug/num_lat_loss': 1751.0, 'epoch': 2.07}

 21%|██        | 151/730 [13:26<51:25,  5.33s/it]
                                                 
{'train/ce_loss': 3.1875, 'train/diffusion_loss': 0.40776801109313965, 'epoch': 2.07}

 21%|██        | 151/730 [13:26<51:25,  5.33s/it]
                                                 
{'train/learning_rate_real': 2.3007483143605542e-05, 'epoch': 2.07}

 21%|██        | 151/730 [13:26<51:25,  5.33s/it]
                                                 
{'debug/num_tok_total': 2636.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2636.0, 'debug/num_lat_loss': 1776.0, 'epoch': 2.07}

 21%|██        | 151/730 [13:27<51:25,  5.33s/it]
                                                 
{'train/ce_loss': 3.3125, 'train/diffusion_loss': 0.41128188371658325, 'epoch': 2.07}

 21%|██        | 151/730 [13:27<51:25,  5.33s/it]
                                                 
{'train/learning_rate_real': 2.3007483143605542e-05, 'epoch': 2.07}

 21%|██        | 151/730 [13:27<51:25,  5.33s/it]
                                                 
{'debug/num_tok_total': 2659.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2659.0, 'debug/num_lat_loss': 1784.0, 'epoch': 2.07}

 21%|██        | 151/730 [13:28<51:25,  5.33s/it]
                                                 
{'train/ce_loss': 3.203125, 'train/diffusion_loss': 0.41530677676200867, 'epoch': 2.07}

 21%|██        | 151/730 [13:28<51:25,  5.33s/it]
                                                 
{'train/learning_rate_real': 2.3007483143605542e-05, 'epoch': 2.07}

 21%|██        | 151/730 [13:28<51:25,  5.33s/it]
                                                 
{'debug/num_tok_total': 2288.0, 'debug/num_tok_loss': 1642.0, 'debug/num_lat_total': 2288.0, 'debug/num_lat_loss': 1642.0, 'epoch': 2.07}

 21%|██        | 151/730 [13:29<51:25,  5.33s/it]
                                                 
{'train/ce_loss': 3.3125, 'train/diffusion_loss': 0.42171525955200195, 'epoch': 2.07}

 21%|██        | 151/730 [13:29<51:25,  5.33s/it]
                                                 
{'train/learning_rate_real': 2.3007483143605542e-05, 'epoch': 2.07}

 21%|██        | 151/730 [13:29<51:25,  5.33s/it]
 21%|██        | 152/730 [13:30<50:51,  5.28s/it]
                                                 
{'debug/num_tok_total': 2645.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2645.0, 'debug/num_lat_loss': 1791.0, 'epoch': 2.08}

 21%|██        | 152/730 [13:31<50:51,  5.28s/it]
                                                 
{'train/ce_loss': 3.21875, 'train/diffusion_loss': 0.419938862323761, 'epoch': 2.08}

 21%|██        | 152/730 [13:31<50:51,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.2977336185181646e-05, 'epoch': 2.08}

 21%|██        | 152/730 [13:31<50:51,  5.28s/it]
                                                 
{'debug/num_tok_total': 3008.0, 'debug/num_tok_loss': 1746.0, 'debug/num_lat_total': 3008.0, 'debug/num_lat_loss': 1746.0, 'epoch': 2.08}

 21%|██        | 152/730 [13:32<50:51,  5.28s/it]
                                                 
{'train/ce_loss': 3.3125, 'train/diffusion_loss': 0.29541534185409546, 'epoch': 2.08}

 21%|██        | 152/730 [13:32<50:51,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.2977336185181646e-05, 'epoch': 2.08}

 21%|██        | 152/730 [13:32<50:51,  5.28s/it]
                                                 
{'debug/num_tok_total': 2838.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2838.0, 'debug/num_lat_loss': 1771.0, 'epoch': 2.08}

 21%|██        | 152/730 [13:33<50:51,  5.28s/it]
                                                 
{'train/ce_loss': 3.171875, 'train/diffusion_loss': 0.3496735095977783, 'epoch': 2.08}

 21%|██        | 152/730 [13:33<50:51,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.2977336185181646e-05, 'epoch': 2.08}

 21%|██        | 152/730 [13:33<50:51,  5.28s/it]
                                                 
{'debug/num_tok_total': 2391.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2391.0, 'debug/num_lat_loss': 1766.0, 'epoch': 2.08}

 21%|██        | 152/730 [13:35<50:51,  5.28s/it]
                                                 
{'train/ce_loss': 3.421875, 'train/diffusion_loss': 0.4392234981060028, 'epoch': 2.08}

 21%|██        | 152/730 [13:35<50:51,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.2977336185181646e-05, 'epoch': 2.08}

 21%|██        | 152/730 [13:35<50:51,  5.28s/it]
 21%|██        | 153/730 [13:35<51:05,  5.31s/it]
                                                 
{'debug/num_tok_total': 2835.0, 'debug/num_tok_loss': 1757.0, 'debug/num_lat_total': 2835.0, 'debug/num_lat_loss': 1757.0, 'epoch': 2.1}

 21%|██        | 153/730 [13:36<51:05,  5.31s/it]
                                                 
{'train/ce_loss': 3.21875, 'train/diffusion_loss': 0.3793810307979584, 'epoch': 2.1}

 21%|██        | 153/730 [13:36<51:05,  5.31s/it]
                                                 
{'train/learning_rate_real': 2.2946982934277832e-05, 'epoch': 2.1}

 21%|██        | 153/730 [13:36<51:05,  5.31s/it]
                                                 
{'debug/num_tok_total': 2184.0, 'debug/num_tok_loss': 1761.0, 'debug/num_lat_total': 2184.0, 'debug/num_lat_loss': 1761.0, 'epoch': 2.1}

 21%|██        | 153/730 [13:37<51:05,  5.31s/it]
                                                 
{'train/ce_loss': 3.09375, 'train/diffusion_loss': 0.4782491624355316, 'epoch': 2.1}

 21%|██        | 153/730 [13:37<51:05,  5.31s/it]
                                                 
{'train/learning_rate_real': 2.2946982934277832e-05, 'epoch': 2.1}

 21%|██        | 153/730 [13:37<51:05,  5.31s/it]
                                                 
{'debug/num_tok_total': 2350.0, 'debug/num_tok_loss': 1646.0, 'debug/num_lat_total': 2350.0, 'debug/num_lat_loss': 1646.0, 'epoch': 2.1}

 21%|██        | 153/730 [13:39<51:05,  5.31s/it]
                                                 
{'train/ce_loss': 3.359375, 'train/diffusion_loss': 0.49244463443756104, 'epoch': 2.1}

 21%|██        | 153/730 [13:39<51:05,  5.31s/it]
                                                 
{'train/learning_rate_real': 2.2946982934277832e-05, 'epoch': 2.1}

 21%|██        | 153/730 [13:39<51:05,  5.31s/it]
                                                 
{'debug/num_tok_total': 2661.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2661.0, 'debug/num_lat_loss': 1802.0, 'epoch': 2.1}

 21%|██        | 153/730 [13:40<51:05,  5.31s/it]
                                                 
{'train/ce_loss': 3.109375, 'train/diffusion_loss': 0.41750162839889526, 'epoch': 2.1}

 21%|██        | 153/730 [13:40<51:05,  5.31s/it]
                                                 
{'train/learning_rate_real': 2.2946982934277832e-05, 'epoch': 2.1}

 21%|██        | 153/730 [13:40<51:05,  5.31s/it]
 21%|██        | 154/730 [13:40<50:53,  5.30s/it]
                                                 
{'debug/num_tok_total': 3111.0, 'debug/num_tok_loss': 1803.0, 'debug/num_lat_total': 3111.0, 'debug/num_lat_loss': 1803.0, 'epoch': 2.11}

 21%|██        | 154/730 [13:42<50:53,  5.30s/it]
                                                 
{'train/ce_loss': 3.171875, 'train/diffusion_loss': 0.32178986072540283, 'epoch': 2.11}

 21%|██        | 154/730 [13:42<50:53,  5.30s/it]
                                                 
{'train/learning_rate_real': 2.2916423988531437e-05, 'epoch': 2.11}

 21%|██        | 154/730 [13:42<50:53,  5.30s/it]
                                                 
{'debug/num_tok_total': 2198.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2198.0, 'debug/num_lat_loss': 1779.0, 'epoch': 2.11}

 21%|██        | 154/730 [13:43<50:53,  5.30s/it]
                                                 
{'train/ce_loss': 3.15625, 'train/diffusion_loss': 0.510726273059845, 'epoch': 2.11}

 21%|██        | 154/730 [13:43<50:53,  5.30s/it]
                                                 
{'train/learning_rate_real': 2.2916423988531437e-05, 'epoch': 2.11}

 21%|██        | 154/730 [13:43<50:53,  5.30s/it]
                                                 
{'debug/num_tok_total': 2662.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2662.0, 'debug/num_lat_loss': 1795.0, 'epoch': 2.11}

 21%|██        | 154/730 [13:44<50:53,  5.30s/it]
                                                 
{'train/ce_loss': 3.109375, 'train/diffusion_loss': 0.4098641872406006, 'epoch': 2.11}

 21%|██        | 154/730 [13:44<50:53,  5.30s/it]
                                                 
{'train/learning_rate_real': 2.2916423988531437e-05, 'epoch': 2.11}

 21%|██        | 154/730 [13:44<50:53,  5.30s/it]
                                                 
{'debug/num_tok_total': 2528.0, 'debug/num_tok_loss': 1669.0, 'debug/num_lat_total': 2528.0, 'debug/num_lat_loss': 1669.0, 'epoch': 2.11}

 21%|██        | 154/730 [13:45<50:53,  5.30s/it]
                                                 
{'train/ce_loss': 3.234375, 'train/diffusion_loss': 0.4132782518863678, 'epoch': 2.11}

 21%|██        | 154/730 [13:45<50:53,  5.30s/it]
                                                 
{'train/learning_rate_real': 2.2916423988531437e-05, 'epoch': 2.11}

 21%|██        | 154/730 [13:45<50:53,  5.30s/it]
 21%|██        | 155/730 [13:46<50:54,  5.31s/it]
                                                 
{'debug/num_tok_total': 2887.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2887.0, 'debug/num_lat_loss': 1799.0, 'epoch': 2.12}

 21%|██        | 155/730 [13:47<50:54,  5.31s/it]
                                                 
{'train/ce_loss': 3.578125, 'train/diffusion_loss': 0.3646351993083954, 'epoch': 2.12}

 21%|██        | 155/730 [13:47<50:54,  5.31s/it]
                                                 
{'train/learning_rate_real': 2.2885659949629828e-05, 'epoch': 2.12}

 21%|██        | 155/730 [13:47<50:54,  5.31s/it]
                                                 
{'debug/num_tok_total': 2654.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2654.0, 'debug/num_lat_loss': 1790.0, 'epoch': 2.12}

 21%|██        | 155/730 [13:48<50:54,  5.31s/it]
                                                 
{'train/ce_loss': 3.15625, 'train/diffusion_loss': 0.38885483145713806, 'epoch': 2.12}

 21%|██        | 155/730 [13:48<50:54,  5.31s/it]
                                                 
{'train/learning_rate_real': 2.2885659949629828e-05, 'epoch': 2.12}

 21%|██        | 155/730 [13:48<50:54,  5.31s/it]
                                                 
{'debug/num_tok_total': 2937.0, 'debug/num_tok_loss': 1831.0, 'debug/num_lat_total': 2937.0, 'debug/num_lat_loss': 1831.0, 'epoch': 2.12}

 21%|██        | 155/730 [13:49<50:54,  5.31s/it]
                                                 
{'train/ce_loss': 3.109375, 'train/diffusion_loss': 0.3445654809474945, 'epoch': 2.12}

 21%|██        | 155/730 [13:49<50:54,  5.31s/it]
                                                 
{'train/learning_rate_real': 2.2885659949629828e-05, 'epoch': 2.12}

 21%|██        | 155/730 [13:49<50:54,  5.31s/it]
                                                 
{'debug/num_tok_total': 1897.0, 'debug/num_tok_loss': 1690.0, 'debug/num_lat_total': 1897.0, 'debug/num_lat_loss': 1690.0, 'epoch': 2.12}

 21%|██        | 155/730 [13:50<50:54,  5.31s/it]
                                                 
{'train/ce_loss': 3.15625, 'train/diffusion_loss': 0.5193563103675842, 'epoch': 2.12}

 21%|██        | 155/730 [13:50<50:54,  5.31s/it]
                                                 
{'train/learning_rate_real': 2.2885659949629828e-05, 'epoch': 2.12}

 21%|██        | 155/730 [13:50<50:54,  5.31s/it]
 21%|██▏       | 156/730 [13:51<50:31,  5.28s/it]
                                                 
{'debug/num_tok_total': 3090.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 3090.0, 'debug/num_lat_loss': 1794.0, 'epoch': 2.14}

 21%|██▏       | 156/730 [13:52<50:31,  5.28s/it]
                                                 
{'train/ce_loss': 3.0625, 'train/diffusion_loss': 0.31100159883499146, 'epoch': 2.14}

 21%|██▏       | 156/730 [13:52<50:31,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.285469142329851e-05, 'epoch': 2.14}

 21%|██▏       | 156/730 [13:52<50:31,  5.28s/it]
                                                 
{'debug/num_tok_total': 3100.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 3100.0, 'debug/num_lat_loss': 1797.0, 'epoch': 2.14}

 21%|██▏       | 156/730 [13:54<50:31,  5.28s/it]
                                                 
{'train/ce_loss': 3.0, 'train/diffusion_loss': 0.3313130736351013, 'epoch': 2.14}

 21%|██▏       | 156/730 [13:54<50:31,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.285469142329851e-05, 'epoch': 2.14}

 21%|██▏       | 156/730 [13:54<50:31,  5.28s/it]
                                                 
{'debug/num_tok_total': 2852.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2852.0, 'debug/num_lat_loss': 1783.0, 'epoch': 2.14}

 21%|██▏       | 156/730 [13:55<50:31,  5.28s/it]
                                                 
{'train/ce_loss': 3.234375, 'train/diffusion_loss': 0.34941917657852173, 'epoch': 2.14}

 21%|██▏       | 156/730 [13:55<50:31,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.285469142329851e-05, 'epoch': 2.14}

 21%|██▏       | 156/730 [13:55<50:31,  5.28s/it]
                                                 
{'debug/num_tok_total': 2620.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2620.0, 'debug/num_lat_loss': 1766.0, 'epoch': 2.14}

 21%|██▏       | 156/730 [13:56<50:31,  5.28s/it]
                                                 
{'train/ce_loss': 3.1875, 'train/diffusion_loss': 0.3860790729522705, 'epoch': 2.14}

 21%|██▏       | 156/730 [13:56<50:31,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.285469142329851e-05, 'epoch': 2.14}

 21%|██▏       | 156/730 [13:56<50:31,  5.28s/it]
 22%|██▏       | 157/730 [13:57<52:40,  5.52s/it]
                                                 
{'debug/num_tok_total': 2786.0, 'debug/num_tok_loss': 1717.0, 'debug/num_lat_total': 2786.0, 'debug/num_lat_loss': 1717.0, 'epoch': 2.15}

 22%|██▏       | 157/730 [13:58<52:40,  5.52s/it]
                                                 
{'train/ce_loss': 3.171875, 'train/diffusion_loss': 0.34918954968452454, 'epoch': 2.15}

 22%|██▏       | 157/730 [13:58<52:40,  5.52s/it]
                                                 
{'train/learning_rate_real': 2.2823519019289245e-05, 'epoch': 2.15}

 22%|██▏       | 157/730 [13:58<52:40,  5.52s/it]
                                                 
{'debug/num_tok_total': 2235.0, 'debug/num_tok_loss': 1798.0, 'debug/num_lat_total': 2235.0, 'debug/num_lat_loss': 1798.0, 'epoch': 2.15}

 22%|██▏       | 157/730 [13:59<52:40,  5.52s/it]
                                                 
{'train/ce_loss': 3.140625, 'train/diffusion_loss': 0.48824694752693176, 'epoch': 2.15}

 22%|██▏       | 157/730 [13:59<52:40,  5.52s/it]
                                                 
{'train/learning_rate_real': 2.2823519019289245e-05, 'epoch': 2.15}

 22%|██▏       | 157/730 [13:59<52:40,  5.52s/it]
                                                 
{'debug/num_tok_total': 2260.0, 'debug/num_tok_loss': 1625.0, 'debug/num_lat_total': 2260.0, 'debug/num_lat_loss': 1625.0, 'epoch': 2.15}

 22%|██▏       | 157/730 [14:00<52:40,  5.52s/it]
                                                 
{'train/ce_loss': 3.359375, 'train/diffusion_loss': 0.4429050385951996, 'epoch': 2.15}

 22%|██▏       | 157/730 [14:00<52:40,  5.52s/it]
                                                 
{'train/learning_rate_real': 2.2823519019289245e-05, 'epoch': 2.15}

 22%|██▏       | 157/730 [14:00<52:40,  5.52s/it]
                                                 
{'debug/num_tok_total': 3258.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 3258.0, 'debug/num_lat_loss': 1769.0, 'epoch': 2.15}

 22%|██▏       | 157/730 [14:02<52:40,  5.52s/it]
                                                 
{'train/ce_loss': 3.140625, 'train/diffusion_loss': 0.2725350856781006, 'epoch': 2.15}

 22%|██▏       | 157/730 [14:02<52:40,  5.52s/it]
                                                 
{'train/learning_rate_real': 2.2823519019289245e-05, 'epoch': 2.15}

 22%|██▏       | 157/730 [14:02<52:40,  5.52s/it]
 22%|██▏       | 158/730 [14:02<51:49,  5.44s/it]
                                                 
{'debug/num_tok_total': 2649.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2649.0, 'debug/num_lat_loss': 1792.0, 'epoch': 2.16}

 22%|██▏       | 158/730 [14:03<51:49,  5.44s/it]
                                                 
{'train/ce_loss': 3.0625, 'train/diffusion_loss': 0.4037143290042877, 'epoch': 2.16}

 22%|██▏       | 158/730 [14:03<51:49,  5.44s/it]
                                                 
{'train/learning_rate_real': 2.2792143351368005e-05, 'epoch': 2.16}

 22%|██▏       | 158/730 [14:03<51:49,  5.44s/it]
                                                 
{'debug/num_tok_total': 2872.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2872.0, 'debug/num_lat_loss': 1789.0, 'epoch': 2.16}

 22%|██▏       | 158/730 [14:05<51:49,  5.44s/it]
                                                 
{'train/ce_loss': 3.234375, 'train/diffusion_loss': 0.343654602766037, 'epoch': 2.16}

 22%|██▏       | 158/730 [14:05<51:49,  5.44s/it]
                                                 
{'train/learning_rate_real': 2.2792143351368005e-05, 'epoch': 2.16}

 22%|██▏       | 158/730 [14:05<51:49,  5.44s/it]
                                                 
{'debug/num_tok_total': 2905.0, 'debug/num_tok_loss': 1694.0, 'debug/num_lat_total': 2905.0, 'debug/num_lat_loss': 1694.0, 'epoch': 2.16}

 22%|██▏       | 158/730 [14:06<51:49,  5.44s/it]
                                                 
{'train/ce_loss': 3.125, 'train/diffusion_loss': 0.2816028594970703, 'epoch': 2.16}

 22%|██▏       | 158/730 [14:06<51:49,  5.44s/it]
                                                 
{'train/learning_rate_real': 2.2792143351368005e-05, 'epoch': 2.16}

 22%|██▏       | 158/730 [14:06<51:49,  5.44s/it]
                                                 
{'debug/num_tok_total': 2201.0, 'debug/num_tok_loss': 1758.0, 'debug/num_lat_total': 2201.0, 'debug/num_lat_loss': 1758.0, 'epoch': 2.16}

 22%|██▏       | 158/730 [14:07<51:49,  5.44s/it]
                                                 
{'train/ce_loss': 3.21875, 'train/diffusion_loss': 0.525771975517273, 'epoch': 2.16}

 22%|██▏       | 158/730 [14:07<51:49,  5.44s/it]
                                                 
{'train/learning_rate_real': 2.2792143351368005e-05, 'epoch': 2.16}

 22%|██▏       | 158/730 [14:07<51:49,  5.44s/it]
 22%|██▏       | 159/730 [14:07<51:16,  5.39s/it]
                                                 
{'debug/num_tok_total': 2697.0, 'debug/num_tok_loss': 1818.0, 'debug/num_lat_total': 2697.0, 'debug/num_lat_loss': 1818.0, 'epoch': 2.18}

 22%|██▏       | 159/730 [14:09<51:16,  5.39s/it]
                                                 
{'train/ce_loss': 3.046875, 'train/diffusion_loss': 0.40741604566574097, 'epoch': 2.18}

 22%|██▏       | 159/730 [14:09<51:16,  5.39s/it]
                                                 
{'train/learning_rate_real': 2.276056503730293e-05, 'epoch': 2.18}

 22%|██▏       | 159/730 [14:09<51:16,  5.39s/it]
                                                 
{'debug/num_tok_total': 2861.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2861.0, 'debug/num_lat_loss': 1784.0, 'epoch': 2.18}

 22%|██▏       | 159/730 [14:10<51:16,  5.39s/it]
                                                 
{'train/ce_loss': 3.53125, 'train/diffusion_loss': 0.37161630392074585, 'epoch': 2.18}

 22%|██▏       | 159/730 [14:10<51:16,  5.39s/it]
                                                 
{'train/learning_rate_real': 2.276056503730293e-05, 'epoch': 2.18}

 22%|██▏       | 159/730 [14:10<51:16,  5.39s/it]
                                                 
{'debug/num_tok_total': 2767.0, 'debug/num_tok_loss': 1737.0, 'debug/num_lat_total': 2767.0, 'debug/num_lat_loss': 1737.0, 'epoch': 2.18}

 22%|██▏       | 159/730 [14:11<51:16,  5.39s/it]
                                                 
{'train/ce_loss': 3.15625, 'train/diffusion_loss': 0.38080403208732605, 'epoch': 2.18}

 22%|██▏       | 159/730 [14:11<51:16,  5.39s/it]
                                                 
{'train/learning_rate_real': 2.276056503730293e-05, 'epoch': 2.18}

 22%|██▏       | 159/730 [14:11<51:16,  5.39s/it]
                                                 
{'debug/num_tok_total': 2836.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 2836.0, 'debug/num_lat_loss': 1768.0, 'epoch': 2.18}

 22%|██▏       | 159/730 [14:12<51:16,  5.39s/it]
                                                 
{'train/ce_loss': 2.984375, 'train/diffusion_loss': 0.3861440420150757, 'epoch': 2.18}

 22%|██▏       | 159/730 [14:12<51:16,  5.39s/it]
                                                 
{'train/learning_rate_real': 2.276056503730293e-05, 'epoch': 2.18}

 22%|██▏       | 159/730 [14:12<51:16,  5.39s/it]03/16/2026 07:01:45 - INFO - __main__ - LoRA debug step 160: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 22%|██▏       | 160/730 [14:13<51:19,  5.40s/it]
                                                 
{'loss': 2.7086, 'grad_norm': 1.617277979850769, 'learning_rate': 2.276056503730293e-05, 'epoch': 2.19}

 22%|██▏       | 160/730 [14:13<51:19,  5.40s/it]
                                                 
{'debug/num_tok_total': 2700.0, 'debug/num_tok_loss': 1812.0, 'debug/num_lat_total': 2700.0, 'debug/num_lat_loss': 1812.0, 'epoch': 2.19}

 22%|██▏       | 160/730 [14:14<51:19,  5.40s/it]
                                                 
{'train/ce_loss': 3.3125, 'train/diffusion_loss': 0.3891316056251526, 'epoch': 2.19}

 22%|██▏       | 160/730 [14:14<51:19,  5.40s/it]
                                                 
{'train/learning_rate_real': 2.2728784698852114e-05, 'epoch': 2.19}

 22%|██▏       | 160/730 [14:14<51:19,  5.40s/it]
                                                 
{'debug/num_tok_total': 1777.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 1777.0, 'debug/num_lat_loss': 1777.0, 'epoch': 2.19}

 22%|██▏       | 160/730 [14:15<51:19,  5.40s/it]
                                                 
{'train/ce_loss': 3.140625, 'train/diffusion_loss': 0.5425366759300232, 'epoch': 2.19}

 22%|██▏       | 160/730 [14:15<51:19,  5.40s/it]
                                                 
{'train/learning_rate_real': 2.2728784698852114e-05, 'epoch': 2.19}

 22%|██▏       | 160/730 [14:15<51:19,  5.40s/it]
                                                 
{'debug/num_tok_total': 2411.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2411.0, 'debug/num_lat_loss': 1763.0, 'epoch': 2.19}

 22%|██▏       | 160/730 [14:16<51:19,  5.40s/it]
                                                 
{'train/ce_loss': 3.125, 'train/diffusion_loss': 0.44423243403434753, 'epoch': 2.19}

 22%|██▏       | 160/730 [14:16<51:19,  5.40s/it]
                                                 
{'train/learning_rate_real': 2.2728784698852114e-05, 'epoch': 2.19}

 22%|██▏       | 160/730 [14:16<51:19,  5.40s/it]
                                                 
{'debug/num_tok_total': 2207.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2207.0, 'debug/num_lat_loss': 1784.0, 'epoch': 2.19}

 22%|██▏       | 160/730 [14:17<51:19,  5.40s/it]
                                                 
{'train/ce_loss': 3.09375, 'train/diffusion_loss': 0.4857548177242279, 'epoch': 2.19}

 22%|██▏       | 160/730 [14:17<51:19,  5.40s/it]
                                                 
{'train/learning_rate_real': 2.2728784698852114e-05, 'epoch': 2.19}

 22%|██▏       | 160/730 [14:17<51:19,  5.40s/it]
 22%|██▏       | 161/730 [14:18<49:07,  5.18s/it]
                                                 
{'debug/num_tok_total': 2631.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2631.0, 'debug/num_lat_loss': 1784.0, 'epoch': 2.21}

 22%|██▏       | 161/730 [14:19<49:07,  5.18s/it]
                                                 
{'train/ce_loss': 3.140625, 'train/diffusion_loss': 0.4045424461364746, 'epoch': 2.21}

 22%|██▏       | 161/730 [14:19<49:07,  5.18s/it]
                                                 
{'train/learning_rate_real': 2.269680296175141e-05, 'epoch': 2.21}

 22%|██▏       | 161/730 [14:19<49:07,  5.18s/it]
                                                 
{'debug/num_tok_total': 3137.0, 'debug/num_tok_loss': 1702.0, 'debug/num_lat_total': 3137.0, 'debug/num_lat_loss': 1702.0, 'epoch': 2.21}

 22%|██▏       | 161/730 [14:20<49:07,  5.18s/it]
                                                 
{'train/ce_loss': 2.9375, 'train/diffusion_loss': 0.26028120517730713, 'epoch': 2.21}

 22%|██▏       | 161/730 [14:20<49:07,  5.18s/it]
                                                 
{'train/learning_rate_real': 2.269680296175141e-05, 'epoch': 2.21}

 22%|██▏       | 161/730 [14:20<49:07,  5.18s/it]
                                                 
{'debug/num_tok_total': 2635.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2635.0, 'debug/num_lat_loss': 1778.0, 'epoch': 2.21}

 22%|██▏       | 161/730 [14:21<49:07,  5.18s/it]
                                                 
{'train/ce_loss': 3.171875, 'train/diffusion_loss': 0.4297439455986023, 'epoch': 2.21}

 22%|██▏       | 161/730 [14:21<49:07,  5.18s/it]
                                                 
{'train/learning_rate_real': 2.269680296175141e-05, 'epoch': 2.21}

 22%|██▏       | 161/730 [14:21<49:07,  5.18s/it]
                                                 
{'debug/num_tok_total': 2632.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2632.0, 'debug/num_lat_loss': 1776.0, 'epoch': 2.21}

 22%|██▏       | 161/730 [14:22<49:07,  5.18s/it]
                                                 
{'train/ce_loss': 3.09375, 'train/diffusion_loss': 0.4013395309448242, 'epoch': 2.21}

 22%|██▏       | 161/730 [14:22<49:07,  5.18s/it]
                                                 
{'train/learning_rate_real': 2.269680296175141e-05, 'epoch': 2.21}

 22%|██▏       | 161/730 [14:22<49:07,  5.18s/it]
 22%|██▏       | 162/730 [14:23<49:27,  5.22s/it]
                                                 
{'debug/num_tok_total': 2443.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2443.0, 'debug/num_lat_loss': 1796.0, 'epoch': 2.22}

 22%|██▏       | 162/730 [14:24<49:27,  5.22s/it]
                                                 
{'train/ce_loss': 3.40625, 'train/diffusion_loss': 0.45490705966949463, 'epoch': 2.22}

 22%|██▏       | 162/730 [14:24<49:27,  5.22s/it]
                                                 
{'train/learning_rate_real': 2.2664620455702084e-05, 'epoch': 2.22}

 22%|██▏       | 162/730 [14:24<49:27,  5.22s/it]
                                                 
{'debug/num_tok_total': 2656.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2656.0, 'debug/num_lat_loss': 1786.0, 'epoch': 2.22}

 22%|██▏       | 162/730 [14:25<49:27,  5.22s/it]
                                                 
{'train/ce_loss': 3.109375, 'train/diffusion_loss': 0.420023649930954, 'epoch': 2.22}

 22%|██▏       | 162/730 [14:25<49:27,  5.22s/it]
                                                 
{'train/learning_rate_real': 2.2664620455702084e-05, 'epoch': 2.22}

 22%|██▏       | 162/730 [14:25<49:27,  5.22s/it]
                                                 
{'debug/num_tok_total': 2893.0, 'debug/num_tok_loss': 1687.0, 'debug/num_lat_total': 2893.0, 'debug/num_lat_loss': 1687.0, 'epoch': 2.22}

 22%|██▏       | 162/730 [14:27<49:27,  5.22s/it]
                                                 
{'train/ce_loss': 3.109375, 'train/diffusion_loss': 0.3170675039291382, 'epoch': 2.22}

 22%|██▏       | 162/730 [14:27<49:27,  5.22s/it]
                                                 
{'train/learning_rate_real': 2.2664620455702084e-05, 'epoch': 2.22}

 22%|██▏       | 162/730 [14:27<49:27,  5.22s/it]
                                                 
{'debug/num_tok_total': 2852.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2852.0, 'debug/num_lat_loss': 1778.0, 'epoch': 2.22}

 22%|██▏       | 162/730 [14:28<49:27,  5.22s/it]
                                                 
{'train/ce_loss': 3.078125, 'train/diffusion_loss': 0.3600369989871979, 'epoch': 2.22}

 22%|██▏       | 162/730 [14:28<49:27,  5.22s/it]
                                                 
{'train/learning_rate_real': 2.2664620455702084e-05, 'epoch': 2.22}

 22%|██▏       | 162/730 [14:28<49:27,  5.22s/it]
 22%|██▏       | 163/730 [14:28<50:17,  5.32s/it]
                                                 
{'debug/num_tok_total': 2555.0, 'debug/num_tok_loss': 1700.0, 'debug/num_lat_total': 2555.0, 'debug/num_lat_loss': 1700.0, 'epoch': 2.23}

 22%|██▏       | 163/730 [14:30<50:17,  5.32s/it]
                                                 
{'train/ce_loss': 3.125, 'train/diffusion_loss': 0.3976871967315674, 'epoch': 2.23}

 22%|██▏       | 163/730 [14:30<50:17,  5.32s/it]
                                                 
{'train/learning_rate_real': 2.2632237814358425e-05, 'epoch': 2.23}

 22%|██▏       | 163/730 [14:30<50:17,  5.32s/it]
                                                 
{'debug/num_tok_total': 2824.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2824.0, 'debug/num_lat_loss': 1770.0, 'epoch': 2.23}

 22%|██▏       | 163/730 [14:31<50:17,  5.32s/it]
                                                 
{'train/ce_loss': 2.953125, 'train/diffusion_loss': 0.36776605248451233, 'epoch': 2.23}

 22%|██▏       | 163/730 [14:31<50:17,  5.32s/it]
                                                 
{'train/learning_rate_real': 2.2632237814358425e-05, 'epoch': 2.23}

 22%|██▏       | 163/730 [14:31<50:17,  5.32s/it]
                                                 
{'debug/num_tok_total': 2770.0, 'debug/num_tok_loss': 1729.0, 'debug/num_lat_total': 2770.0, 'debug/num_lat_loss': 1729.0, 'epoch': 2.23}

 22%|██▏       | 163/730 [14:32<50:17,  5.32s/it]
                                                 
{'train/ce_loss': 3.109375, 'train/diffusion_loss': 0.3672686219215393, 'epoch': 2.23}

 22%|██▏       | 163/730 [14:32<50:17,  5.32s/it]
                                                 
{'train/learning_rate_real': 2.2632237814358425e-05, 'epoch': 2.23}

 22%|██▏       | 163/730 [14:32<50:17,  5.32s/it]
                                                 
{'debug/num_tok_total': 2433.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2433.0, 'debug/num_lat_loss': 1792.0, 'epoch': 2.23}

 22%|██▏       | 163/730 [14:33<50:17,  5.32s/it]
                                                 
{'train/ce_loss': 3.390625, 'train/diffusion_loss': 0.4243660271167755, 'epoch': 2.23}

 22%|██▏       | 163/730 [14:33<50:17,  5.32s/it]
                                                 
{'train/learning_rate_real': 2.2632237814358425e-05, 'epoch': 2.23}

 22%|██▏       | 163/730 [14:33<50:17,  5.32s/it]
 22%|██▏       | 164/730 [14:34<50:00,  5.30s/it]
                                                 
{'debug/num_tok_total': 2607.0, 'debug/num_tok_loss': 1756.0, 'debug/num_lat_total': 2607.0, 'debug/num_lat_loss': 1756.0, 'epoch': 2.25}

 22%|██▏       | 164/730 [14:35<50:00,  5.30s/it]
                                                 
{'train/ce_loss': 2.953125, 'train/diffusion_loss': 0.4174022078514099, 'epoch': 2.25}

 22%|██▏       | 164/730 [14:35<50:00,  5.30s/it]
                                                 
{'train/learning_rate_real': 2.2599655675315256e-05, 'epoch': 2.25}

 22%|██▏       | 164/730 [14:35<50:00,  5.30s/it]
                                                 
{'debug/num_tok_total': 2235.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2235.0, 'debug/num_lat_loss': 1799.0, 'epoch': 2.25}

 22%|██▏       | 164/730 [14:36<50:00,  5.30s/it]
                                                 
{'train/ce_loss': 2.984375, 'train/diffusion_loss': 0.5091489553451538, 'epoch': 2.25}

 22%|██▏       | 164/730 [14:36<50:00,  5.30s/it]
                                                 
{'train/learning_rate_real': 2.2599655675315256e-05, 'epoch': 2.25}

 22%|██▏       | 164/730 [14:36<50:00,  5.30s/it]
                                                 
{'debug/num_tok_total': 2889.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2889.0, 'debug/num_lat_loss': 1794.0, 'epoch': 2.25}

 22%|██▏       | 164/730 [14:37<50:00,  5.30s/it]
                                                 
{'train/ce_loss': 2.984375, 'train/diffusion_loss': 0.36695384979248047, 'epoch': 2.25}

 22%|██▏       | 164/730 [14:37<50:00,  5.30s/it]
                                                 
{'train/learning_rate_real': 2.2599655675315256e-05, 'epoch': 2.25}

 22%|██▏       | 164/730 [14:37<50:00,  5.30s/it]
                                                 
{'debug/num_tok_total': 2242.0, 'debug/num_tok_loss': 1803.0, 'debug/num_lat_total': 2242.0, 'debug/num_lat_loss': 1803.0, 'epoch': 2.25}

 22%|██▏       | 164/730 [14:38<50:00,  5.30s/it]
                                                 
{'train/ce_loss': 3.078125, 'train/diffusion_loss': 0.4847750663757324, 'epoch': 2.25}

 22%|██▏       | 164/730 [14:38<50:00,  5.30s/it]
                                                 
{'train/learning_rate_real': 2.2599655675315256e-05, 'epoch': 2.25}

 22%|██▏       | 164/730 [14:38<50:00,  5.30s/it]
 23%|██▎       | 165/730 [14:39<49:28,  5.25s/it]
                                                 
{'debug/num_tok_total': 2627.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2627.0, 'debug/num_lat_loss': 1780.0, 'epoch': 2.26}

 23%|██▎       | 165/730 [14:40<49:28,  5.25s/it]
                                                 
{'train/ce_loss': 2.96875, 'train/diffusion_loss': 0.3915446400642395, 'epoch': 2.26}

 23%|██▎       | 165/730 [14:40<49:28,  5.25s/it]
                                                 
{'train/learning_rate_real': 2.2566874680095403e-05, 'epoch': 2.26}

 23%|██▎       | 165/730 [14:40<49:28,  5.25s/it]
                                                 
{'debug/num_tok_total': 2390.0, 'debug/num_tok_loss': 1761.0, 'debug/num_lat_total': 2390.0, 'debug/num_lat_loss': 1761.0, 'epoch': 2.26}

 23%|██▎       | 165/730 [14:41<49:28,  5.25s/it]
                                                 
{'train/ce_loss': 2.859375, 'train/diffusion_loss': 0.45723363757133484, 'epoch': 2.26}

 23%|██▎       | 165/730 [14:41<49:28,  5.25s/it]
                                                 
{'train/learning_rate_real': 2.2566874680095403e-05, 'epoch': 2.26}

 23%|██▎       | 165/730 [14:41<49:28,  5.25s/it]
                                                 
{'debug/num_tok_total': 2692.0, 'debug/num_tok_loss': 1811.0, 'debug/num_lat_total': 2692.0, 'debug/num_lat_loss': 1811.0, 'epoch': 2.26}

 23%|██▎       | 165/730 [14:42<49:28,  5.25s/it]
                                                 
{'train/ce_loss': 3.015625, 'train/diffusion_loss': 0.4079940617084503, 'epoch': 2.26}

 23%|██▎       | 165/730 [14:42<49:28,  5.25s/it]
                                                 
{'train/learning_rate_real': 2.2566874680095403e-05, 'epoch': 2.26}

 23%|██▎       | 165/730 [14:42<49:28,  5.25s/it]
                                                 
{'debug/num_tok_total': 2602.0, 'debug/num_tok_loss': 1760.0, 'debug/num_lat_total': 2602.0, 'debug/num_lat_loss': 1760.0, 'epoch': 2.26}

 23%|██▎       | 165/730 [14:44<49:28,  5.25s/it]
                                                 
{'train/ce_loss': 2.953125, 'train/diffusion_loss': 0.4096970856189728, 'epoch': 2.26}

 23%|██▎       | 165/730 [14:44<49:28,  5.25s/it]
                                                 
{'train/learning_rate_real': 2.2566874680095403e-05, 'epoch': 2.26}

 23%|██▎       | 165/730 [14:44<49:28,  5.25s/it]
 23%|██▎       | 166/730 [14:44<49:24,  5.26s/it]
                                                 
{'debug/num_tok_total': 2875.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2875.0, 'debug/num_lat_loss': 1790.0, 'epoch': 2.27}

 23%|██▎       | 166/730 [14:45<49:24,  5.26s/it]
                                                 
{'train/ce_loss': 3.0625, 'train/diffusion_loss': 0.34252914786338806, 'epoch': 2.27}

 23%|██▎       | 166/730 [14:45<49:24,  5.26s/it]
                                                 
{'train/learning_rate_real': 2.2533895474137047e-05, 'epoch': 2.27}

 23%|██▎       | 166/730 [14:45<49:24,  5.26s/it]
                                                 
{'debug/num_tok_total': 2829.0, 'debug/num_tok_loss': 1759.0, 'debug/num_lat_total': 2829.0, 'debug/num_lat_loss': 1759.0, 'epoch': 2.27}

 23%|██▎       | 166/730 [14:47<49:24,  5.26s/it]
                                                 
{'train/ce_loss': 2.921875, 'train/diffusion_loss': 0.3555743098258972, 'epoch': 2.27}

 23%|██▎       | 166/730 [14:47<49:24,  5.26s/it]
                                                 
{'train/learning_rate_real': 2.2533895474137047e-05, 'epoch': 2.27}

 23%|██▎       | 166/730 [14:47<49:24,  5.26s/it]
                                                 
{'debug/num_tok_total': 2697.0, 'debug/num_tok_loss': 1811.0, 'debug/num_lat_total': 2697.0, 'debug/num_lat_loss': 1811.0, 'epoch': 2.27}

 23%|██▎       | 166/730 [14:48<49:24,  5.26s/it]
                                                 
{'train/ce_loss': 2.953125, 'train/diffusion_loss': 0.4044460952281952, 'epoch': 2.27}

 23%|██▎       | 166/730 [14:48<49:24,  5.26s/it]
                                                 
{'train/learning_rate_real': 2.2533895474137047e-05, 'epoch': 2.27}

 23%|██▎       | 166/730 [14:48<49:24,  5.26s/it]
                                                 
{'debug/num_tok_total': 2482.0, 'debug/num_tok_loss': 1812.0, 'debug/num_lat_total': 2482.0, 'debug/num_lat_loss': 1812.0, 'epoch': 2.27}

 23%|██▎       | 166/730 [14:49<49:24,  5.26s/it]
                                                 
{'train/ce_loss': 2.984375, 'train/diffusion_loss': 0.4411238431930542, 'epoch': 2.27}

 23%|██▎       | 166/730 [14:49<49:24,  5.26s/it]
                                                 
{'train/learning_rate_real': 2.2533895474137047e-05, 'epoch': 2.27}

 23%|██▎       | 166/730 [14:49<49:24,  5.26s/it]
 23%|██▎       | 167/730 [14:49<49:27,  5.27s/it]
                                                 
{'debug/num_tok_total': 2951.0, 'debug/num_tok_loss': 1717.0, 'debug/num_lat_total': 2951.0, 'debug/num_lat_loss': 1717.0, 'epoch': 2.29}

 23%|██▎       | 167/730 [14:51<49:27,  5.27s/it]
                                                 
{'train/ce_loss': 2.8125, 'train/diffusion_loss': 0.28139710426330566, 'epoch': 2.29}

 23%|██▎       | 167/730 [14:51<49:27,  5.27s/it]
                                                 
{'train/learning_rate_real': 2.2500718706781014e-05, 'epoch': 2.29}

 23%|██▎       | 167/730 [14:51<49:27,  5.27s/it]
                                                 
{'debug/num_tok_total': 2426.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2426.0, 'debug/num_lat_loss': 1784.0, 'epoch': 2.29}

 23%|██▎       | 167/730 [14:52<49:27,  5.27s/it]
                                                 
{'train/ce_loss': 2.859375, 'train/diffusion_loss': 0.44409722089767456, 'epoch': 2.29}

 23%|██▎       | 167/730 [14:52<49:27,  5.27s/it]
                                                 
{'train/learning_rate_real': 2.2500718706781014e-05, 'epoch': 2.29}

 23%|██▎       | 167/730 [14:52<49:27,  5.27s/it]
                                                 
{'debug/num_tok_total': 2416.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2416.0, 'debug/num_lat_loss': 1771.0, 'epoch': 2.29}

 23%|██▎       | 167/730 [14:53<49:27,  5.27s/it]
                                                 
{'train/ce_loss': 3.03125, 'train/diffusion_loss': 0.4660814702510834, 'epoch': 2.29}

 23%|██▎       | 167/730 [14:53<49:27,  5.27s/it]
                                                 
{'train/learning_rate_real': 2.2500718706781014e-05, 'epoch': 2.29}

 23%|██▎       | 167/730 [14:53<49:27,  5.27s/it]
                                                 
{'debug/num_tok_total': 2417.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2417.0, 'debug/num_lat_loss': 1771.0, 'epoch': 2.29}

 23%|██▎       | 167/730 [14:54<49:27,  5.27s/it]
                                                 
{'train/ce_loss': 2.875, 'train/diffusion_loss': 0.46190908551216125, 'epoch': 2.29}

 23%|██▎       | 167/730 [14:54<49:27,  5.27s/it]
                                                 
{'train/learning_rate_real': 2.2500718706781014e-05, 'epoch': 2.29}

 23%|██▎       | 167/730 [14:54<49:27,  5.27s/it]
 23%|██▎       | 168/730 [14:55<49:43,  5.31s/it]
                                                 
{'debug/num_tok_total': 2146.0, 'debug/num_tok_loss': 1653.0, 'debug/num_lat_total': 2146.0, 'debug/num_lat_loss': 1653.0, 'epoch': 2.3}

 23%|██▎       | 168/730 [14:56<49:43,  5.31s/it]
                                                 
{'train/ce_loss': 2.859375, 'train/diffusion_loss': 0.4429917335510254, 'epoch': 2.3}

 23%|██▎       | 168/730 [14:56<49:43,  5.31s/it]
                                                 
{'train/learning_rate_real': 2.2467345031258007e-05, 'epoch': 2.3}

 23%|██▎       | 168/730 [14:56<49:43,  5.31s/it]
                                                 
{'debug/num_tok_total': 2650.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2650.0, 'debug/num_lat_loss': 1793.0, 'epoch': 2.3}

 23%|██▎       | 168/730 [14:57<49:43,  5.31s/it]
                                                 
{'train/ce_loss': 3.046875, 'train/diffusion_loss': 0.4134480059146881, 'epoch': 2.3}

 23%|██▎       | 168/730 [14:57<49:43,  5.31s/it]
                                                 
{'train/learning_rate_real': 2.2467345031258007e-05, 'epoch': 2.3}

 23%|██▎       | 168/730 [14:57<49:43,  5.31s/it]
                                                 
{'debug/num_tok_total': 2248.0, 'debug/num_tok_loss': 1615.0, 'debug/num_lat_total': 2248.0, 'debug/num_lat_loss': 1615.0, 'epoch': 2.3}

 23%|██▎       | 168/730 [14:58<49:43,  5.31s/it]
                                                 
{'train/ce_loss': 3.015625, 'train/diffusion_loss': 0.4329485595226288, 'epoch': 2.3}

 23%|██▎       | 168/730 [14:58<49:43,  5.31s/it]
                                                 
{'train/learning_rate_real': 2.2467345031258007e-05, 'epoch': 2.3}

 23%|██▎       | 168/730 [14:58<49:43,  5.31s/it]
                                                 
{'debug/num_tok_total': 2625.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2625.0, 'debug/num_lat_loss': 1773.0, 'epoch': 2.3}

 23%|██▎       | 168/730 [14:59<49:43,  5.31s/it]
                                                 
{'train/ce_loss': 3.015625, 'train/diffusion_loss': 0.4059813618659973, 'epoch': 2.3}

 23%|██▎       | 168/730 [14:59<49:43,  5.31s/it]
                                                 
{'train/learning_rate_real': 2.2467345031258007e-05, 'epoch': 2.3}

 23%|██▎       | 168/730 [14:59<49:43,  5.31s/it]
 23%|██▎       | 169/730 [15:00<49:01,  5.24s/it]
                                                 
{'debug/num_tok_total': 3095.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 3095.0, 'debug/num_lat_loss': 1795.0, 'epoch': 2.32}

 23%|██▎       | 169/730 [15:01<49:01,  5.24s/it]
                                                 
{'train/ce_loss': 3.015625, 'train/diffusion_loss': 0.33460959792137146, 'epoch': 2.32}

 23%|██▎       | 169/730 [15:01<49:01,  5.24s/it]
                                                 
{'train/learning_rate_real': 2.243377510467572e-05, 'epoch': 2.32}

 23%|██▎       | 169/730 [15:01<49:01,  5.24s/it]
                                                 
{'debug/num_tok_total': 2592.0, 'debug/num_tok_loss': 1748.0, 'debug/num_lat_total': 2592.0, 'debug/num_lat_loss': 1748.0, 'epoch': 2.32}

 23%|██▎       | 169/730 [15:02<49:01,  5.24s/it]
                                                 
{'train/ce_loss': 2.875, 'train/diffusion_loss': 0.37922653555870056, 'epoch': 2.32}

 23%|██▎       | 169/730 [15:02<49:01,  5.24s/it]
                                                 
{'train/learning_rate_real': 2.243377510467572e-05, 'epoch': 2.32}

 23%|██▎       | 169/730 [15:02<49:01,  5.24s/it]
                                                 
{'debug/num_tok_total': 2911.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2911.0, 'debug/num_lat_loss': 1799.0, 'epoch': 2.32}

 23%|██▎       | 169/730 [15:04<49:01,  5.24s/it]
                                                 
{'train/ce_loss': 3.03125, 'train/diffusion_loss': 0.3680286109447479, 'epoch': 2.32}

 23%|██▎       | 169/730 [15:04<49:01,  5.24s/it]
                                                 
{'train/learning_rate_real': 2.243377510467572e-05, 'epoch': 2.32}

 23%|██▎       | 169/730 [15:04<49:01,  5.24s/it]
                                                 
{'debug/num_tok_total': 2198.0, 'debug/num_tok_loss': 1678.0, 'debug/num_lat_total': 2198.0, 'debug/num_lat_loss': 1678.0, 'epoch': 2.32}

 23%|██▎       | 169/730 [15:05<49:01,  5.24s/it]
                                                 
{'train/ce_loss': 3.25, 'train/diffusion_loss': 0.46388769149780273, 'epoch': 2.32}

 23%|██▎       | 169/730 [15:05<49:01,  5.24s/it]
                                                 
{'train/learning_rate_real': 2.243377510467572e-05, 'epoch': 2.32}

 23%|██▎       | 169/730 [15:05<49:01,  5.24s/it]03/16/2026 07:02:37 - INFO - __main__ - LoRA debug step 170: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 23%|██▎       | 170/730 [15:05<49:14,  5.28s/it]
                                                 
{'loss': 2.7758, 'grad_norm': 1.618893027305603, 'learning_rate': 2.243377510467572e-05, 'epoch': 2.33}

 23%|██▎       | 170/730 [15:05<49:14,  5.28s/it]
                                                 
{'debug/num_tok_total': 2429.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2429.0, 'debug/num_lat_loss': 1779.0, 'epoch': 2.33}

 23%|██▎       | 170/730 [15:06<49:14,  5.28s/it]
                                                 
{'train/ce_loss': 2.96875, 'train/diffusion_loss': 0.453939825296402, 'epoch': 2.33}

 23%|██▎       | 170/730 [15:06<49:14,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.2400009588005925e-05, 'epoch': 2.33}

 23%|██▎       | 170/730 [15:06<49:14,  5.28s/it]
                                                 
{'debug/num_tok_total': 3149.0, 'debug/num_tok_loss': 1824.0, 'debug/num_lat_total': 3149.0, 'debug/num_lat_loss': 1824.0, 'epoch': 2.33}

 23%|██▎       | 170/730 [15:08<49:14,  5.28s/it]
                                                 
{'train/ce_loss': 3.0, 'train/diffusion_loss': 0.3223628103733063, 'epoch': 2.33}

 23%|██▎       | 170/730 [15:08<49:14,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.2400009588005925e-05, 'epoch': 2.33}

 23%|██▎       | 170/730 [15:08<49:14,  5.28s/it]
                                                 
{'debug/num_tok_total': 3055.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 3055.0, 'debug/num_lat_loss': 1775.0, 'epoch': 2.33}

 23%|██▎       | 170/730 [15:09<49:14,  5.28s/it]
                                                 
{'train/ce_loss': 2.984375, 'train/diffusion_loss': 0.325356125831604, 'epoch': 2.33}

 23%|██▎       | 170/730 [15:09<49:14,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.2400009588005925e-05, 'epoch': 2.33}

 23%|██▎       | 170/730 [15:09<49:14,  5.28s/it]
                                                 
{'debug/num_tok_total': 2391.0, 'debug/num_tok_loss': 1758.0, 'debug/num_lat_total': 2391.0, 'debug/num_lat_loss': 1758.0, 'epoch': 2.33}

 23%|██▎       | 170/730 [15:10<49:14,  5.28s/it]
                                                 
{'train/ce_loss': 3.03125, 'train/diffusion_loss': 0.4393155872821808, 'epoch': 2.33}

 23%|██▎       | 170/730 [15:10<49:14,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.2400009588005925e-05, 'epoch': 2.33}

 23%|██▎       | 170/730 [15:10<49:14,  5.28s/it]
 23%|██▎       | 171/730 [15:11<49:35,  5.32s/it]
                                                 
{'debug/num_tok_total': 2237.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2237.0, 'debug/num_lat_loss': 1790.0, 'epoch': 2.34}

 23%|██▎       | 171/730 [15:12<49:35,  5.32s/it]
                                                 
{'train/ce_loss': 2.953125, 'train/diffusion_loss': 0.4934636950492859, 'epoch': 2.34}

 23%|██▎       | 171/730 [15:12<49:35,  5.32s/it]
                                                 
{'train/learning_rate_real': 2.236604914607143e-05, 'epoch': 2.34}

 23%|██▎       | 171/730 [15:12<49:35,  5.32s/it]
                                                 
{'debug/num_tok_total': 2625.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2625.0, 'debug/num_lat_loss': 1766.0, 'epoch': 2.34}

 23%|██▎       | 171/730 [15:13<49:35,  5.32s/it]
                                                 
{'train/ce_loss': 2.90625, 'train/diffusion_loss': 0.41351816058158875, 'epoch': 2.34}

 23%|██▎       | 171/730 [15:13<49:35,  5.32s/it]
                                                 
{'train/learning_rate_real': 2.236604914607143e-05, 'epoch': 2.34}

 23%|██▎       | 171/730 [15:13<49:35,  5.32s/it]
                                                 
{'debug/num_tok_total': 2635.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2635.0, 'debug/num_lat_loss': 1787.0, 'epoch': 2.34}

 23%|██▎       | 171/730 [15:14<49:35,  5.32s/it]
                                                 
{'train/ce_loss': 2.921875, 'train/diffusion_loss': 0.3951073884963989, 'epoch': 2.34}

 23%|██▎       | 171/730 [15:14<49:35,  5.32s/it]
                                                 
{'train/learning_rate_real': 2.236604914607143e-05, 'epoch': 2.34}

 23%|██▎       | 171/730 [15:14<49:35,  5.32s/it]
                                                 
{'debug/num_tok_total': 3076.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 3076.0, 'debug/num_lat_loss': 1783.0, 'epoch': 2.34}

 23%|██▎       | 171/730 [15:16<49:35,  5.32s/it]
                                                 
{'train/ce_loss': 2.859375, 'train/diffusion_loss': 0.30464741587638855, 'epoch': 2.34}

 23%|██▎       | 171/730 [15:16<49:35,  5.32s/it]
                                                 
{'train/learning_rate_real': 2.236604914607143e-05, 'epoch': 2.34}

 23%|██▎       | 171/730 [15:16<49:35,  5.32s/it]
 24%|██▎       | 172/730 [15:16<49:57,  5.37s/it]
                                                 
{'debug/num_tok_total': 3094.0, 'debug/num_tok_loss': 1800.0, 'debug/num_lat_total': 3094.0, 'debug/num_lat_loss': 1800.0, 'epoch': 2.36}

 24%|██▎       | 172/730 [15:17<49:57,  5.37s/it]
                                                 
{'train/ce_loss': 3.234375, 'train/diffusion_loss': 0.31877073645591736, 'epoch': 2.36}

 24%|██▎       | 172/730 [15:17<49:57,  5.37s/it]
                                                 
{'train/learning_rate_real': 2.2331894447533035e-05, 'epoch': 2.36}

 24%|██▎       | 172/730 [15:17<49:57,  5.37s/it]
                                                 
{'debug/num_tok_total': 2888.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2888.0, 'debug/num_lat_loss': 1792.0, 'epoch': 2.36}

 24%|██▎       | 172/730 [15:19<49:57,  5.37s/it]
                                                 
{'train/ce_loss': 2.875, 'train/diffusion_loss': 0.35470494627952576, 'epoch': 2.36}

 24%|██▎       | 172/730 [15:19<49:57,  5.37s/it]
                                                 
{'train/learning_rate_real': 2.2331894447533035e-05, 'epoch': 2.36}

 24%|██▎       | 172/730 [15:19<49:57,  5.37s/it]
                                                 
{'debug/num_tok_total': 2699.0, 'debug/num_tok_loss': 1816.0, 'debug/num_lat_total': 2699.0, 'debug/num_lat_loss': 1816.0, 'epoch': 2.36}

 24%|██▎       | 172/730 [15:20<49:57,  5.37s/it]
                                                 
{'train/ce_loss': 2.859375, 'train/diffusion_loss': 0.4036940038204193, 'epoch': 2.36}

 24%|██▎       | 172/730 [15:20<49:57,  5.37s/it]
                                                 
{'train/learning_rate_real': 2.2331894447533035e-05, 'epoch': 2.36}

 24%|██▎       | 172/730 [15:20<49:57,  5.37s/it]
                                                 
{'debug/num_tok_total': 2632.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2632.0, 'debug/num_lat_loss': 1781.0, 'epoch': 2.36}

 24%|██▎       | 172/730 [15:21<49:57,  5.37s/it]
                                                 
{'train/ce_loss': 2.96875, 'train/diffusion_loss': 0.4525442123413086, 'epoch': 2.36}

 24%|██▎       | 172/730 [15:21<49:57,  5.37s/it]
                                                 
{'train/learning_rate_real': 2.2331894447533035e-05, 'epoch': 2.36}

 24%|██▎       | 172/730 [15:21<49:57,  5.37s/it]
 24%|██▎       | 173/730 [15:22<50:31,  5.44s/it]
                                                 
{'debug/num_tok_total': 2609.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2609.0, 'debug/num_lat_loss': 1776.0, 'epoch': 2.37}

 24%|██▎       | 173/730 [15:23<50:31,  5.44s/it]
                                                 
{'train/ce_loss': 2.890625, 'train/diffusion_loss': 0.40328508615493774, 'epoch': 2.37}

 24%|██▎       | 173/730 [15:23<50:31,  5.44s/it]
                                                 
{'train/learning_rate_real': 2.2297546164876292e-05, 'epoch': 2.37}

 24%|██▎       | 173/730 [15:23<50:31,  5.44s/it]
                                                 
{'debug/num_tok_total': 2631.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2631.0, 'debug/num_lat_loss': 1778.0, 'epoch': 2.37}

 24%|██▎       | 173/730 [15:24<50:31,  5.44s/it]
                                                 
{'train/ce_loss': 2.890625, 'train/diffusion_loss': 0.3900800347328186, 'epoch': 2.37}

 24%|██▎       | 173/730 [15:24<50:31,  5.44s/it]
                                                 
{'train/learning_rate_real': 2.2297546164876292e-05, 'epoch': 2.37}

 24%|██▎       | 173/730 [15:24<50:31,  5.44s/it]
                                                 
{'debug/num_tok_total': 2653.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2653.0, 'debug/num_lat_loss': 1791.0, 'epoch': 2.37}

 24%|██▎       | 173/730 [15:25<50:31,  5.44s/it]
                                                 
{'train/ce_loss': 2.9375, 'train/diffusion_loss': 0.39137205481529236, 'epoch': 2.37}

 24%|██▎       | 173/730 [15:25<50:31,  5.44s/it]
                                                 
{'train/learning_rate_real': 2.2297546164876292e-05, 'epoch': 2.37}

 24%|██▎       | 173/730 [15:25<50:31,  5.44s/it]
                                                 
{'debug/num_tok_total': 2645.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2645.0, 'debug/num_lat_loss': 1784.0, 'epoch': 2.37}

 24%|██▎       | 173/730 [15:27<50:31,  5.44s/it]
                                                 
{'train/ce_loss': 2.953125, 'train/diffusion_loss': 0.4043900966644287, 'epoch': 2.37}

 24%|██▎       | 173/730 [15:27<50:31,  5.44s/it]
                                                 
{'train/learning_rate_real': 2.2297546164876292e-05, 'epoch': 2.37}

 24%|██▎       | 173/730 [15:27<50:31,  5.44s/it]
 24%|██▍       | 174/730 [15:27<50:07,  5.41s/it]
                                                 
{'debug/num_tok_total': 2389.0, 'debug/num_tok_loss': 1756.0, 'debug/num_lat_total': 2389.0, 'debug/num_lat_loss': 1756.0, 'epoch': 2.38}

 24%|██▍       | 174/730 [15:28<50:07,  5.41s/it]
                                                 
{'train/ce_loss': 2.921875, 'train/diffusion_loss': 0.45027777552604675, 'epoch': 2.38}

 24%|██▍       | 174/730 [15:28<50:07,  5.41s/it]
                                                 
{'train/learning_rate_real': 2.2263004974398352e-05, 'epoch': 2.38}

 24%|██▍       | 174/730 [15:28<50:07,  5.41s/it]
                                                 
{'debug/num_tok_total': 2453.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2453.0, 'debug/num_lat_loss': 1792.0, 'epoch': 2.38}

 24%|██▍       | 174/730 [15:29<50:07,  5.41s/it]
                                                 
{'train/ce_loss': 2.828125, 'train/diffusion_loss': 0.45082005858421326, 'epoch': 2.38}

 24%|██▍       | 174/730 [15:29<50:07,  5.41s/it]
                                                 
{'train/learning_rate_real': 2.2263004974398352e-05, 'epoch': 2.38}

 24%|██▍       | 174/730 [15:29<50:07,  5.41s/it]
                                                 
{'debug/num_tok_total': 2808.0, 'debug/num_tok_loss': 1755.0, 'debug/num_lat_total': 2808.0, 'debug/num_lat_loss': 1755.0, 'epoch': 2.38}

 24%|██▍       | 174/730 [15:31<50:07,  5.41s/it]
                                                 
{'train/ce_loss': 2.875, 'train/diffusion_loss': 0.36461949348449707, 'epoch': 2.38}

 24%|██▍       | 174/730 [15:31<50:07,  5.41s/it]
                                                 
{'train/learning_rate_real': 2.2263004974398352e-05, 'epoch': 2.38}

 24%|██▍       | 174/730 [15:31<50:07,  5.41s/it]
                                                 
{'debug/num_tok_total': 3082.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 3082.0, 'debug/num_lat_loss': 1783.0, 'epoch': 2.38}

 24%|██▍       | 174/730 [15:32<50:07,  5.41s/it]
                                                 
{'train/ce_loss': 2.84375, 'train/diffusion_loss': 0.32322970032691956, 'epoch': 2.38}

 24%|██▍       | 174/730 [15:32<50:07,  5.41s/it]
                                                 
{'train/learning_rate_real': 2.2263004974398352e-05, 'epoch': 2.38}

 24%|██▍       | 174/730 [15:32<50:07,  5.41s/it]
 24%|██▍       | 175/730 [15:32<49:59,  5.40s/it]
                                                 
{'debug/num_tok_total': 2219.0, 'debug/num_tok_loss': 1579.0, 'debug/num_lat_total': 2219.0, 'debug/num_lat_loss': 1579.0, 'epoch': 2.4}

 24%|██▍       | 175/730 [15:34<49:59,  5.40s/it]
                                                 
{'train/ce_loss': 2.921875, 'train/diffusion_loss': 0.44200700521469116, 'epoch': 2.4}

 24%|██▍       | 175/730 [15:34<49:59,  5.40s/it]
                                                 
{'train/learning_rate_real': 2.2228271556194575e-05, 'epoch': 2.4}

 24%|██▍       | 175/730 [15:34<49:59,  5.40s/it]
                                                 
{'debug/num_tok_total': 2431.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2431.0, 'debug/num_lat_loss': 1780.0, 'epoch': 2.4}

 24%|██▍       | 175/730 [15:35<49:59,  5.40s/it]
                                                 
{'train/ce_loss': 3.015625, 'train/diffusion_loss': 0.4691259562969208, 'epoch': 2.4}

 24%|██▍       | 175/730 [15:35<49:59,  5.40s/it]
                                                 
{'train/learning_rate_real': 2.2228271556194575e-05, 'epoch': 2.4}

 24%|██▍       | 175/730 [15:35<49:59,  5.40s/it]
                                                 
{'debug/num_tok_total': 2876.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2876.0, 'debug/num_lat_loss': 1787.0, 'epoch': 2.4}

 24%|██▍       | 175/730 [15:36<49:59,  5.40s/it]
                                                 
{'train/ce_loss': 3.375, 'train/diffusion_loss': 0.34737053513526917, 'epoch': 2.4}

 24%|██▍       | 175/730 [15:36<49:59,  5.40s/it]
                                                 
{'train/learning_rate_real': 2.2228271556194575e-05, 'epoch': 2.4}

 24%|██▍       | 175/730 [15:36<49:59,  5.40s/it]
                                                 
{'debug/num_tok_total': 2589.0, 'debug/num_tok_loss': 1725.0, 'debug/num_lat_total': 2589.0, 'debug/num_lat_loss': 1725.0, 'epoch': 2.4}

 24%|██▍       | 175/730 [15:37<49:59,  5.40s/it]
                                                 
{'train/ce_loss': 2.84375, 'train/diffusion_loss': 0.43164265155792236, 'epoch': 2.4}

 24%|██▍       | 175/730 [15:37<49:59,  5.40s/it]
                                                 
{'train/learning_rate_real': 2.2228271556194575e-05, 'epoch': 2.4}

 24%|██▍       | 175/730 [15:37<49:59,  5.40s/it]
 24%|██▍       | 176/730 [15:38<49:26,  5.36s/it]
                                                 
{'debug/num_tok_total': 2614.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2614.0, 'debug/num_lat_loss': 1766.0, 'epoch': 2.41}

 24%|██▍       | 176/730 [15:39<49:26,  5.36s/it]
                                                 
{'train/ce_loss': 3.0, 'train/diffusion_loss': 0.429606169462204, 'epoch': 2.41}

 24%|██▍       | 176/730 [15:39<49:26,  5.36s/it]
                                                 
{'train/learning_rate_real': 2.2193346594145188e-05, 'epoch': 2.41}

 24%|██▍       | 176/730 [15:39<49:26,  5.36s/it]
                                                 
{'debug/num_tok_total': 2658.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 2658.0, 'debug/num_lat_loss': 1797.0, 'epoch': 2.41}

 24%|██▍       | 176/730 [15:40<49:26,  5.36s/it]
                                                 
{'train/ce_loss': 2.84375, 'train/diffusion_loss': 0.3894621431827545, 'epoch': 2.41}

 24%|██▍       | 176/730 [15:40<49:26,  5.36s/it]
                                                 
{'train/learning_rate_real': 2.2193346594145188e-05, 'epoch': 2.41}

 24%|██▍       | 176/730 [15:40<49:26,  5.36s/it]
                                                 
{'debug/num_tok_total': 2654.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2654.0, 'debug/num_lat_loss': 1777.0, 'epoch': 2.41}

 24%|██▍       | 176/730 [15:41<49:26,  5.36s/it]
                                                 
{'train/ce_loss': 2.859375, 'train/diffusion_loss': 0.4027283191680908, 'epoch': 2.41}

 24%|██▍       | 176/730 [15:41<49:26,  5.36s/it]
                                                 
{'train/learning_rate_real': 2.2193346594145188e-05, 'epoch': 2.41}

 24%|██▍       | 176/730 [15:41<49:26,  5.36s/it]
                                                 
{'debug/num_tok_total': 2690.0, 'debug/num_tok_loss': 1808.0, 'debug/num_lat_total': 2690.0, 'debug/num_lat_loss': 1808.0, 'epoch': 2.41}

 24%|██▍       | 176/730 [15:42<49:26,  5.36s/it]
                                                 
{'train/ce_loss': 2.6875, 'train/diffusion_loss': 0.38133347034454346, 'epoch': 2.41}

 24%|██▍       | 176/730 [15:42<49:26,  5.36s/it]
                                                 
{'train/learning_rate_real': 2.2193346594145188e-05, 'epoch': 2.41}

 24%|██▍       | 176/730 [15:42<49:26,  5.36s/it]
 24%|██▍       | 177/730 [15:43<48:56,  5.31s/it]
                                                 
{'debug/num_tok_total': 2837.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 2837.0, 'debug/num_lat_loss': 1767.0, 'epoch': 2.42}

 24%|██▍       | 177/730 [15:44<48:56,  5.31s/it]
                                                 
{'train/ce_loss': 2.859375, 'train/diffusion_loss': 0.3693162500858307, 'epoch': 2.42}

 24%|██▍       | 177/730 [15:44<48:56,  5.31s/it]
                                                 
{'train/learning_rate_real': 2.215823077590179e-05, 'epoch': 2.42}

 24%|██▍       | 177/730 [15:44<48:56,  5.31s/it]
                                                 
{'debug/num_tok_total': 2252.0, 'debug/num_tok_loss': 1619.0, 'debug/num_lat_total': 2252.0, 'debug/num_lat_loss': 1619.0, 'epoch': 2.42}

 24%|██▍       | 177/730 [15:45<48:56,  5.31s/it]
                                                 
{'train/ce_loss': 2.796875, 'train/diffusion_loss': 0.4304879307746887, 'epoch': 2.42}

 24%|██▍       | 177/730 [15:45<48:56,  5.31s/it]
                                                 
{'train/learning_rate_real': 2.215823077590179e-05, 'epoch': 2.42}

 24%|██▍       | 177/730 [15:45<48:56,  5.31s/it]
                                                 
{'debug/num_tok_total': 2425.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2425.0, 'debug/num_lat_loss': 1780.0, 'epoch': 2.42}

 24%|██▍       | 177/730 [15:46<48:56,  5.31s/it]
                                                 
{'train/ce_loss': 2.890625, 'train/diffusion_loss': 0.4411391615867615, 'epoch': 2.42}

 24%|██▍       | 177/730 [15:46<48:56,  5.31s/it]
                                                 
{'train/learning_rate_real': 2.215823077590179e-05, 'epoch': 2.42}

 24%|██▍       | 177/730 [15:46<48:56,  5.31s/it]
                                                 
{'debug/num_tok_total': 2431.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2431.0, 'debug/num_lat_loss': 1792.0, 'epoch': 2.42}

 24%|██▍       | 177/730 [15:48<48:56,  5.31s/it]
                                                 
{'train/ce_loss': 3.15625, 'train/diffusion_loss': 0.47264569997787476, 'epoch': 2.42}

 24%|██▍       | 177/730 [15:48<48:56,  5.31s/it]
                                                 
{'train/learning_rate_real': 2.215823077590179e-05, 'epoch': 2.42}

 24%|██▍       | 177/730 [15:48<48:56,  5.31s/it]
 24%|██▍       | 178/730 [15:48<48:08,  5.23s/it]
                                                 
{'debug/num_tok_total': 2395.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2395.0, 'debug/num_lat_loss': 1763.0, 'epoch': 2.44}

 24%|██▍       | 178/730 [15:49<48:08,  5.23s/it]
                                                 
{'train/ce_loss': 2.890625, 'train/diffusion_loss': 0.453601598739624, 'epoch': 2.44}

 24%|██▍       | 178/730 [15:49<48:08,  5.23s/it]
                                                 
{'train/learning_rate_real': 2.2122924792873827e-05, 'epoch': 2.44}

 24%|██▍       | 178/730 [15:49<48:08,  5.23s/it]
                                                 
{'debug/num_tok_total': 3096.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 3096.0, 'debug/num_lat_loss': 1794.0, 'epoch': 2.44}

 24%|██▍       | 178/730 [15:50<48:08,  5.23s/it]
                                                 
{'train/ce_loss': 2.9375, 'train/diffusion_loss': 0.31741419434547424, 'epoch': 2.44}

 24%|██▍       | 178/730 [15:50<48:08,  5.23s/it]
                                                 
{'train/learning_rate_real': 2.2122924792873827e-05, 'epoch': 2.44}

 24%|██▍       | 178/730 [15:50<48:08,  5.23s/it]
                                                 
{'debug/num_tok_total': 3100.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 3100.0, 'debug/num_lat_loss': 1796.0, 'epoch': 2.44}

 24%|██▍       | 178/730 [15:52<48:08,  5.23s/it]
                                                 
{'train/ce_loss': 2.78125, 'train/diffusion_loss': 0.2818582057952881, 'epoch': 2.44}

 24%|██▍       | 178/730 [15:52<48:08,  5.23s/it]
                                                 
{'train/learning_rate_real': 2.2122924792873827e-05, 'epoch': 2.44}

 24%|██▍       | 178/730 [15:52<48:08,  5.23s/it]
                                                 
{'debug/num_tok_total': 2653.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2653.0, 'debug/num_lat_loss': 1792.0, 'epoch': 2.44}

 24%|██▍       | 178/730 [15:53<48:08,  5.23s/it]
                                                 
{'train/ce_loss': 2.8125, 'train/diffusion_loss': 0.4282552897930145, 'epoch': 2.44}

 24%|██▍       | 178/730 [15:53<48:08,  5.23s/it]
                                                 
{'train/learning_rate_real': 2.2122924792873827e-05, 'epoch': 2.44}

 24%|██▍       | 178/730 [15:53<48:08,  5.23s/it]
 25%|██▍       | 179/730 [15:53<48:29,  5.28s/it]
                                                 
{'debug/num_tok_total': 2639.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2639.0, 'debug/num_lat_loss': 1783.0, 'epoch': 2.45}

 25%|██▍       | 179/730 [15:54<48:29,  5.28s/it]
                                                 
{'train/ce_loss': 2.90625, 'train/diffusion_loss': 0.40325939655303955, 'epoch': 2.45}

 25%|██▍       | 179/730 [15:54<48:29,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.208742934021499e-05, 'epoch': 2.45}

 25%|██▍       | 179/730 [15:54<48:29,  5.28s/it]
                                                 
{'debug/num_tok_total': 2614.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 2614.0, 'debug/num_lat_loss': 1769.0, 'epoch': 2.45}

 25%|██▍       | 179/730 [15:56<48:29,  5.28s/it]
                                                 
{'train/ce_loss': 2.84375, 'train/diffusion_loss': 0.4070988595485687, 'epoch': 2.45}

 25%|██▍       | 179/730 [15:56<48:29,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.208742934021499e-05, 'epoch': 2.45}

 25%|██▍       | 179/730 [15:56<48:29,  5.28s/it]
                                                 
{'debug/num_tok_total': 2631.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2631.0, 'debug/num_lat_loss': 1772.0, 'epoch': 2.45}

 25%|██▍       | 179/730 [15:57<48:29,  5.28s/it]
                                                 
{'train/ce_loss': 2.671875, 'train/diffusion_loss': 0.40981626510620117, 'epoch': 2.45}

 25%|██▍       | 179/730 [15:57<48:29,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.208742934021499e-05, 'epoch': 2.45}

 25%|██▍       | 179/730 [15:57<48:29,  5.28s/it]
                                                 
{'debug/num_tok_total': 2441.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2441.0, 'debug/num_lat_loss': 1784.0, 'epoch': 2.45}

 25%|██▍       | 179/730 [15:58<48:29,  5.28s/it]
                                                 
{'train/ce_loss': 2.9375, 'train/diffusion_loss': 0.43094131350517273, 'epoch': 2.45}

 25%|██▍       | 179/730 [15:58<48:29,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.208742934021499e-05, 'epoch': 2.45}

 25%|██▍       | 179/730 [15:58<48:29,  5.28s/it]03/16/2026 07:03:31 - INFO - __main__ - LoRA debug step 180: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 25%|██▍       | 180/730 [15:59<48:16,  5.27s/it]
                                                 
{'loss': 2.7061, 'grad_norm': 1.5310542583465576, 'learning_rate': 2.208742934021499e-05, 'epoch': 2.47}

 25%|██▍       | 180/730 [15:59<48:16,  5.27s/it]
                                                 
{'debug/num_tok_total': 2117.0, 'debug/num_tok_loss': 1620.0, 'debug/num_lat_total': 2117.0, 'debug/num_lat_loss': 1620.0, 'epoch': 2.47}

 25%|██▍       | 180/730 [16:00<48:16,  5.27s/it]
                                                 
{'train/ce_loss': 2.921875, 'train/diffusion_loss': 0.4778527617454529, 'epoch': 2.47}

 25%|██▍       | 180/730 [16:00<48:16,  5.27s/it]
                                                 
{'train/learning_rate_real': 2.205174511680949e-05, 'epoch': 2.47}

 25%|██▍       | 180/730 [16:00<48:16,  5.27s/it]
                                                 
{'debug/num_tok_total': 3109.0, 'debug/num_tok_loss': 1805.0, 'debug/num_lat_total': 3109.0, 'debug/num_lat_loss': 1805.0, 'epoch': 2.47}

 25%|██▍       | 180/730 [16:01<48:16,  5.27s/it]
                                                 
{'train/ce_loss': 2.734375, 'train/diffusion_loss': 0.3286422789096832, 'epoch': 2.47}

 25%|██▍       | 180/730 [16:01<48:16,  5.27s/it]
                                                 
{'train/learning_rate_real': 2.205174511680949e-05, 'epoch': 2.47}

 25%|██▍       | 180/730 [16:01<48:16,  5.27s/it]
                                                 
{'debug/num_tok_total': 2616.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 2616.0, 'debug/num_lat_loss': 1767.0, 'epoch': 2.47}

 25%|██▍       | 180/730 [16:02<48:16,  5.27s/it]
                                                 
{'train/ce_loss': 2.765625, 'train/diffusion_loss': 0.4315715730190277, 'epoch': 2.47}

 25%|██▍       | 180/730 [16:02<48:16,  5.27s/it]
                                                 
{'train/learning_rate_real': 2.205174511680949e-05, 'epoch': 2.47}

 25%|██▍       | 180/730 [16:02<48:16,  5.27s/it]
                                                 
{'debug/num_tok_total': 2088.0, 'debug/num_tok_loss': 1606.0, 'debug/num_lat_total': 2088.0, 'debug/num_lat_loss': 1606.0, 'epoch': 2.47}

 25%|██▍       | 180/730 [16:03<48:16,  5.27s/it]
                                                 
{'train/ce_loss': 2.75, 'train/diffusion_loss': 0.47049593925476074, 'epoch': 2.47}

 25%|██▍       | 180/730 [16:03<48:16,  5.27s/it]
                                                 
{'train/learning_rate_real': 2.205174511680949e-05, 'epoch': 2.47}

 25%|██▍       | 180/730 [16:03<48:16,  5.27s/it]
 25%|██▍       | 181/730 [16:04<48:09,  5.26s/it]
                                                 
{'debug/num_tok_total': 2638.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2638.0, 'debug/num_lat_loss': 1784.0, 'epoch': 2.48}

 25%|██▍       | 181/730 [16:05<48:09,  5.26s/it]
                                                 
{'train/ce_loss': 2.84375, 'train/diffusion_loss': 0.38567018508911133, 'epoch': 2.48}

 25%|██▍       | 181/730 [16:05<48:09,  5.26s/it]
                                                 
{'train/learning_rate_real': 2.2015872825258333e-05, 'epoch': 2.48}

 25%|██▍       | 181/730 [16:05<48:09,  5.26s/it]
                                                 
{'debug/num_tok_total': 2402.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 2402.0, 'debug/num_lat_loss': 1768.0, 'epoch': 2.48}

 25%|██▍       | 181/730 [16:06<48:09,  5.26s/it]
                                                 
{'train/ce_loss': 2.75, 'train/diffusion_loss': 0.44906651973724365, 'epoch': 2.48}

 25%|██▍       | 181/730 [16:06<48:09,  5.26s/it]
                                                 
{'train/learning_rate_real': 2.2015872825258333e-05, 'epoch': 2.48}

 25%|██▍       | 181/730 [16:06<48:09,  5.26s/it]
                                                 
{'debug/num_tok_total': 3334.0, 'debug/num_tok_loss': 1809.0, 'debug/num_lat_total': 3334.0, 'debug/num_lat_loss': 1809.0, 'epoch': 2.48}

 25%|██▍       | 181/730 [16:07<48:09,  5.26s/it]
                                                 
{'train/ce_loss': 2.890625, 'train/diffusion_loss': 0.29875418543815613, 'epoch': 2.48}

 25%|██▍       | 181/730 [16:07<48:09,  5.26s/it]
                                                 
{'train/learning_rate_real': 2.2015872825258333e-05, 'epoch': 2.48}

 25%|██▍       | 181/730 [16:07<48:09,  5.26s/it]
                                                 
{'debug/num_tok_total': 2429.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2429.0, 'debug/num_lat_loss': 1780.0, 'epoch': 2.48}

 25%|██▍       | 181/730 [16:09<48:09,  5.26s/it]
                                                 
{'train/ce_loss': 2.75, 'train/diffusion_loss': 0.441359281539917, 'epoch': 2.48}

 25%|██▍       | 181/730 [16:09<48:09,  5.26s/it]
                                                 
{'train/learning_rate_real': 2.2015872825258333e-05, 'epoch': 2.48}

 25%|██▍       | 181/730 [16:09<48:09,  5.26s/it]
 25%|██▍       | 182/730 [16:09<48:04,  5.26s/it]
                                                 
{'debug/num_tok_total': 2816.0, 'debug/num_tok_loss': 1759.0, 'debug/num_lat_total': 2816.0, 'debug/num_lat_loss': 1759.0, 'epoch': 2.49}

 25%|██▍       | 182/730 [16:10<48:04,  5.26s/it]
                                                 
{'train/ce_loss': 2.75, 'train/diffusion_loss': 0.3667384386062622, 'epoch': 2.49}

 25%|██▍       | 182/730 [16:10<48:04,  5.26s/it]
                                                 
{'train/learning_rate_real': 2.1979813171865483e-05, 'epoch': 2.49}

 25%|██▍       | 182/730 [16:10<48:04,  5.26s/it]
                                                 
{'debug/num_tok_total': 2450.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2450.0, 'debug/num_lat_loss': 1794.0, 'epoch': 2.49}

 25%|██▍       | 182/730 [16:12<48:04,  5.26s/it]
                                                 
{'train/ce_loss': 3.0, 'train/diffusion_loss': 0.4418801963329315, 'epoch': 2.49}

 25%|██▍       | 182/730 [16:12<48:04,  5.26s/it]
                                                 
{'train/learning_rate_real': 2.1979813171865483e-05, 'epoch': 2.49}

 25%|██▍       | 182/730 [16:12<48:04,  5.26s/it]
                                                 
{'debug/num_tok_total': 2594.0, 'debug/num_tok_loss': 1727.0, 'debug/num_lat_total': 2594.0, 'debug/num_lat_loss': 1727.0, 'epoch': 2.49}

 25%|██▍       | 182/730 [16:13<48:04,  5.26s/it]
                                                 
{'train/ce_loss': 2.8125, 'train/diffusion_loss': 0.444356769323349, 'epoch': 2.49}

 25%|██▍       | 182/730 [16:13<48:04,  5.26s/it]
                                                 
{'train/learning_rate_real': 2.1979813171865483e-05, 'epoch': 2.49}

 25%|██▍       | 182/730 [16:13<48:04,  5.26s/it]
                                                 
{'debug/num_tok_total': 2457.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 2457.0, 'debug/num_lat_loss': 1801.0, 'epoch': 2.49}

 25%|██▍       | 182/730 [16:14<48:04,  5.26s/it]
                                                 
{'train/ce_loss': 3.015625, 'train/diffusion_loss': 0.4595908522605896, 'epoch': 2.49}

 25%|██▍       | 182/730 [16:14<48:04,  5.26s/it]
                                                 
{'train/learning_rate_real': 2.1979813171865483e-05, 'epoch': 2.49}

 25%|██▍       | 182/730 [16:14<48:04,  5.26s/it]
 25%|██▌       | 183/730 [16:14<48:06,  5.28s/it]
                                                 
{'debug/num_tok_total': 2448.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2448.0, 'debug/num_lat_loss': 1793.0, 'epoch': 2.51}

 25%|██▌       | 183/730 [16:15<48:06,  5.28s/it]
                                                 
{'train/ce_loss': 2.78125, 'train/diffusion_loss': 0.43447527289390564, 'epoch': 2.51}

 25%|██▌       | 183/730 [16:15<48:06,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.1943566866623926e-05, 'epoch': 2.51}

 25%|██▌       | 183/730 [16:15<48:06,  5.28s/it]
                                                 
{'debug/num_tok_total': 2625.0, 'debug/num_tok_loss': 1668.0, 'debug/num_lat_total': 2625.0, 'debug/num_lat_loss': 1668.0, 'epoch': 2.51}

 25%|██▌       | 183/730 [16:17<48:06,  5.28s/it]
                                                 
{'train/ce_loss': 2.703125, 'train/diffusion_loss': 0.3590381145477295, 'epoch': 2.51}

 25%|██▌       | 183/730 [16:17<48:06,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.1943566866623926e-05, 'epoch': 2.51}

 25%|██▌       | 183/730 [16:17<48:06,  5.28s/it]
                                                 
{'debug/num_tok_total': 1989.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 1989.0, 'debug/num_lat_loss': 1768.0, 'epoch': 2.51}

 25%|██▌       | 183/730 [16:18<48:06,  5.28s/it]
                                                 
{'train/ce_loss': 2.703125, 'train/diffusion_loss': 0.5137954354286194, 'epoch': 2.51}

 25%|██▌       | 183/730 [16:18<48:06,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.1943566866623926e-05, 'epoch': 2.51}

 25%|██▌       | 183/730 [16:18<48:06,  5.28s/it]
                                                 
{'debug/num_tok_total': 2186.0, 'debug/num_tok_loss': 1760.0, 'debug/num_lat_total': 2186.0, 'debug/num_lat_loss': 1760.0, 'epoch': 2.51}

 25%|██▌       | 183/730 [16:19<48:06,  5.28s/it]
                                                 
{'train/ce_loss': 2.765625, 'train/diffusion_loss': 0.4624958336353302, 'epoch': 2.51}

 25%|██▌       | 183/730 [16:19<48:06,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.1943566866623926e-05, 'epoch': 2.51}

 25%|██▌       | 183/730 [16:19<48:06,  5.28s/it]
 25%|██▌       | 184/730 [16:19<47:01,  5.17s/it]
                                                 
{'debug/num_tok_total': 2318.0, 'debug/num_tok_loss': 1720.0, 'debug/num_lat_total': 2318.0, 'debug/num_lat_loss': 1720.0, 'epoch': 2.52}

 25%|██▌       | 184/730 [16:20<47:01,  5.17s/it]
                                                 
{'train/ce_loss': 2.78125, 'train/diffusion_loss': 0.4255053699016571, 'epoch': 2.52}

 25%|██▌       | 184/730 [16:20<47:01,  5.17s/it]
                                                 
{'train/learning_rate_real': 2.190713462320173e-05, 'epoch': 2.52}

 25%|██▌       | 184/730 [16:20<47:01,  5.17s/it]
                                                 
{'debug/num_tok_total': 2850.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2850.0, 'debug/num_lat_loss': 1772.0, 'epoch': 2.52}

 25%|██▌       | 184/730 [16:22<47:01,  5.17s/it]
                                                 
{'train/ce_loss': 2.953125, 'train/diffusion_loss': 0.34381103515625, 'epoch': 2.52}

 25%|██▌       | 184/730 [16:22<47:01,  5.17s/it]
                                                 
{'train/learning_rate_real': 2.190713462320173e-05, 'epoch': 2.52}

 25%|██▌       | 184/730 [16:22<47:01,  5.17s/it]
                                                 
{'debug/num_tok_total': 2360.0, 'debug/num_tok_loss': 1647.0, 'debug/num_lat_total': 2360.0, 'debug/num_lat_loss': 1647.0, 'epoch': 2.52}

 25%|██▌       | 184/730 [16:23<47:01,  5.17s/it]
                                                 
{'train/ce_loss': 2.703125, 'train/diffusion_loss': 0.4182201325893402, 'epoch': 2.52}

 25%|██▌       | 184/730 [16:23<47:01,  5.17s/it]
                                                 
{'train/learning_rate_real': 2.190713462320173e-05, 'epoch': 2.52}

 25%|██▌       | 184/730 [16:23<47:01,  5.17s/it]
                                                 
{'debug/num_tok_total': 2648.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2648.0, 'debug/num_lat_loss': 1780.0, 'epoch': 2.52}

 25%|██▌       | 184/730 [16:24<47:01,  5.17s/it]
                                                 
{'train/ce_loss': 2.71875, 'train/diffusion_loss': 0.39211803674697876, 'epoch': 2.52}

 25%|██▌       | 184/730 [16:24<47:01,  5.17s/it]
                                                 
{'train/learning_rate_real': 2.190713462320173e-05, 'epoch': 2.52}

 25%|██▌       | 184/730 [16:24<47:01,  5.17s/it]
 25%|██▌       | 185/730 [16:25<47:03,  5.18s/it]
                                                 
{'debug/num_tok_total': 2405.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2405.0, 'debug/num_lat_loss': 1779.0, 'epoch': 2.53}

 25%|██▌       | 185/730 [16:26<47:03,  5.18s/it]
                                                 
{'train/ce_loss': 2.765625, 'train/diffusion_loss': 0.4424284100532532, 'epoch': 2.53}

 25%|██▌       | 185/730 [16:26<47:03,  5.18s/it]
                                                 
{'train/learning_rate_real': 2.187051715892797e-05, 'epoch': 2.53}

 25%|██▌       | 185/730 [16:26<47:03,  5.18s/it]
                                                 
{'debug/num_tok_total': 2670.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 2670.0, 'debug/num_lat_loss': 1797.0, 'epoch': 2.53}

 25%|██▌       | 185/730 [16:27<47:03,  5.18s/it]
                                                 
{'train/ce_loss': 2.640625, 'train/diffusion_loss': 0.3992789387702942, 'epoch': 2.53}

 25%|██▌       | 185/730 [16:27<47:03,  5.18s/it]
                                                 
{'train/learning_rate_real': 2.187051715892797e-05, 'epoch': 2.53}

 25%|██▌       | 185/730 [16:27<47:03,  5.18s/it]
                                                 
{'debug/num_tok_total': 2434.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2434.0, 'debug/num_lat_loss': 1782.0, 'epoch': 2.53}

 25%|██▌       | 185/730 [16:28<47:03,  5.18s/it]
                                                 
{'train/ce_loss': 2.53125, 'train/diffusion_loss': 0.43806523084640503, 'epoch': 2.53}

 25%|██▌       | 185/730 [16:28<47:03,  5.18s/it]
                                                 
{'train/learning_rate_real': 2.187051715892797e-05, 'epoch': 2.53}

 25%|██▌       | 185/730 [16:28<47:03,  5.18s/it]
                                                 
{'debug/num_tok_total': 2858.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2858.0, 'debug/num_lat_loss': 1791.0, 'epoch': 2.53}

 25%|██▌       | 185/730 [16:29<47:03,  5.18s/it]
                                                 
{'train/ce_loss': 2.671875, 'train/diffusion_loss': 0.3659980595111847, 'epoch': 2.53}

 25%|██▌       | 185/730 [16:29<47:03,  5.18s/it]
                                                 
{'train/learning_rate_real': 2.187051715892797e-05, 'epoch': 2.53}

 25%|██▌       | 185/730 [16:29<47:03,  5.18s/it]
 25%|██▌       | 186/730 [16:30<47:18,  5.22s/it]
                                                 
{'debug/num_tok_total': 2647.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2647.0, 'debug/num_lat_loss': 1794.0, 'epoch': 2.55}

 25%|██▌       | 186/730 [16:31<47:18,  5.22s/it]
                                                 
{'train/ce_loss': 2.671875, 'train/diffusion_loss': 0.3946780860424042, 'epoch': 2.55}

 25%|██▌       | 186/730 [16:31<47:18,  5.22s/it]
                                                 
{'train/learning_rate_real': 2.18337151947786e-05, 'epoch': 2.55}

 25%|██▌       | 186/730 [16:31<47:18,  5.22s/it]
                                                 
{'debug/num_tok_total': 2853.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2853.0, 'debug/num_lat_loss': 1789.0, 'epoch': 2.55}

 25%|██▌       | 186/730 [16:32<47:18,  5.22s/it]
                                                 
{'train/ce_loss': 2.703125, 'train/diffusion_loss': 0.3429841101169586, 'epoch': 2.55}

 25%|██▌       | 186/730 [16:32<47:18,  5.22s/it]
                                                 
{'train/learning_rate_real': 2.18337151947786e-05, 'epoch': 2.55}

 25%|██▌       | 186/730 [16:32<47:18,  5.22s/it]
                                                 
{'debug/num_tok_total': 2101.0, 'debug/num_tok_loss': 1579.0, 'debug/num_lat_total': 2101.0, 'debug/num_lat_loss': 1579.0, 'epoch': 2.55}

 25%|██▌       | 186/730 [16:34<47:18,  5.22s/it]
                                                 
{'train/ce_loss': 2.671875, 'train/diffusion_loss': 0.4572349488735199, 'epoch': 2.55}

 25%|██▌       | 186/730 [16:34<47:18,  5.22s/it]
                                                 
{'train/learning_rate_real': 2.18337151947786e-05, 'epoch': 2.55}

 25%|██▌       | 186/730 [16:34<47:18,  5.22s/it]
                                                 
{'debug/num_tok_total': 2377.0, 'debug/num_tok_loss': 1750.0, 'debug/num_lat_total': 2377.0, 'debug/num_lat_loss': 1750.0, 'epoch': 2.55}

 25%|██▌       | 186/730 [16:35<47:18,  5.22s/it]
                                                 
{'train/ce_loss': 2.71875, 'train/diffusion_loss': 0.4656928777694702, 'epoch': 2.55}

 25%|██▌       | 186/730 [16:35<47:18,  5.22s/it]
                                                 
{'train/learning_rate_real': 2.18337151947786e-05, 'epoch': 2.55}

 25%|██▌       | 186/730 [16:35<47:18,  5.22s/it]
 26%|██▌       | 187/730 [16:35<47:34,  5.26s/it]
                                                 
{'debug/num_tok_total': 2465.0, 'debug/num_tok_loss': 1812.0, 'debug/num_lat_total': 2465.0, 'debug/num_lat_loss': 1812.0, 'epoch': 2.56}

 26%|██▌       | 187/730 [16:36<47:34,  5.26s/it]
                                                 
{'train/ce_loss': 2.71875, 'train/diffusion_loss': 0.46677687764167786, 'epoch': 2.56}

 26%|██▌       | 187/730 [16:36<47:34,  5.26s/it]
                                                 
{'train/learning_rate_real': 2.179672945536227e-05, 'epoch': 2.56}

 26%|██▌       | 187/730 [16:36<47:34,  5.26s/it]
                                                 
{'debug/num_tok_total': 3053.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 3053.0, 'debug/num_lat_loss': 1777.0, 'epoch': 2.56}

 26%|██▌       | 187/730 [16:38<47:34,  5.26s/it]
                                                 
{'train/ce_loss': 2.71875, 'train/diffusion_loss': 0.32807257771492004, 'epoch': 2.56}

 26%|██▌       | 187/730 [16:38<47:34,  5.26s/it]
                                                 
{'train/learning_rate_real': 2.179672945536227e-05, 'epoch': 2.56}

 26%|██▌       | 187/730 [16:38<47:34,  5.26s/it]
                                                 
{'debug/num_tok_total': 2667.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2667.0, 'debug/num_lat_loss': 1799.0, 'epoch': 2.56}

 26%|██▌       | 187/730 [16:39<47:34,  5.26s/it]
                                                 
{'train/ce_loss': 2.734375, 'train/diffusion_loss': 0.3864240050315857, 'epoch': 2.56}

 26%|██▌       | 187/730 [16:39<47:34,  5.26s/it]
                                                 
{'train/learning_rate_real': 2.179672945536227e-05, 'epoch': 2.56}

 26%|██▌       | 187/730 [16:39<47:34,  5.26s/it]
                                                 
{'debug/num_tok_total': 2261.0, 'debug/num_tok_loss': 1818.0, 'debug/num_lat_total': 2261.0, 'debug/num_lat_loss': 1818.0, 'epoch': 2.56}

 26%|██▌       | 187/730 [16:40<47:34,  5.26s/it]
                                                 
{'train/ce_loss': 2.65625, 'train/diffusion_loss': 0.5067983865737915, 'epoch': 2.56}

 26%|██▌       | 187/730 [16:40<47:34,  5.26s/it]
                                                 
{'train/learning_rate_real': 2.179672945536227e-05, 'epoch': 2.56}

 26%|██▌       | 187/730 [16:40<47:34,  5.26s/it]
 26%|██▌       | 188/730 [16:40<47:26,  5.25s/it]
                                                 
{'debug/num_tok_total': 2997.0, 'debug/num_tok_loss': 1743.0, 'debug/num_lat_total': 2997.0, 'debug/num_lat_loss': 1743.0, 'epoch': 2.58}

 26%|██▌       | 188/730 [16:42<47:26,  5.25s/it]
                                                 
{'train/ce_loss': 2.671875, 'train/diffusion_loss': 0.3241000175476074, 'epoch': 2.58}

 26%|██▌       | 188/730 [16:42<47:26,  5.25s/it]
                                                 
{'train/learning_rate_real': 2.1759560668906064e-05, 'epoch': 2.58}

 26%|██▌       | 188/730 [16:42<47:26,  5.25s/it]
                                                 
{'debug/num_tok_total': 2853.0, 'debug/num_tok_loss': 1668.0, 'debug/num_lat_total': 2853.0, 'debug/num_lat_loss': 1668.0, 'epoch': 2.58}

 26%|██▌       | 188/730 [16:43<47:26,  5.25s/it]
                                                 
{'train/ce_loss': 2.625, 'train/diffusion_loss': 0.28809213638305664, 'epoch': 2.58}

 26%|██▌       | 188/730 [16:43<47:26,  5.25s/it]
                                                 
{'train/learning_rate_real': 2.1759560668906064e-05, 'epoch': 2.58}

 26%|██▌       | 188/730 [16:43<47:26,  5.25s/it]
                                                 
{'debug/num_tok_total': 2237.0, 'debug/num_tok_loss': 1701.0, 'debug/num_lat_total': 2237.0, 'debug/num_lat_loss': 1701.0, 'epoch': 2.58}

 26%|██▌       | 188/730 [16:44<47:26,  5.25s/it]
                                                 
{'train/ce_loss': 2.59375, 'train/diffusion_loss': 0.45437008142471313, 'epoch': 2.58}

 26%|██▌       | 188/730 [16:44<47:26,  5.25s/it]
                                                 
{'train/learning_rate_real': 2.1759560668906064e-05, 'epoch': 2.58}

 26%|██▌       | 188/730 [16:44<47:26,  5.25s/it]
                                                 
{'debug/num_tok_total': 2898.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2898.0, 'debug/num_lat_loss': 1796.0, 'epoch': 2.58}

 26%|██▌       | 188/730 [16:45<47:26,  5.25s/it]
                                                 
{'train/ce_loss': 2.53125, 'train/diffusion_loss': 0.3584519326686859, 'epoch': 2.58}

 26%|██▌       | 188/730 [16:45<47:26,  5.25s/it]
                                                 
{'train/learning_rate_real': 2.1759560668906064e-05, 'epoch': 2.58}

 26%|██▌       | 188/730 [16:45<47:26,  5.25s/it]
 26%|██▌       | 189/730 [16:46<48:01,  5.33s/it]
                                                 
{'debug/num_tok_total': 2631.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2631.0, 'debug/num_lat_loss': 1778.0, 'epoch': 2.59}

 26%|██▌       | 189/730 [16:47<48:01,  5.33s/it]
                                                 
{'train/ce_loss': 2.484375, 'train/diffusion_loss': 0.4072858393192291, 'epoch': 2.59}

 26%|██▌       | 189/730 [16:47<48:01,  5.33s/it]
                                                 
{'train/learning_rate_real': 2.172220956724114e-05, 'epoch': 2.59}

 26%|██▌       | 189/730 [16:47<48:01,  5.33s/it]
                                                 
{'debug/num_tok_total': 2878.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2878.0, 'debug/num_lat_loss': 1799.0, 'epoch': 2.59}

 26%|██▌       | 189/730 [16:48<48:01,  5.33s/it]
                                                 
{'train/ce_loss': 2.703125, 'train/diffusion_loss': 0.3507998287677765, 'epoch': 2.59}

 26%|██▌       | 189/730 [16:48<48:01,  5.33s/it]
                                                 
{'train/learning_rate_real': 2.172220956724114e-05, 'epoch': 2.59}

 26%|██▌       | 189/730 [16:48<48:01,  5.33s/it]
                                                 
{'debug/num_tok_total': 2216.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2216.0, 'debug/num_lat_loss': 1790.0, 'epoch': 2.59}

 26%|██▌       | 189/730 [16:50<48:01,  5.33s/it]
                                                 
{'train/ce_loss': 2.59375, 'train/diffusion_loss': 0.4703916013240814, 'epoch': 2.59}

 26%|██▌       | 189/730 [16:50<48:01,  5.33s/it]
                                                 
{'train/learning_rate_real': 2.172220956724114e-05, 'epoch': 2.59}

 26%|██▌       | 189/730 [16:50<48:01,  5.33s/it]
                                                 
{'debug/num_tok_total': 2653.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2653.0, 'debug/num_lat_loss': 1786.0, 'epoch': 2.59}

 26%|██▌       | 189/730 [16:51<48:01,  5.33s/it]
                                                 
{'train/ce_loss': 2.65625, 'train/diffusion_loss': 0.40979763865470886, 'epoch': 2.59}

 26%|██▌       | 189/730 [16:51<48:01,  5.33s/it]
                                                 
{'train/learning_rate_real': 2.172220956724114e-05, 'epoch': 2.59}

 26%|██▌       | 189/730 [16:51<48:01,  5.33s/it]03/16/2026 07:04:23 - INFO - __main__ - LoRA debug step 190: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 26%|██▌       | 190/730 [16:51<47:41,  5.30s/it]
                                                 
{'loss': 2.733, 'grad_norm': 1.401570200920105, 'learning_rate': 2.172220956724114e-05, 'epoch': 2.6}

 26%|██▌       | 190/730 [16:51<47:41,  5.30s/it]
                                                 
{'debug/num_tok_total': 2553.0, 'debug/num_tok_loss': 1683.0, 'debug/num_lat_total': 2553.0, 'debug/num_lat_loss': 1683.0, 'epoch': 2.6}

 26%|██▌       | 190/730 [16:52<47:41,  5.30s/it]
                                                 
{'train/ce_loss': 2.578125, 'train/diffusion_loss': 0.41486406326293945, 'epoch': 2.6}

 26%|██▌       | 190/730 [16:52<47:41,  5.30s/it]
                                                 
{'train/learning_rate_real': 2.168467688578834e-05, 'epoch': 2.6}

 26%|██▌       | 190/730 [16:52<47:41,  5.30s/it]
                                                 
{'debug/num_tok_total': 2580.0, 'debug/num_tok_loss': 1746.0, 'debug/num_lat_total': 2580.0, 'debug/num_lat_loss': 1746.0, 'epoch': 2.6}

 26%|██▌       | 190/730 [16:54<47:41,  5.30s/it]
                                                 
{'train/ce_loss': 2.75, 'train/diffusion_loss': 0.40670496225357056, 'epoch': 2.6}

 26%|██▌       | 190/730 [16:54<47:41,  5.30s/it]
                                                 
{'train/learning_rate_real': 2.168467688578834e-05, 'epoch': 2.6}

 26%|██▌       | 190/730 [16:54<47:41,  5.30s/it]
                                                 
{'debug/num_tok_total': 2869.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2869.0, 'debug/num_lat_loss': 1787.0, 'epoch': 2.6}

 26%|██▌       | 190/730 [16:55<47:41,  5.30s/it]
                                                 
{'train/ce_loss': 2.75, 'train/diffusion_loss': 0.33270514011383057, 'epoch': 2.6}

 26%|██▌       | 190/730 [16:55<47:41,  5.30s/it]
                                                 
{'train/learning_rate_real': 2.168467688578834e-05, 'epoch': 2.6}

 26%|██▌       | 190/730 [16:55<47:41,  5.30s/it]
                                                 
{'debug/num_tok_total': 2411.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2411.0, 'debug/num_lat_loss': 1770.0, 'epoch': 2.6}

 26%|██▌       | 190/730 [16:56<47:41,  5.30s/it]
                                                 
{'train/ce_loss': 2.59375, 'train/diffusion_loss': 0.45577895641326904, 'epoch': 2.6}

 26%|██▌       | 190/730 [16:56<47:41,  5.30s/it]
                                                 
{'train/learning_rate_real': 2.168467688578834e-05, 'epoch': 2.6}

 26%|██▌       | 190/730 [16:56<47:41,  5.30s/it]
 26%|██▌       | 191/730 [16:56<47:31,  5.29s/it]
                                                 
{'debug/num_tok_total': 2683.0, 'debug/num_tok_loss': 1811.0, 'debug/num_lat_total': 2683.0, 'debug/num_lat_loss': 1811.0, 'epoch': 2.62}

 26%|██▌       | 191/730 [16:58<47:31,  5.29s/it]
                                                 
{'train/ce_loss': 2.75, 'train/diffusion_loss': 0.37813183665275574, 'epoch': 2.62}

 26%|██▌       | 191/730 [16:58<47:31,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.164696336354371e-05, 'epoch': 2.62}

 26%|██▌       | 191/730 [16:58<47:31,  5.29s/it]
                                                 
{'debug/num_tok_total': 1947.0, 'debug/num_tok_loss': 1640.0, 'debug/num_lat_total': 1947.0, 'debug/num_lat_loss': 1640.0, 'epoch': 2.62}

 26%|██▌       | 191/730 [16:59<47:31,  5.29s/it]
                                                 
{'train/ce_loss': 2.578125, 'train/diffusion_loss': 0.49075964093208313, 'epoch': 2.62}

 26%|██▌       | 191/730 [16:59<47:31,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.164696336354371e-05, 'epoch': 2.62}

 26%|██▌       | 191/730 [16:59<47:31,  5.29s/it]
                                                 
{'debug/num_tok_total': 2454.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2454.0, 'debug/num_lat_loss': 1795.0, 'epoch': 2.62}

 26%|██▌       | 191/730 [17:00<47:31,  5.29s/it]
                                                 
{'train/ce_loss': 2.625, 'train/diffusion_loss': 0.4536711573600769, 'epoch': 2.62}

 26%|██▌       | 191/730 [17:00<47:31,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.164696336354371e-05, 'epoch': 2.62}

 26%|██▌       | 191/730 [17:00<47:31,  5.29s/it]
                                                 
{'debug/num_tok_total': 2471.0, 'debug/num_tok_loss': 1808.0, 'debug/num_lat_total': 2471.0, 'debug/num_lat_loss': 1808.0, 'epoch': 2.62}

 26%|██▌       | 191/730 [17:01<47:31,  5.29s/it]
                                                 
{'train/ce_loss': 2.734375, 'train/diffusion_loss': 0.4252324104309082, 'epoch': 2.62}

 26%|██▌       | 191/730 [17:01<47:31,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.164696336354371e-05, 'epoch': 2.62}

 26%|██▌       | 191/730 [17:01<47:31,  5.29s/it]
 26%|██▋       | 192/730 [17:01<46:20,  5.17s/it]
                                                 
{'debug/num_tok_total': 2596.0, 'debug/num_tok_loss': 1753.0, 'debug/num_lat_total': 2596.0, 'debug/num_lat_loss': 1753.0, 'epoch': 2.63}

 26%|██▋       | 192/730 [17:02<46:20,  5.17s/it]
                                                 
{'train/ce_loss': 2.5, 'train/diffusion_loss': 0.4330429434776306, 'epoch': 2.63}

 26%|██▋       | 192/730 [17:02<46:20,  5.17s/it]
                                                 
{'train/learning_rate_real': 2.1609069743063927e-05, 'epoch': 2.63}

 26%|██▋       | 192/730 [17:02<46:20,  5.17s/it]
                                                 
{'debug/num_tok_total': 2177.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2177.0, 'debug/num_lat_loss': 1764.0, 'epoch': 2.63}

 26%|██▋       | 192/730 [17:04<46:20,  5.17s/it]
                                                 
{'train/ce_loss': 2.640625, 'train/diffusion_loss': 0.4865707755088806, 'epoch': 2.63}

 26%|██▋       | 192/730 [17:04<46:20,  5.17s/it]
                                                 
{'train/learning_rate_real': 2.1609069743063927e-05, 'epoch': 2.63}

 26%|██▋       | 192/730 [17:04<46:20,  5.17s/it]
                                                 
{'debug/num_tok_total': 2853.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2853.0, 'debug/num_lat_loss': 1776.0, 'epoch': 2.63}

 26%|██▋       | 192/730 [17:05<46:20,  5.17s/it]
                                                 
{'train/ce_loss': 2.53125, 'train/diffusion_loss': 0.3472568392753601, 'epoch': 2.63}

 26%|██▋       | 192/730 [17:05<46:20,  5.17s/it]
                                                 
{'train/learning_rate_real': 2.1609069743063927e-05, 'epoch': 2.63}

 26%|██▋       | 192/730 [17:05<46:20,  5.17s/it]
                                                 
{'debug/num_tok_total': 2822.0, 'debug/num_tok_loss': 1761.0, 'debug/num_lat_total': 2822.0, 'debug/num_lat_loss': 1761.0, 'epoch': 2.63}

 26%|██▋       | 192/730 [17:06<46:20,  5.17s/it]
                                                 
{'train/ce_loss': 2.5, 'train/diffusion_loss': 0.3912636339664459, 'epoch': 2.63}

 26%|██▋       | 192/730 [17:06<46:20,  5.17s/it]
                                                 
{'train/learning_rate_real': 2.1609069743063927e-05, 'epoch': 2.63}

 26%|██▋       | 192/730 [17:06<46:20,  5.17s/it]
 26%|██▋       | 193/730 [17:07<46:18,  5.17s/it]
                                                 
{'debug/num_tok_total': 2314.0, 'debug/num_tok_loss': 1662.0, 'debug/num_lat_total': 2314.0, 'debug/num_lat_loss': 1662.0, 'epoch': 2.64}

 26%|██▋       | 193/730 [17:08<46:18,  5.17s/it]
                                                 
{'train/ce_loss': 2.765625, 'train/diffusion_loss': 0.4403693377971649, 'epoch': 2.64}

 26%|██▋       | 193/730 [17:08<46:18,  5.17s/it]
                                                 
{'train/learning_rate_real': 2.1570996770451702e-05, 'epoch': 2.64}

 26%|██▋       | 193/730 [17:08<46:18,  5.17s/it]
                                                 
{'debug/num_tok_total': 1999.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 1999.0, 'debug/num_lat_loss': 1785.0, 'epoch': 2.64}

 26%|██▋       | 193/730 [17:09<46:18,  5.17s/it]
                                                 
{'train/ce_loss': 2.609375, 'train/diffusion_loss': 0.517785370349884, 'epoch': 2.64}

 26%|██▋       | 193/730 [17:09<46:18,  5.17s/it]
                                                 
{'train/learning_rate_real': 2.1570996770451702e-05, 'epoch': 2.64}

 26%|██▋       | 193/730 [17:09<46:18,  5.17s/it]
                                                 
{'debug/num_tok_total': 2614.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2614.0, 'debug/num_lat_loss': 1771.0, 'epoch': 2.64}

 26%|██▋       | 193/730 [17:10<46:18,  5.17s/it]
                                                 
{'train/ce_loss': 2.75, 'train/diffusion_loss': 0.3927980065345764, 'epoch': 2.64}

 26%|██▋       | 193/730 [17:10<46:18,  5.17s/it]
                                                 
{'train/learning_rate_real': 2.1570996770451702e-05, 'epoch': 2.64}

 26%|██▋       | 193/730 [17:10<46:18,  5.17s/it]
                                                 
{'debug/num_tok_total': 2649.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2649.0, 'debug/num_lat_loss': 1793.0, 'epoch': 2.64}

 26%|██▋       | 193/730 [17:11<46:18,  5.17s/it]
                                                 
{'train/ce_loss': 2.546875, 'train/diffusion_loss': 0.41709792613983154, 'epoch': 2.64}

 26%|██▋       | 193/730 [17:11<46:18,  5.17s/it]
                                                 
{'train/learning_rate_real': 2.1570996770451702e-05, 'epoch': 2.64}

 26%|██▋       | 193/730 [17:11<46:18,  5.17s/it]
 27%|██▋       | 194/730 [17:12<46:00,  5.15s/it]
                                                 
{'debug/num_tok_total': 2651.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2651.0, 'debug/num_lat_loss': 1785.0, 'epoch': 2.66}

 27%|██▋       | 194/730 [17:13<46:00,  5.15s/it]
                                                 
{'train/ce_loss': 2.65625, 'train/diffusion_loss': 0.41924428939819336, 'epoch': 2.66}

 27%|██▋       | 194/730 [17:13<46:00,  5.15s/it]
                                                 
{'train/learning_rate_real': 2.1532745195341093e-05, 'epoch': 2.66}

 27%|██▋       | 194/730 [17:13<46:00,  5.15s/it]
                                                 
{'debug/num_tok_total': 2288.0, 'debug/num_tok_loss': 1658.0, 'debug/num_lat_total': 2288.0, 'debug/num_lat_loss': 1658.0, 'epoch': 2.66}

 27%|██▋       | 194/730 [17:14<46:00,  5.15s/it]
                                                 
{'train/ce_loss': 2.640625, 'train/diffusion_loss': 0.44974976778030396, 'epoch': 2.66}

 27%|██▋       | 194/730 [17:14<46:00,  5.15s/it]
                                                 
{'train/learning_rate_real': 2.1532745195341093e-05, 'epoch': 2.66}

 27%|██▋       | 194/730 [17:14<46:00,  5.15s/it]
                                                 
{'debug/num_tok_total': 2803.0, 'debug/num_tok_loss': 1724.0, 'debug/num_lat_total': 2803.0, 'debug/num_lat_loss': 1724.0, 'epoch': 2.66}

 27%|██▋       | 194/730 [17:15<46:00,  5.15s/it]
                                                 
{'train/ce_loss': 2.796875, 'train/diffusion_loss': 0.3286336064338684, 'epoch': 2.66}

 27%|██▋       | 194/730 [17:15<46:00,  5.15s/it]
                                                 
{'train/learning_rate_real': 2.1532745195341093e-05, 'epoch': 2.66}

 27%|██▋       | 194/730 [17:15<46:00,  5.15s/it]
                                                 
{'debug/num_tok_total': 2682.0, 'debug/num_tok_loss': 1810.0, 'debug/num_lat_total': 2682.0, 'debug/num_lat_loss': 1810.0, 'epoch': 2.66}

 27%|██▋       | 194/730 [17:16<46:00,  5.15s/it]
                                                 
{'train/ce_loss': 2.765625, 'train/diffusion_loss': 0.4254552721977234, 'epoch': 2.66}

 27%|██▋       | 194/730 [17:16<46:00,  5.15s/it]
                                                 
{'train/learning_rate_real': 2.1532745195341093e-05, 'epoch': 2.66}

 27%|██▋       | 194/730 [17:16<46:00,  5.15s/it]
 27%|██▋       | 195/730 [17:17<46:16,  5.19s/it]
                                                 
{'debug/num_tok_total': 2440.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2440.0, 'debug/num_lat_loss': 1787.0, 'epoch': 2.67}

 27%|██▋       | 195/730 [17:18<46:16,  5.19s/it]
                                                 
{'train/ce_loss': 2.53125, 'train/diffusion_loss': 0.4401073455810547, 'epoch': 2.67}

 27%|██▋       | 195/730 [17:18<46:16,  5.19s/it]
                                                 
{'train/learning_rate_real': 2.1494315770882713e-05, 'epoch': 2.67}

 27%|██▋       | 195/730 [17:18<46:16,  5.19s/it]
                                                 
{'debug/num_tok_total': 2192.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2192.0, 'debug/num_lat_loss': 1764.0, 'epoch': 2.67}

 27%|██▋       | 195/730 [17:19<46:16,  5.19s/it]
                                                 
{'train/ce_loss': 2.78125, 'train/diffusion_loss': 0.5180473923683167, 'epoch': 2.67}

 27%|██▋       | 195/730 [17:19<46:16,  5.19s/it]
                                                 
{'train/learning_rate_real': 2.1494315770882713e-05, 'epoch': 2.67}

 27%|██▋       | 195/730 [17:19<46:16,  5.19s/it]
                                                 
{'debug/num_tok_total': 2282.0, 'debug/num_tok_loss': 1644.0, 'debug/num_lat_total': 2282.0, 'debug/num_lat_loss': 1644.0, 'epoch': 2.67}

 27%|██▋       | 195/730 [17:20<46:16,  5.19s/it]
                                                 
{'train/ce_loss': 2.609375, 'train/diffusion_loss': 0.4051307737827301, 'epoch': 2.67}

 27%|██▋       | 195/730 [17:20<46:16,  5.19s/it]
                                                 
{'train/learning_rate_real': 2.1494315770882713e-05, 'epoch': 2.67}

 27%|██▋       | 195/730 [17:20<46:16,  5.19s/it]
                                                 
{'debug/num_tok_total': 3111.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 3111.0, 'debug/num_lat_loss': 1801.0, 'epoch': 2.67}

 27%|██▋       | 195/730 [17:21<46:16,  5.19s/it]
                                                 
{'train/ce_loss': 2.703125, 'train/diffusion_loss': 0.3186243176460266, 'epoch': 2.67}

 27%|██▋       | 195/730 [17:21<46:16,  5.19s/it]
                                                 
{'train/learning_rate_real': 2.1494315770882713e-05, 'epoch': 2.67}

 27%|██▋       | 195/730 [17:21<46:16,  5.19s/it]
 27%|██▋       | 196/730 [17:22<46:07,  5.18s/it]
                                                 
{'debug/num_tok_total': 2097.0, 'debug/num_tok_loss': 1417.0, 'debug/num_lat_total': 2097.0, 'debug/num_lat_loss': 1417.0, 'epoch': 2.68}

 27%|██▋       | 196/730 [17:23<46:07,  5.18s/it]
                                                 
{'train/ce_loss': 2.65625, 'train/diffusion_loss': 0.4389093816280365, 'epoch': 2.68}

 27%|██▋       | 196/730 [17:23<46:07,  5.18s/it]
                                                 
{'train/learning_rate_real': 2.145570925372894e-05, 'epoch': 2.68}

 27%|██▋       | 196/730 [17:23<46:07,  5.18s/it]
                                                 
{'debug/num_tok_total': 2651.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2651.0, 'debug/num_lat_loss': 1784.0, 'epoch': 2.68}

 27%|██▋       | 196/730 [17:24<46:07,  5.18s/it]
                                                 
{'train/ce_loss': 2.609375, 'train/diffusion_loss': 0.3858262300491333, 'epoch': 2.68}

 27%|██▋       | 196/730 [17:24<46:07,  5.18s/it]
                                                 
{'train/learning_rate_real': 2.145570925372894e-05, 'epoch': 2.68}

 27%|██▋       | 196/730 [17:24<46:07,  5.18s/it]
                                                 
{'debug/num_tok_total': 2682.0, 'debug/num_tok_loss': 1810.0, 'debug/num_lat_total': 2682.0, 'debug/num_lat_loss': 1810.0, 'epoch': 2.68}

 27%|██▋       | 196/730 [17:26<46:07,  5.18s/it]
                                                 
{'train/ce_loss': 2.875, 'train/diffusion_loss': 0.40185388922691345, 'epoch': 2.68}

 27%|██▋       | 196/730 [17:26<46:07,  5.18s/it]
                                                 
{'train/learning_rate_real': 2.145570925372894e-05, 'epoch': 2.68}

 27%|██▋       | 196/730 [17:26<46:07,  5.18s/it]
                                                 
{'debug/num_tok_total': 1666.0, 'debug/num_tok_loss': 1426.0, 'debug/num_lat_total': 1666.0, 'debug/num_lat_loss': 1426.0, 'epoch': 2.68}

 27%|██▋       | 196/730 [17:27<46:07,  5.18s/it]
                                                 
{'train/ce_loss': 2.53125, 'train/diffusion_loss': 0.4992228150367737, 'epoch': 2.68}

 27%|██▋       | 196/730 [17:27<46:07,  5.18s/it]
                                                 
{'train/learning_rate_real': 2.145570925372894e-05, 'epoch': 2.68}

 27%|██▋       | 196/730 [17:27<46:07,  5.18s/it]
 27%|██▋       | 197/730 [17:27<45:37,  5.14s/it]
                                                 
{'debug/num_tok_total': 2659.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2659.0, 'debug/num_lat_loss': 1790.0, 'epoch': 2.7}

 27%|██▋       | 197/730 [17:28<45:37,  5.14s/it]
                                                 
{'train/ce_loss': 2.53125, 'train/diffusion_loss': 0.37056055665016174, 'epoch': 2.7}

 27%|██▋       | 197/730 [17:28<45:37,  5.14s/it]
                                                 
{'train/learning_rate_real': 2.1416926404018994e-05, 'epoch': 2.7}

 27%|██▋       | 197/730 [17:28<45:37,  5.14s/it]
                                                 
{'debug/num_tok_total': 2850.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2850.0, 'debug/num_lat_loss': 1780.0, 'epoch': 2.7}

 27%|██▋       | 197/730 [17:30<45:37,  5.14s/it]
                                                 
{'train/ce_loss': 2.578125, 'train/diffusion_loss': 0.3587437570095062, 'epoch': 2.7}

 27%|██▋       | 197/730 [17:30<45:37,  5.14s/it]
                                                 
{'train/learning_rate_real': 2.1416926404018994e-05, 'epoch': 2.7}

 27%|██▋       | 197/730 [17:30<45:37,  5.14s/it]
                                                 
{'debug/num_tok_total': 3000.0, 'debug/num_tok_loss': 1743.0, 'debug/num_lat_total': 3000.0, 'debug/num_lat_loss': 1743.0, 'epoch': 2.7}

 27%|██▋       | 197/730 [17:31<45:37,  5.14s/it]
                                                 
{'train/ce_loss': 2.546875, 'train/diffusion_loss': 0.31014642119407654, 'epoch': 2.7}

 27%|██▋       | 197/730 [17:31<45:37,  5.14s/it]
                                                 
{'train/learning_rate_real': 2.1416926404018994e-05, 'epoch': 2.7}

 27%|██▋       | 197/730 [17:31<45:37,  5.14s/it]
                                                 
{'debug/num_tok_total': 2639.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2639.0, 'debug/num_lat_loss': 1777.0, 'epoch': 2.7}

 27%|██▋       | 197/730 [17:32<45:37,  5.14s/it]
                                                 
{'train/ce_loss': 2.65625, 'train/diffusion_loss': 0.41646116971969604, 'epoch': 2.7}

 27%|██▋       | 197/730 [17:32<45:37,  5.14s/it]
                                                 
{'train/learning_rate_real': 2.1416926404018994e-05, 'epoch': 2.7}

 27%|██▋       | 197/730 [17:32<45:37,  5.14s/it]
 27%|██▋       | 198/730 [17:33<46:25,  5.24s/it]
                                                 
{'debug/num_tok_total': 2249.0, 'debug/num_tok_loss': 1808.0, 'debug/num_lat_total': 2249.0, 'debug/num_lat_loss': 1808.0, 'epoch': 2.71}

 27%|██▋       | 198/730 [17:34<46:25,  5.24s/it]
                                                 
{'train/ce_loss': 2.578125, 'train/diffusion_loss': 0.5441051721572876, 'epoch': 2.71}

 27%|██▋       | 198/730 [17:34<46:25,  5.24s/it]
                                                 
{'train/learning_rate_real': 2.137796798536398e-05, 'epoch': 2.71}

 27%|██▋       | 198/730 [17:34<46:25,  5.24s/it]
                                                 
{'debug/num_tok_total': 2345.0, 'debug/num_tok_loss': 1728.0, 'debug/num_lat_total': 2345.0, 'debug/num_lat_loss': 1728.0, 'epoch': 2.71}

 27%|██▋       | 198/730 [17:35<46:25,  5.24s/it]
                                                 
{'train/ce_loss': 2.484375, 'train/diffusion_loss': 0.4259048104286194, 'epoch': 2.71}

 27%|██▋       | 198/730 [17:35<46:25,  5.24s/it]
                                                 
{'train/learning_rate_real': 2.137796798536398e-05, 'epoch': 2.71}

 27%|██▋       | 198/730 [17:35<46:25,  5.24s/it]
                                                 
{'debug/num_tok_total': 2249.0, 'debug/num_tok_loss': 1803.0, 'debug/num_lat_total': 2249.0, 'debug/num_lat_loss': 1803.0, 'epoch': 2.71}

 27%|██▋       | 198/730 [17:36<46:25,  5.24s/it]
                                                 
{'train/ce_loss': 2.53125, 'train/diffusion_loss': 0.5017729997634888, 'epoch': 2.71}

 27%|██▋       | 198/730 [17:36<46:25,  5.24s/it]
                                                 
{'train/learning_rate_real': 2.137796798536398e-05, 'epoch': 2.71}

 27%|██▋       | 198/730 [17:36<46:25,  5.24s/it]
                                                 
{'debug/num_tok_total': 2480.0, 'debug/num_tok_loss': 1589.0, 'debug/num_lat_total': 2480.0, 'debug/num_lat_loss': 1589.0, 'epoch': 2.71}

 27%|██▋       | 198/730 [17:37<46:25,  5.24s/it]
                                                 
{'train/ce_loss': 2.90625, 'train/diffusion_loss': 0.3666616976261139, 'epoch': 2.71}

 27%|██▋       | 198/730 [17:37<46:25,  5.24s/it]
                                                 
{'train/learning_rate_real': 2.137796798536398e-05, 'epoch': 2.71}

 27%|██▋       | 198/730 [17:37<46:25,  5.24s/it]
 27%|██▋       | 199/730 [17:38<45:43,  5.17s/it]
                                                 
{'debug/num_tok_total': 2620.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2620.0, 'debug/num_lat_loss': 1776.0, 'epoch': 2.73}

 27%|██▋       | 199/730 [17:39<45:43,  5.17s/it]
                                                 
{'train/ce_loss': 2.53125, 'train/diffusion_loss': 0.4006059169769287, 'epoch': 2.73}

 27%|██▋       | 199/730 [17:39<45:43,  5.17s/it]
                                                 
{'train/learning_rate_real': 2.1338834764831845e-05, 'epoch': 2.73}

 27%|██▋       | 199/730 [17:39<45:43,  5.17s/it]
                                                 
{'debug/num_tok_total': 3061.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 3061.0, 'debug/num_lat_loss': 1775.0, 'epoch': 2.73}

 27%|██▋       | 199/730 [17:40<45:43,  5.17s/it]
                                                 
{'train/ce_loss': 2.59375, 'train/diffusion_loss': 0.322833776473999, 'epoch': 2.73}

 27%|██▋       | 199/730 [17:40<45:43,  5.17s/it]
                                                 
{'train/learning_rate_real': 2.1338834764831845e-05, 'epoch': 2.73}

 27%|██▋       | 199/730 [17:40<45:43,  5.17s/it]
                                                 
{'debug/num_tok_total': 2646.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2646.0, 'debug/num_lat_loss': 1787.0, 'epoch': 2.73}

 27%|██▋       | 199/730 [17:41<45:43,  5.17s/it]
                                                 
{'train/ce_loss': 2.671875, 'train/diffusion_loss': 0.4167260527610779, 'epoch': 2.73}

 27%|██▋       | 199/730 [17:41<45:43,  5.17s/it]
                                                 
{'train/learning_rate_real': 2.1338834764831845e-05, 'epoch': 2.73}

 27%|██▋       | 199/730 [17:41<45:43,  5.17s/it]
                                                 
{'debug/num_tok_total': 2461.0, 'debug/num_tok_loss': 1809.0, 'debug/num_lat_total': 2461.0, 'debug/num_lat_loss': 1809.0, 'epoch': 2.73}

 27%|██▋       | 199/730 [17:42<45:43,  5.17s/it]
                                                 
{'train/ce_loss': 2.390625, 'train/diffusion_loss': 0.40909343957901, 'epoch': 2.73}

 27%|██▋       | 199/730 [17:42<45:43,  5.17s/it]
                                                 
{'train/learning_rate_real': 2.1338834764831845e-05, 'epoch': 2.73}

 27%|██▋       | 199/730 [17:42<45:43,  5.17s/it]03/16/2026 07:05:15 - INFO - __main__ - LoRA debug step 200: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 27%|██▋       | 200/730 [17:43<46:04,  5.22s/it]
                                                 
{'loss': 2.7536, 'grad_norm': 1.5308177471160889, 'learning_rate': 2.1338834764831845e-05, 'epoch': 2.74}

 27%|██▋       | 200/730 [17:43<46:04,  5.22s/it]/home/ubuntu/.local/lib/python3.10/site-packages/peft/utils/save_and_load.py:295: UserWarning: Could not find a config file in  - will assume that the vocabulary was not modified.
  warnings.warn(

                                                 
{'debug/num_tok_total': 2228.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2228.0, 'debug/num_lat_loss': 1786.0, 'epoch': 2.74}

 27%|██▋       | 200/730 [17:46<46:04,  5.22s/it]
                                                 
{'train/ce_loss': 2.546875, 'train/diffusion_loss': 0.5127332806587219, 'epoch': 2.74}

 27%|██▋       | 200/730 [17:46<46:04,  5.22s/it]
                                                 
{'train/learning_rate_real': 2.129952751293229e-05, 'epoch': 2.74}

 27%|██▋       | 200/730 [17:46<46:04,  5.22s/it]
                                                 
{'debug/num_tok_total': 3063.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 3063.0, 'debug/num_lat_loss': 1780.0, 'epoch': 2.74}

 27%|██▋       | 200/730 [17:47<46:04,  5.22s/it]
                                                 
{'train/ce_loss': 2.53125, 'train/diffusion_loss': 0.3455106317996979, 'epoch': 2.74}

 27%|██▋       | 200/730 [17:47<46:04,  5.22s/it]
                                                 
{'train/learning_rate_real': 2.129952751293229e-05, 'epoch': 2.74}

 27%|██▋       | 200/730 [17:47<46:04,  5.22s/it]
                                                 
{'debug/num_tok_total': 2945.0, 'debug/num_tok_loss': 1611.0, 'debug/num_lat_total': 2945.0, 'debug/num_lat_loss': 1611.0, 'epoch': 2.74}

 27%|██▋       | 200/730 [17:48<46:04,  5.22s/it]
                                                 
{'train/ce_loss': 2.71875, 'train/diffusion_loss': 0.3095567524433136, 'epoch': 2.74}

 27%|██▋       | 200/730 [17:48<46:04,  5.22s/it]
                                                 
{'train/learning_rate_real': 2.129952751293229e-05, 'epoch': 2.74}

 27%|██▋       | 200/730 [17:48<46:04,  5.22s/it]
                                                 
{'debug/num_tok_total': 2855.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2855.0, 'debug/num_lat_loss': 1780.0, 'epoch': 2.74}

 27%|██▋       | 200/730 [17:50<46:04,  5.22s/it]
                                                 
{'train/ce_loss': 2.609375, 'train/diffusion_loss': 0.3999537229537964, 'epoch': 2.74}

 27%|██▋       | 200/730 [17:50<46:04,  5.22s/it]
                                                 
{'train/learning_rate_real': 2.129952751293229e-05, 'epoch': 2.74}

 27%|██▋       | 200/730 [17:50<46:04,  5.22s/it]
 28%|██▊       | 201/730 [17:50<51:04,  5.79s/it]
                                                 
{'debug/num_tok_total': 2412.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2412.0, 'debug/num_lat_loss': 1786.0, 'epoch': 2.75}

 28%|██▊       | 201/730 [17:51<51:04,  5.79s/it]
                                                 
{'train/ce_loss': 2.515625, 'train/diffusion_loss': 0.46536168456077576, 'epoch': 2.75}

 28%|██▊       | 201/730 [17:51<51:04,  5.79s/it]
                                                 
{'train/learning_rate_real': 2.1260047003601576e-05, 'epoch': 2.75}

 28%|██▊       | 201/730 [17:51<51:04,  5.79s/it]
                                                 
{'debug/num_tok_total': 2739.0, 'debug/num_tok_loss': 1718.0, 'debug/num_lat_total': 2739.0, 'debug/num_lat_loss': 1718.0, 'epoch': 2.75}

 28%|██▊       | 201/730 [17:52<51:04,  5.79s/it]
                                                 
{'train/ce_loss': 2.5625, 'train/diffusion_loss': 0.39696988463401794, 'epoch': 2.75}

 28%|██▊       | 201/730 [17:52<51:04,  5.79s/it]
                                                 
{'train/learning_rate_real': 2.1260047003601576e-05, 'epoch': 2.75}

 28%|██▊       | 201/730 [17:52<51:04,  5.79s/it]
                                                 
{'debug/num_tok_total': 3074.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 3074.0, 'debug/num_lat_loss': 1777.0, 'epoch': 2.75}

 28%|██▊       | 201/730 [17:54<51:04,  5.79s/it]
                                                 
{'train/ce_loss': 2.421875, 'train/diffusion_loss': 0.3201371431350708, 'epoch': 2.75}

 28%|██▊       | 201/730 [17:54<51:04,  5.79s/it]
                                                 
{'train/learning_rate_real': 2.1260047003601576e-05, 'epoch': 2.75}

 28%|██▊       | 201/730 [17:54<51:04,  5.79s/it]
                                                 
{'debug/num_tok_total': 3307.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 3307.0, 'debug/num_lat_loss': 1795.0, 'epoch': 2.75}

 28%|██▊       | 201/730 [17:55<51:04,  5.79s/it]
                                                 
{'train/ce_loss': 2.671875, 'train/diffusion_loss': 0.3175411522388458, 'epoch': 2.75}

 28%|██▊       | 201/730 [17:55<51:04,  5.79s/it]
                                                 
{'train/learning_rate_real': 2.1260047003601576e-05, 'epoch': 2.75}

 28%|██▊       | 201/730 [17:55<51:04,  5.79s/it]
 28%|██▊       | 202/730 [17:56<50:23,  5.73s/it]
                                                 
{'debug/num_tok_total': 2385.0, 'debug/num_tok_loss': 1746.0, 'debug/num_lat_total': 2385.0, 'debug/num_lat_loss': 1746.0, 'epoch': 2.77}

 28%|██▊       | 202/730 [17:57<50:23,  5.73s/it]
                                                 
{'train/ce_loss': 2.5, 'train/diffusion_loss': 0.4627111554145813, 'epoch': 2.77}

 28%|██▊       | 202/730 [17:57<50:23,  5.73s/it]
                                                 
{'train/learning_rate_real': 2.1220394014187312e-05, 'epoch': 2.77}

 28%|██▊       | 202/730 [17:57<50:23,  5.73s/it]
                                                 
{'debug/num_tok_total': 2919.0, 'debug/num_tok_loss': 1813.0, 'debug/num_lat_total': 2919.0, 'debug/num_lat_loss': 1813.0, 'epoch': 2.77}

 28%|██▊       | 202/730 [17:58<50:23,  5.73s/it]
                                                 
{'train/ce_loss': 2.5625, 'train/diffusion_loss': 0.3691011071205139, 'epoch': 2.77}

 28%|██▊       | 202/730 [17:58<50:23,  5.73s/it]
                                                 
{'train/learning_rate_real': 2.1220394014187312e-05, 'epoch': 2.77}

 28%|██▊       | 202/730 [17:58<50:23,  5.73s/it]
                                                 
{'debug/num_tok_total': 2869.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2869.0, 'debug/num_lat_loss': 1788.0, 'epoch': 2.77}

 28%|██▊       | 202/730 [17:59<50:23,  5.73s/it]
                                                 
{'train/ce_loss': 2.453125, 'train/diffusion_loss': 0.3681058883666992, 'epoch': 2.77}

 28%|██▊       | 202/730 [17:59<50:23,  5.73s/it]
                                                 
{'train/learning_rate_real': 2.1220394014187312e-05, 'epoch': 2.77}

 28%|██▊       | 202/730 [17:59<50:23,  5.73s/it]
                                                 
{'debug/num_tok_total': 2761.0, 'debug/num_tok_loss': 1742.0, 'debug/num_lat_total': 2761.0, 'debug/num_lat_loss': 1742.0, 'epoch': 2.77}

 28%|██▊       | 202/730 [18:01<50:23,  5.73s/it]
                                                 
{'train/ce_loss': 2.625, 'train/diffusion_loss': 0.4040772616863251, 'epoch': 2.77}

 28%|██▊       | 202/730 [18:01<50:23,  5.73s/it]
                                                 
{'train/learning_rate_real': 2.1220394014187312e-05, 'epoch': 2.77}

 28%|██▊       | 202/730 [18:01<50:23,  5.73s/it]
 28%|██▊       | 203/730 [18:01<49:16,  5.61s/it]
                                                 
{'debug/num_tok_total': 3058.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 3058.0, 'debug/num_lat_loss': 1778.0, 'epoch': 2.78}

 28%|██▊       | 203/730 [18:02<49:16,  5.61s/it]
                                                 
{'train/ce_loss': 2.390625, 'train/diffusion_loss': 0.3349795341491699, 'epoch': 2.78}

 28%|██▊       | 203/730 [18:02<49:16,  5.61s/it]
                                                 
{'train/learning_rate_real': 2.1180569325433132e-05, 'epoch': 2.78}

 28%|██▊       | 203/730 [18:02<49:16,  5.61s/it]
                                                 
{'debug/num_tok_total': 2268.0, 'debug/num_tok_loss': 1821.0, 'debug/num_lat_total': 2268.0, 'debug/num_lat_loss': 1821.0, 'epoch': 2.78}

 28%|██▊       | 203/730 [18:03<49:16,  5.61s/it]
                                                 
{'train/ce_loss': 2.40625, 'train/diffusion_loss': 0.4954097867012024, 'epoch': 2.78}

 28%|██▊       | 203/730 [18:03<49:16,  5.61s/it]
                                                 
{'train/learning_rate_real': 2.1180569325433132e-05, 'epoch': 2.78}

 28%|██▊       | 203/730 [18:03<49:16,  5.61s/it]
                                                 
{'debug/num_tok_total': 2661.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2661.0, 'debug/num_lat_loss': 1790.0, 'epoch': 2.78}

 28%|██▊       | 203/730 [18:05<49:16,  5.61s/it]
                                                 
{'train/ce_loss': 2.515625, 'train/diffusion_loss': 0.44348451495170593, 'epoch': 2.78}

 28%|██▊       | 203/730 [18:05<49:16,  5.61s/it]
                                                 
{'train/learning_rate_real': 2.1180569325433132e-05, 'epoch': 2.78}

 28%|██▊       | 203/730 [18:05<49:16,  5.61s/it]
                                                 
{'debug/num_tok_total': 2685.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 2685.0, 'debug/num_lat_loss': 1801.0, 'epoch': 2.78}

 28%|██▊       | 203/730 [18:06<49:16,  5.61s/it]
                                                 
{'train/ce_loss': 2.46875, 'train/diffusion_loss': 0.3852037191390991, 'epoch': 2.78}

 28%|██▊       | 203/730 [18:06<49:16,  5.61s/it]
                                                 
{'train/learning_rate_real': 2.1180569325433132e-05, 'epoch': 2.78}

 28%|██▊       | 203/730 [18:06<49:16,  5.61s/it]
 28%|██▊       | 204/730 [18:06<48:26,  5.53s/it]
                                                 
{'debug/num_tok_total': 2232.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2232.0, 'debug/num_lat_loss': 1791.0, 'epoch': 2.79}

 28%|██▊       | 204/730 [18:07<48:26,  5.53s/it]
                                                 
{'train/ce_loss': 2.5625, 'train/diffusion_loss': 0.49917471408843994, 'epoch': 2.79}

 28%|██▊       | 204/730 [18:07<48:26,  5.53s/it]
                                                 
{'train/learning_rate_real': 2.114057372146332e-05, 'epoch': 2.79}

 28%|██▊       | 204/730 [18:07<48:26,  5.53s/it]
                                                 
{'debug/num_tok_total': 2410.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 2410.0, 'debug/num_lat_loss': 1768.0, 'epoch': 2.79}

 28%|██▊       | 204/730 [18:09<48:26,  5.53s/it]
                                                 
{'train/ce_loss': 2.640625, 'train/diffusion_loss': 0.502190351486206, 'epoch': 2.79}

 28%|██▊       | 204/730 [18:09<48:26,  5.53s/it]
                                                 
{'train/learning_rate_real': 2.114057372146332e-05, 'epoch': 2.79}

 28%|██▊       | 204/730 [18:09<48:26,  5.53s/it]
                                                 
{'debug/num_tok_total': 2865.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2865.0, 'debug/num_lat_loss': 1776.0, 'epoch': 2.79}

 28%|██▊       | 204/730 [18:10<48:26,  5.53s/it]
                                                 
{'train/ce_loss': 2.578125, 'train/diffusion_loss': 0.40779879689216614, 'epoch': 2.79}

 28%|██▊       | 204/730 [18:10<48:26,  5.53s/it]
                                                 
{'train/learning_rate_real': 2.114057372146332e-05, 'epoch': 2.79}

 28%|██▊       | 204/730 [18:10<48:26,  5.53s/it]
                                                 
{'debug/num_tok_total': 2773.0, 'debug/num_tok_loss': 1628.0, 'debug/num_lat_total': 2773.0, 'debug/num_lat_loss': 1628.0, 'epoch': 2.79}

 28%|██▊       | 204/730 [18:11<48:26,  5.53s/it]
                                                 
{'train/ce_loss': 2.3125, 'train/diffusion_loss': 0.3184314966201782, 'epoch': 2.79}

 28%|██▊       | 204/730 [18:11<48:26,  5.53s/it]
                                                 
{'train/learning_rate_real': 2.114057372146332e-05, 'epoch': 2.79}

 28%|██▊       | 204/730 [18:11<48:26,  5.53s/it]
 28%|██▊       | 205/730 [18:12<47:43,  5.46s/it]
                                                 
{'debug/num_tok_total': 2456.0, 'debug/num_tok_loss': 1812.0, 'debug/num_lat_total': 2456.0, 'debug/num_lat_loss': 1812.0, 'epoch': 2.81}

 28%|██▊       | 205/730 [18:13<47:43,  5.46s/it]
                                                 
{'train/ce_loss': 2.484375, 'train/diffusion_loss': 0.43674784898757935, 'epoch': 2.81}

 28%|██▊       | 205/730 [18:13<47:43,  5.46s/it]
                                                 
{'train/learning_rate_real': 2.1100407989767395e-05, 'epoch': 2.81}

 28%|██▊       | 205/730 [18:13<47:43,  5.46s/it]
                                                 
{'debug/num_tok_total': 2690.0, 'debug/num_tok_loss': 1706.0, 'debug/num_lat_total': 2690.0, 'debug/num_lat_loss': 1706.0, 'epoch': 2.81}

 28%|██▊       | 205/730 [18:14<47:43,  5.46s/it]
                                                 
{'train/ce_loss': 2.4375, 'train/diffusion_loss': 0.3507964015007019, 'epoch': 2.81}

 28%|██▊       | 205/730 [18:14<47:43,  5.46s/it]
                                                 
{'train/learning_rate_real': 2.1100407989767395e-05, 'epoch': 2.81}

 28%|██▊       | 205/730 [18:14<47:43,  5.46s/it]
                                                 
{'debug/num_tok_total': 2607.0, 'debug/num_tok_loss': 1761.0, 'debug/num_lat_total': 2607.0, 'debug/num_lat_loss': 1761.0, 'epoch': 2.81}

 28%|██▊       | 205/730 [18:15<47:43,  5.46s/it]
                                                 
{'train/ce_loss': 2.5, 'train/diffusion_loss': 0.43347081542015076, 'epoch': 2.81}

 28%|██▊       | 205/730 [18:15<47:43,  5.46s/it]
                                                 
{'train/learning_rate_real': 2.1100407989767395e-05, 'epoch': 2.81}

 28%|██▊       | 205/730 [18:15<47:43,  5.46s/it]
                                                 
{'debug/num_tok_total': 2637.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2637.0, 'debug/num_lat_loss': 1783.0, 'epoch': 2.81}

 28%|██▊       | 205/730 [18:16<47:43,  5.46s/it]
                                                 
{'train/ce_loss': 2.375, 'train/diffusion_loss': 0.42806491255760193, 'epoch': 2.81}

 28%|██▊       | 205/730 [18:16<47:43,  5.46s/it]
                                                 
{'train/learning_rate_real': 2.1100407989767395e-05, 'epoch': 2.81}

 28%|██▊       | 205/730 [18:16<47:43,  5.46s/it]
 28%|██▊       | 206/730 [18:17<47:12,  5.41s/it]
                                                 
{'debug/num_tok_total': 2845.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 2845.0, 'debug/num_lat_loss': 1765.0, 'epoch': 2.82}

 28%|██▊       | 206/730 [18:18<47:12,  5.41s/it]
                                                 
{'train/ce_loss': 2.40625, 'train/diffusion_loss': 0.3783215582370758, 'epoch': 2.82}

 28%|██▊       | 206/730 [18:18<47:12,  5.41s/it]
                                                 
{'train/learning_rate_real': 2.106007292118457e-05, 'epoch': 2.82}

 28%|██▊       | 206/730 [18:18<47:12,  5.41s/it]
                                                 
{'debug/num_tok_total': 2449.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2449.0, 'debug/num_lat_loss': 1795.0, 'epoch': 2.82}

 28%|██▊       | 206/730 [18:19<47:12,  5.41s/it]
                                                 
{'train/ce_loss': 2.40625, 'train/diffusion_loss': 0.4463353157043457, 'epoch': 2.82}

 28%|██▊       | 206/730 [18:19<47:12,  5.41s/it]
                                                 
{'train/learning_rate_real': 2.106007292118457e-05, 'epoch': 2.82}

 28%|██▊       | 206/730 [18:19<47:12,  5.41s/it]
                                                 
{'debug/num_tok_total': 2400.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 2400.0, 'debug/num_lat_loss': 1767.0, 'epoch': 2.82}

 28%|██▊       | 206/730 [18:21<47:12,  5.41s/it]
                                                 
{'train/ce_loss': 2.421875, 'train/diffusion_loss': 0.44486525654792786, 'epoch': 2.82}

 28%|██▊       | 206/730 [18:21<47:12,  5.41s/it]
                                                 
{'train/learning_rate_real': 2.106007292118457e-05, 'epoch': 2.82}

 28%|██▊       | 206/730 [18:21<47:12,  5.41s/it]
                                                 
{'debug/num_tok_total': 2861.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2861.0, 'debug/num_lat_loss': 1792.0, 'epoch': 2.82}

 28%|██▊       | 206/730 [18:22<47:12,  5.41s/it]
                                                 
{'train/ce_loss': 2.578125, 'train/diffusion_loss': 0.37948623299598694, 'epoch': 2.82}

 28%|██▊       | 206/730 [18:22<47:12,  5.41s/it]
                                                 
{'train/learning_rate_real': 2.106007292118457e-05, 'epoch': 2.82}

 28%|██▊       | 206/730 [18:22<47:12,  5.41s/it]
 28%|██▊       | 207/730 [18:22<47:22,  5.44s/it]
                                                 
{'debug/num_tok_total': 2969.0, 'debug/num_tok_loss': 1727.0, 'debug/num_lat_total': 2969.0, 'debug/num_lat_loss': 1727.0, 'epoch': 2.84}

 28%|██▊       | 207/730 [18:24<47:22,  5.44s/it]
                                                 
{'train/ce_loss': 2.484375, 'train/diffusion_loss': 0.3458055555820465, 'epoch': 2.84}

 28%|██▊       | 207/730 [18:24<47:22,  5.44s/it]
                                                 
{'train/learning_rate_real': 2.1019569309888216e-05, 'epoch': 2.84}

 28%|██▊       | 207/730 [18:24<47:22,  5.44s/it]
                                                 
{'debug/num_tok_total': 2899.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2899.0, 'debug/num_lat_loss': 1796.0, 'epoch': 2.84}

 28%|██▊       | 207/730 [18:25<47:22,  5.44s/it]
                                                 
{'train/ce_loss': 2.375, 'train/diffusion_loss': 0.35966601967811584, 'epoch': 2.84}

 28%|██▊       | 207/730 [18:25<47:22,  5.44s/it]
                                                 
{'train/learning_rate_real': 2.1019569309888216e-05, 'epoch': 2.84}

 28%|██▊       | 207/730 [18:25<47:22,  5.44s/it]
                                                 
{'debug/num_tok_total': 2866.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2866.0, 'debug/num_lat_loss': 1787.0, 'epoch': 2.84}

 28%|██▊       | 207/730 [18:26<47:22,  5.44s/it]
                                                 
{'train/ce_loss': 2.515625, 'train/diffusion_loss': 0.3660614788532257, 'epoch': 2.84}

 28%|██▊       | 207/730 [18:26<47:22,  5.44s/it]
                                                 
{'train/learning_rate_real': 2.1019569309888216e-05, 'epoch': 2.84}

 28%|██▊       | 207/730 [18:26<47:22,  5.44s/it]
                                                 
{'debug/num_tok_total': 3065.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 3065.0, 'debug/num_lat_loss': 1773.0, 'epoch': 2.84}

 28%|██▊       | 207/730 [18:28<47:22,  5.44s/it]
                                                 
{'train/ce_loss': 2.515625, 'train/diffusion_loss': 0.3202878534793854, 'epoch': 2.84}

 28%|██▊       | 207/730 [18:28<47:22,  5.44s/it]
                                                 
{'train/learning_rate_real': 2.1019569309888216e-05, 'epoch': 2.84}

 28%|██▊       | 207/730 [18:28<47:22,  5.44s/it]
 28%|██▊       | 208/730 [18:28<48:04,  5.53s/it]
                                                 
{'debug/num_tok_total': 2190.0, 'debug/num_tok_loss': 1758.0, 'debug/num_lat_total': 2190.0, 'debug/num_lat_loss': 1758.0, 'epoch': 2.85}

 28%|██▊       | 208/730 [18:29<48:04,  5.53s/it]
                                                 
{'train/ce_loss': 2.375, 'train/diffusion_loss': 0.5103146433830261, 'epoch': 2.85}

 28%|██▊       | 208/730 [18:29<48:04,  5.53s/it]
                                                 
{'train/learning_rate_real': 2.0978897953370204e-05, 'epoch': 2.85}

 28%|██▊       | 208/730 [18:29<48:04,  5.53s/it]
                                                 
{'debug/num_tok_total': 2629.0, 'debug/num_tok_loss': 1756.0, 'debug/num_lat_total': 2629.0, 'debug/num_lat_loss': 1756.0, 'epoch': 2.85}

 28%|██▊       | 208/730 [18:30<48:04,  5.53s/it]
                                                 
{'train/ce_loss': 2.40625, 'train/diffusion_loss': 0.3646072745323181, 'epoch': 2.85}

 28%|██▊       | 208/730 [18:30<48:04,  5.53s/it]
                                                 
{'train/learning_rate_real': 2.0978897953370204e-05, 'epoch': 2.85}

 28%|██▊       | 208/730 [18:30<48:04,  5.53s/it]
                                                 
{'debug/num_tok_total': 2529.0, 'debug/num_tok_loss': 1679.0, 'debug/num_lat_total': 2529.0, 'debug/num_lat_loss': 1679.0, 'epoch': 2.85}

 28%|██▊       | 208/730 [18:32<48:04,  5.53s/it]
                                                 
{'train/ce_loss': 2.703125, 'train/diffusion_loss': 0.4177684485912323, 'epoch': 2.85}

 28%|██▊       | 208/730 [18:32<48:04,  5.53s/it]
                                                 
{'train/learning_rate_real': 2.0978897953370204e-05, 'epoch': 2.85}

 28%|██▊       | 208/730 [18:32<48:04,  5.53s/it]
                                                 
{'debug/num_tok_total': 2852.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 2852.0, 'debug/num_lat_loss': 1768.0, 'epoch': 2.85}

 28%|██▊       | 208/730 [18:33<48:04,  5.53s/it]
                                                 
{'train/ce_loss': 2.453125, 'train/diffusion_loss': 0.3632550835609436, 'epoch': 2.85}

 28%|██▊       | 208/730 [18:33<48:04,  5.53s/it]
                                                 
{'train/learning_rate_real': 2.0978897953370204e-05, 'epoch': 2.85}

 28%|██▊       | 208/730 [18:33<48:04,  5.53s/it]
 29%|██▊       | 209/730 [18:33<47:01,  5.42s/it]
                                                 
{'debug/num_tok_total': 2360.0, 'debug/num_tok_loss': 1707.0, 'debug/num_lat_total': 2360.0, 'debug/num_lat_loss': 1707.0, 'epoch': 2.86}

 29%|██▊       | 209/730 [18:34<47:01,  5.42s/it]
                                                 
{'train/ce_loss': 2.765625, 'train/diffusion_loss': 0.45615288615226746, 'epoch': 2.86}

 29%|██▊       | 209/730 [18:34<47:01,  5.42s/it]
                                                 
{'train/learning_rate_real': 2.0938059652425196e-05, 'epoch': 2.86}

 29%|██▊       | 209/730 [18:34<47:01,  5.42s/it]
                                                 
{'debug/num_tok_total': 2475.0, 'debug/num_tok_loss': 1817.0, 'debug/num_lat_total': 2475.0, 'debug/num_lat_loss': 1817.0, 'epoch': 2.86}

 29%|██▊       | 209/730 [18:36<47:01,  5.42s/it]
                                                 
{'train/ce_loss': 2.40625, 'train/diffusion_loss': 0.4588419795036316, 'epoch': 2.86}

 29%|██▊       | 209/730 [18:36<47:01,  5.42s/it]
                                                 
{'train/learning_rate_real': 2.0938059652425196e-05, 'epoch': 2.86}

 29%|██▊       | 209/730 [18:36<47:01,  5.42s/it]
                                                 
{'debug/num_tok_total': 2627.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2627.0, 'debug/num_lat_loss': 1774.0, 'epoch': 2.86}

 29%|██▊       | 209/730 [18:37<47:01,  5.42s/it]
                                                 
{'train/ce_loss': 2.4375, 'train/diffusion_loss': 0.40409165620803833, 'epoch': 2.86}

 29%|██▊       | 209/730 [18:37<47:01,  5.42s/it]
                                                 
{'train/learning_rate_real': 2.0938059652425196e-05, 'epoch': 2.86}

 29%|██▊       | 209/730 [18:37<47:01,  5.42s/it]
                                                 
{'debug/num_tok_total': 1866.0, 'debug/num_tok_loss': 1648.0, 'debug/num_lat_total': 1866.0, 'debug/num_lat_loss': 1648.0, 'epoch': 2.86}

 29%|██▊       | 209/730 [18:38<47:01,  5.42s/it]
                                                 
{'train/ce_loss': 2.53125, 'train/diffusion_loss': 0.5563381910324097, 'epoch': 2.86}

 29%|██▊       | 209/730 [18:38<47:01,  5.42s/it]
                                                 
{'train/learning_rate_real': 2.0938059652425196e-05, 'epoch': 2.86}

 29%|██▊       | 209/730 [18:38<47:01,  5.42s/it]03/16/2026 07:06:10 - INFO - __main__ - LoRA debug step 210: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 29%|██▉       | 210/730 [18:38<46:01,  5.31s/it]
                                                 
{'loss': 2.6799, 'grad_norm': 1.6836766004562378, 'learning_rate': 2.0938059652425196e-05, 'epoch': 2.88}

 29%|██▉       | 210/730 [18:38<46:01,  5.31s/it]
                                                 
{'debug/num_tok_total': 2425.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2425.0, 'debug/num_lat_loss': 1783.0, 'epoch': 2.88}

 29%|██▉       | 210/730 [18:39<46:01,  5.31s/it]
                                                 
{'train/ce_loss': 2.421875, 'train/diffusion_loss': 0.478886216878891, 'epoch': 2.88}

 29%|██▉       | 210/730 [18:39<46:01,  5.31s/it]
                                                 
{'train/learning_rate_real': 2.0897055211134912e-05, 'epoch': 2.88}

 29%|██▉       | 210/730 [18:39<46:01,  5.31s/it]
                                                 
{'debug/num_tok_total': 2417.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2417.0, 'debug/num_lat_loss': 1777.0, 'epoch': 2.88}

 29%|██▉       | 210/730 [18:41<46:01,  5.31s/it]
                                                 
{'train/ce_loss': 2.421875, 'train/diffusion_loss': 0.48459577560424805, 'epoch': 2.88}

 29%|██▉       | 210/730 [18:41<46:01,  5.31s/it]
                                                 
{'train/learning_rate_real': 2.0897055211134912e-05, 'epoch': 2.88}

 29%|██▉       | 210/730 [18:41<46:01,  5.31s/it]
                                                 
{'debug/num_tok_total': 2684.0, 'debug/num_tok_loss': 1696.0, 'debug/num_lat_total': 2684.0, 'debug/num_lat_loss': 1696.0, 'epoch': 2.88}

 29%|██▉       | 210/730 [18:42<46:01,  5.31s/it]
                                                 
{'train/ce_loss': 2.53125, 'train/diffusion_loss': 0.37361788749694824, 'epoch': 2.88}

 29%|██▉       | 210/730 [18:42<46:01,  5.31s/it]
                                                 
{'train/learning_rate_real': 2.0897055211134912e-05, 'epoch': 2.88}

 29%|██▉       | 210/730 [18:42<46:01,  5.31s/it]
                                                 
{'debug/num_tok_total': 2366.0, 'debug/num_tok_loss': 1734.0, 'debug/num_lat_total': 2366.0, 'debug/num_lat_loss': 1734.0, 'epoch': 2.88}

 29%|██▉       | 210/730 [18:43<46:01,  5.31s/it]
                                                 
{'train/ce_loss': 2.359375, 'train/diffusion_loss': 0.45766681432724, 'epoch': 2.88}

 29%|██▉       | 210/730 [18:43<46:01,  5.31s/it]
                                                 
{'train/learning_rate_real': 2.0897055211134912e-05, 'epoch': 2.88}

 29%|██▉       | 210/730 [18:43<46:01,  5.31s/it]
 29%|██▉       | 211/730 [18:43<45:33,  5.27s/it]
                                                 
{'debug/num_tok_total': 3123.0, 'debug/num_tok_loss': 1805.0, 'debug/num_lat_total': 3123.0, 'debug/num_lat_loss': 1805.0, 'epoch': 2.89}

 29%|██▉       | 211/730 [18:45<45:33,  5.27s/it]
                                                 
{'train/ce_loss': 2.390625, 'train/diffusion_loss': 0.343945175409317, 'epoch': 2.89}

 29%|██▉       | 211/730 [18:45<45:33,  5.27s/it]
                                                 
{'train/learning_rate_real': 2.0855885436852256e-05, 'epoch': 2.89}

 29%|██▉       | 211/730 [18:45<45:33,  5.27s/it]
                                                 
{'debug/num_tok_total': 2211.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2211.0, 'debug/num_lat_loss': 1794.0, 'epoch': 2.89}

 29%|██▉       | 211/730 [18:46<45:33,  5.27s/it]
                                                 
{'train/ce_loss': 2.53125, 'train/diffusion_loss': 0.5252296924591064, 'epoch': 2.89}

 29%|██▉       | 211/730 [18:46<45:33,  5.27s/it]
                                                 
{'train/learning_rate_real': 2.0855885436852256e-05, 'epoch': 2.89}

 29%|██▉       | 211/730 [18:46<45:33,  5.27s/it]
                                                 
{'debug/num_tok_total': 2873.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2873.0, 'debug/num_lat_loss': 1792.0, 'epoch': 2.89}

 29%|██▉       | 211/730 [18:47<45:33,  5.27s/it]
                                                 
{'train/ce_loss': 2.359375, 'train/diffusion_loss': 0.3848054111003876, 'epoch': 2.89}

 29%|██▉       | 211/730 [18:47<45:33,  5.27s/it]
                                                 
{'train/learning_rate_real': 2.0855885436852256e-05, 'epoch': 2.89}

 29%|██▉       | 211/730 [18:47<45:33,  5.27s/it]
                                                 
{'debug/num_tok_total': 2198.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2198.0, 'debug/num_lat_loss': 1780.0, 'epoch': 2.89}

 29%|██▉       | 211/730 [18:48<45:33,  5.27s/it]
                                                 
{'train/ce_loss': 2.453125, 'train/diffusion_loss': 0.49421730637550354, 'epoch': 2.89}

 29%|██▉       | 211/730 [18:48<45:33,  5.27s/it]
                                                 
{'train/learning_rate_real': 2.0855885436852256e-05, 'epoch': 2.89}

 29%|██▉       | 211/730 [18:48<45:33,  5.27s/it]
 29%|██▉       | 212/730 [18:49<45:37,  5.29s/it]
                                                 
{'debug/num_tok_total': 2587.0, 'debug/num_tok_loss': 1744.0, 'debug/num_lat_total': 2587.0, 'debug/num_lat_loss': 1744.0, 'epoch': 2.9}

 29%|██▉       | 212/730 [18:50<45:37,  5.29s/it]
                                                 
{'train/ce_loss': 2.4375, 'train/diffusion_loss': 0.40978991985321045, 'epoch': 2.9}

 29%|██▉       | 212/730 [18:50<45:37,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.0814551140185452e-05, 'epoch': 2.9}

 29%|██▉       | 212/730 [18:50<45:37,  5.29s/it]
                                                 
{'debug/num_tok_total': 3320.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 3320.0, 'debug/num_lat_loss': 1795.0, 'epoch': 2.9}

 29%|██▉       | 212/730 [18:51<45:37,  5.29s/it]
                                                 
{'train/ce_loss': 2.453125, 'train/diffusion_loss': 0.2829601466655731, 'epoch': 2.9}

 29%|██▉       | 212/730 [18:51<45:37,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.0814551140185452e-05, 'epoch': 2.9}

 29%|██▉       | 212/730 [18:51<45:37,  5.29s/it]
                                                 
{'debug/num_tok_total': 2921.0, 'debug/num_tok_loss': 1819.0, 'debug/num_lat_total': 2921.0, 'debug/num_lat_loss': 1819.0, 'epoch': 2.9}

 29%|██▉       | 212/730 [18:53<45:37,  5.29s/it]
                                                 
{'train/ce_loss': 2.4375, 'train/diffusion_loss': 0.4063016176223755, 'epoch': 2.9}

 29%|██▉       | 212/730 [18:53<45:37,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.0814551140185452e-05, 'epoch': 2.9}

 29%|██▉       | 212/730 [18:53<45:37,  5.29s/it]
                                                 
{'debug/num_tok_total': 1976.0, 'debug/num_tok_loss': 1653.0, 'debug/num_lat_total': 1976.0, 'debug/num_lat_loss': 1653.0, 'epoch': 2.9}

 29%|██▉       | 212/730 [18:54<45:37,  5.29s/it]
                                                 
{'train/ce_loss': 2.484375, 'train/diffusion_loss': 0.48956847190856934, 'epoch': 2.9}

 29%|██▉       | 212/730 [18:54<45:37,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.0814551140185452e-05, 'epoch': 2.9}

 29%|██▉       | 212/730 [18:54<45:37,  5.29s/it]
 29%|██▉       | 213/730 [18:54<46:01,  5.34s/it]
                                                 
{'debug/num_tok_total': 2225.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2225.0, 'debug/num_lat_loss': 1783.0, 'epoch': 2.92}

 29%|██▉       | 213/730 [18:55<46:01,  5.34s/it]
                                                 
{'train/ce_loss': 2.34375, 'train/diffusion_loss': 0.49811357259750366, 'epoch': 2.92}

 29%|██▉       | 213/730 [18:55<46:01,  5.34s/it]
                                                 
{'train/learning_rate_real': 2.0773053134982063e-05, 'epoch': 2.92}

 29%|██▉       | 213/730 [18:55<46:01,  5.34s/it]
                                                 
{'debug/num_tok_total': 2644.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2644.0, 'debug/num_lat_loss': 1783.0, 'epoch': 2.92}

 29%|██▉       | 213/730 [18:57<46:01,  5.34s/it]
                                                 
{'train/ce_loss': 2.53125, 'train/diffusion_loss': 0.43500933051109314, 'epoch': 2.92}

 29%|██▉       | 213/730 [18:57<46:01,  5.34s/it]
                                                 
{'train/learning_rate_real': 2.0773053134982063e-05, 'epoch': 2.92}

 29%|██▉       | 213/730 [18:57<46:01,  5.34s/it]
                                                 
{'debug/num_tok_total': 3028.0, 'debug/num_tok_loss': 1544.0, 'debug/num_lat_total': 3028.0, 'debug/num_lat_loss': 1544.0, 'epoch': 2.92}

 29%|██▉       | 213/730 [18:58<46:01,  5.34s/it]
                                                 
{'train/ce_loss': 2.3125, 'train/diffusion_loss': 0.23581035435199738, 'epoch': 2.92}

 29%|██▉       | 213/730 [18:58<46:01,  5.34s/it]
                                                 
{'train/learning_rate_real': 2.0773053134982063e-05, 'epoch': 2.92}

 29%|██▉       | 213/730 [18:58<46:01,  5.34s/it]
                                                 
{'debug/num_tok_total': 2457.0, 'debug/num_tok_loss': 1698.0, 'debug/num_lat_total': 2457.0, 'debug/num_lat_loss': 1698.0, 'epoch': 2.92}

 29%|██▉       | 213/730 [18:59<46:01,  5.34s/it]
                                                 
{'train/ce_loss': 2.4375, 'train/diffusion_loss': 0.423569917678833, 'epoch': 2.92}

 29%|██▉       | 213/730 [18:59<46:01,  5.34s/it]
                                                 
{'train/learning_rate_real': 2.0773053134982063e-05, 'epoch': 2.92}

 29%|██▉       | 213/730 [18:59<46:01,  5.34s/it]
 29%|██▉       | 214/730 [19:00<46:23,  5.39s/it]
                                                 
{'debug/num_tok_total': 2201.0, 'debug/num_tok_loss': 1762.0, 'debug/num_lat_total': 2201.0, 'debug/num_lat_loss': 1762.0, 'epoch': 2.93}

 29%|██▉       | 214/730 [19:01<46:23,  5.39s/it]
                                                 
{'train/ce_loss': 2.53125, 'train/diffusion_loss': 0.514683723449707, 'epoch': 2.93}

 29%|██▉       | 214/730 [19:01<46:23,  5.39s/it]
                                                 
{'train/learning_rate_real': 2.0731392238312985e-05, 'epoch': 2.93}

 29%|██▉       | 214/730 [19:01<46:23,  5.39s/it]
                                                 
{'debug/num_tok_total': 2424.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2424.0, 'debug/num_lat_loss': 1781.0, 'epoch': 2.93}

 29%|██▉       | 214/730 [19:02<46:23,  5.39s/it]
                                                 
{'train/ce_loss': 2.609375, 'train/diffusion_loss': 0.4812524616718292, 'epoch': 2.93}

 29%|██▉       | 214/730 [19:02<46:23,  5.39s/it]
                                                 
{'train/learning_rate_real': 2.0731392238312985e-05, 'epoch': 2.93}

 29%|██▉       | 214/730 [19:02<46:23,  5.39s/it]
                                                 
{'debug/num_tok_total': 1998.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 1998.0, 'debug/num_lat_loss': 1788.0, 'epoch': 2.93}

 29%|██▉       | 214/730 [19:03<46:23,  5.39s/it]
                                                 
{'train/ce_loss': 2.5, 'train/diffusion_loss': 0.5399637222290039, 'epoch': 2.93}

 29%|██▉       | 214/730 [19:03<46:23,  5.39s/it]
                                                 
{'train/learning_rate_real': 2.0731392238312985e-05, 'epoch': 2.93}

 29%|██▉       | 214/730 [19:03<46:23,  5.39s/it]
                                                 
{'debug/num_tok_total': 2877.0, 'debug/num_tok_loss': 1683.0, 'debug/num_lat_total': 2877.0, 'debug/num_lat_loss': 1683.0, 'epoch': 2.93}

 29%|██▉       | 214/730 [19:04<46:23,  5.39s/it]
                                                 
{'train/ce_loss': 2.484375, 'train/diffusion_loss': 0.3324914872646332, 'epoch': 2.93}

 29%|██▉       | 214/730 [19:04<46:23,  5.39s/it]
                                                 
{'train/learning_rate_real': 2.0731392238312985e-05, 'epoch': 2.93}

 29%|██▉       | 214/730 [19:04<46:23,  5.39s/it]
 29%|██▉       | 215/730 [19:05<45:30,  5.30s/it]
                                                 
{'debug/num_tok_total': 2618.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2618.0, 'debug/num_lat_loss': 1771.0, 'epoch': 2.95}

 29%|██▉       | 215/730 [19:06<45:30,  5.30s/it]
                                                 
{'train/ce_loss': 2.390625, 'train/diffusion_loss': 0.4014064073562622, 'epoch': 2.95}

 29%|██▉       | 215/730 [19:06<45:30,  5.30s/it]
                                                 
{'train/learning_rate_real': 2.0689569270456337e-05, 'epoch': 2.95}

 29%|██▉       | 215/730 [19:06<45:30,  5.30s/it]
                                                 
{'debug/num_tok_total': 2896.0, 'debug/num_tok_loss': 1804.0, 'debug/num_lat_total': 2896.0, 'debug/num_lat_loss': 1804.0, 'epoch': 2.95}

 29%|██▉       | 215/730 [19:07<45:30,  5.30s/it]
                                                 
{'train/ce_loss': 2.59375, 'train/diffusion_loss': 0.37228548526763916, 'epoch': 2.95}

 29%|██▉       | 215/730 [19:07<45:30,  5.30s/it]
                                                 
{'train/learning_rate_real': 2.0689569270456337e-05, 'epoch': 2.95}

 29%|██▉       | 215/730 [19:07<45:30,  5.30s/it]
                                                 
{'debug/num_tok_total': 2388.0, 'debug/num_tok_loss': 1748.0, 'debug/num_lat_total': 2388.0, 'debug/num_lat_loss': 1748.0, 'epoch': 2.95}

 29%|██▉       | 215/730 [19:09<45:30,  5.30s/it]
                                                 
{'train/ce_loss': 2.40625, 'train/diffusion_loss': 0.4462081789970398, 'epoch': 2.95}

 29%|██▉       | 215/730 [19:09<45:30,  5.30s/it]
                                                 
{'train/learning_rate_real': 2.0689569270456337e-05, 'epoch': 2.95}

 29%|██▉       | 215/730 [19:09<45:30,  5.30s/it]
                                                 
{'debug/num_tok_total': 2409.0, 'debug/num_tok_loss': 1757.0, 'debug/num_lat_total': 2409.0, 'debug/num_lat_loss': 1757.0, 'epoch': 2.95}

 29%|██▉       | 215/730 [19:10<45:30,  5.30s/it]
                                                 
{'train/ce_loss': 2.390625, 'train/diffusion_loss': 0.4493180215358734, 'epoch': 2.95}

 29%|██▉       | 215/730 [19:10<45:30,  5.30s/it]
                                                 
{'train/learning_rate_real': 2.0689569270456337e-05, 'epoch': 2.95}

 29%|██▉       | 215/730 [19:10<45:30,  5.30s/it]
 30%|██▉       | 216/730 [19:10<45:16,  5.29s/it]
                                                 
{'debug/num_tok_total': 2334.0, 'debug/num_tok_loss': 1700.0, 'debug/num_lat_total': 2334.0, 'debug/num_lat_loss': 1700.0, 'epoch': 2.96}

 30%|██▉       | 216/730 [19:11<45:16,  5.29s/it]
                                                 
{'train/ce_loss': 2.421875, 'train/diffusion_loss': 0.4480825364589691, 'epoch': 2.96}

 30%|██▉       | 216/730 [19:11<45:16,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.0647585054881343e-05, 'epoch': 2.96}

 30%|██▉       | 216/730 [19:11<45:16,  5.29s/it]
                                                 
{'debug/num_tok_total': 2465.0, 'debug/num_tok_loss': 1813.0, 'debug/num_lat_total': 2465.0, 'debug/num_lat_loss': 1813.0, 'epoch': 2.96}

 30%|██▉       | 216/730 [19:12<45:16,  5.29s/it]
                                                 
{'train/ce_loss': 2.40625, 'train/diffusion_loss': 0.4440514147281647, 'epoch': 2.96}

 30%|██▉       | 216/730 [19:12<45:16,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.0647585054881343e-05, 'epoch': 2.96}

 30%|██▉       | 216/730 [19:12<45:16,  5.29s/it]
                                                 
{'debug/num_tok_total': 2321.0, 'debug/num_tok_loss': 1624.0, 'debug/num_lat_total': 2321.0, 'debug/num_lat_loss': 1624.0, 'epoch': 2.96}

 30%|██▉       | 216/730 [19:14<45:16,  5.29s/it]
                                                 
{'train/ce_loss': 2.421875, 'train/diffusion_loss': 0.4189290702342987, 'epoch': 2.96}

 30%|██▉       | 216/730 [19:14<45:16,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.0647585054881343e-05, 'epoch': 2.96}

 30%|██▉       | 216/730 [19:14<45:16,  5.29s/it]
                                                 
{'debug/num_tok_total': 3103.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 3103.0, 'debug/num_lat_loss': 1790.0, 'epoch': 2.96}

 30%|██▉       | 216/730 [19:15<45:16,  5.29s/it]
                                                 
{'train/ce_loss': 2.484375, 'train/diffusion_loss': 0.339295357465744, 'epoch': 2.96}

 30%|██▉       | 216/730 [19:15<45:16,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.0647585054881343e-05, 'epoch': 2.96}

 30%|██▉       | 216/730 [19:15<45:16,  5.29s/it]
 30%|██▉       | 217/730 [19:15<45:13,  5.29s/it]
                                                 
{'debug/num_tok_total': 2631.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2631.0, 'debug/num_lat_loss': 1780.0, 'epoch': 2.97}

 30%|██▉       | 217/730 [19:17<45:13,  5.29s/it]
                                                 
{'train/ce_loss': 2.5625, 'train/diffusion_loss': 0.4040837287902832, 'epoch': 2.97}

 30%|██▉       | 217/730 [19:17<45:13,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.0605440418232066e-05, 'epoch': 2.97}

 30%|██▉       | 217/730 [19:17<45:13,  5.29s/it]
                                                 
{'debug/num_tok_total': 2672.0, 'debug/num_tok_loss': 1579.0, 'debug/num_lat_total': 2672.0, 'debug/num_lat_loss': 1579.0, 'epoch': 2.97}

 30%|██▉       | 217/730 [19:18<45:13,  5.29s/it]
                                                 
{'train/ce_loss': 2.234375, 'train/diffusion_loss': 0.3396430015563965, 'epoch': 2.97}

 30%|██▉       | 217/730 [19:18<45:13,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.0605440418232066e-05, 'epoch': 2.97}

 30%|██▉       | 217/730 [19:18<45:13,  5.29s/it]
                                                 
{'debug/num_tok_total': 2432.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2432.0, 'debug/num_lat_loss': 1782.0, 'epoch': 2.97}

 30%|██▉       | 217/730 [19:19<45:13,  5.29s/it]
                                                 
{'train/ce_loss': 2.3125, 'train/diffusion_loss': 0.4572865664958954, 'epoch': 2.97}

 30%|██▉       | 217/730 [19:19<45:13,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.0605440418232066e-05, 'epoch': 2.97}

 30%|██▉       | 217/730 [19:19<45:13,  5.29s/it]
                                                 
{'debug/num_tok_total': 2584.0, 'debug/num_tok_loss': 1648.0, 'debug/num_lat_total': 2584.0, 'debug/num_lat_loss': 1648.0, 'epoch': 2.97}

 30%|██▉       | 217/730 [19:20<45:13,  5.29s/it]
                                                 
{'train/ce_loss': 2.4375, 'train/diffusion_loss': 0.3818599283695221, 'epoch': 2.97}

 30%|██▉       | 217/730 [19:20<45:13,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.0605440418232066e-05, 'epoch': 2.97}

 30%|██▉       | 217/730 [19:20<45:13,  5.29s/it]
 30%|██▉       | 218/730 [19:21<45:01,  5.28s/it]
                                                 
{'debug/num_tok_total': 2894.0, 'debug/num_tok_loss': 1808.0, 'debug/num_lat_total': 2894.0, 'debug/num_lat_loss': 1808.0, 'epoch': 2.99}

 30%|██▉       | 218/730 [19:22<45:01,  5.28s/it]
                                                 
{'train/ce_loss': 2.359375, 'train/diffusion_loss': 0.35479843616485596, 'epoch': 2.99}

 30%|██▉       | 218/730 [19:22<45:01,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.0563136190311195e-05, 'epoch': 2.99}

 30%|██▉       | 218/730 [19:22<45:01,  5.28s/it]
                                                 
{'debug/num_tok_total': 1839.0, 'debug/num_tok_loss': 1386.0, 'debug/num_lat_total': 1839.0, 'debug/num_lat_loss': 1386.0, 'epoch': 2.99}

 30%|██▉       | 218/730 [19:23<45:01,  5.28s/it]
                                                 
{'train/ce_loss': 2.328125, 'train/diffusion_loss': 0.4515104293823242, 'epoch': 2.99}

 30%|██▉       | 218/730 [19:23<45:01,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.0563136190311195e-05, 'epoch': 2.99}

 30%|██▉       | 218/730 [19:23<45:01,  5.28s/it]
                                                 
{'debug/num_tok_total': 2863.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2863.0, 'debug/num_lat_loss': 1790.0, 'epoch': 2.99}

 30%|██▉       | 218/730 [19:24<45:01,  5.28s/it]
                                                 
{'train/ce_loss': 2.375, 'train/diffusion_loss': 0.37860241532325745, 'epoch': 2.99}

 30%|██▉       | 218/730 [19:24<45:01,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.0563136190311195e-05, 'epoch': 2.99}

 30%|██▉       | 218/730 [19:24<45:01,  5.28s/it]
                                                 
{'debug/num_tok_total': 864.0, 'debug/num_tok_loss': 440.0, 'debug/num_lat_total': 864.0, 'debug/num_lat_loss': 440.0, 'epoch': 2.99}

 30%|██▉       | 218/730 [19:25<45:01,  5.28s/it]
                                                 
{'train/ce_loss': 2.3125, 'train/diffusion_loss': 0.2369404137134552, 'epoch': 2.99}

 30%|██▉       | 218/730 [19:25<45:01,  5.28s/it]
                                                 
{'train/learning_rate_real': 2.0563136190311195e-05, 'epoch': 2.99}

 30%|██▉       | 218/730 [19:25<45:01,  5.28s/it]
 30%|███       | 219/730 [19:25<42:41,  5.01s/it]
                                                 
{'debug/num_tok_total': 3075.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 3075.0, 'debug/num_lat_loss': 1774.0, 'epoch': 3.0}

 30%|███       | 219/730 [19:26<42:41,  5.01s/it]
                                                 
{'train/ce_loss': 2.375, 'train/diffusion_loss': 0.2987498342990875, 'epoch': 3.0}

 30%|███       | 219/730 [19:26<42:41,  5.01s/it]
                                                 
{'train/learning_rate_real': 2.0520673204063662e-05, 'epoch': 3.0}

 30%|███       | 219/730 [19:26<42:41,  5.01s/it]
                                                 
{'debug/num_tok_total': 2680.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2680.0, 'debug/num_lat_loss': 1796.0, 'epoch': 3.0}

 30%|███       | 219/730 [19:28<42:41,  5.01s/it]
                                                 
{'train/ce_loss': 2.484375, 'train/diffusion_loss': 0.39434877038002014, 'epoch': 3.0}

 30%|███       | 219/730 [19:28<42:41,  5.01s/it]
                                                 
{'train/learning_rate_real': 2.0520673204063662e-05, 'epoch': 3.0}

 30%|███       | 219/730 [19:28<42:41,  5.01s/it]
                                                 
{'debug/num_tok_total': 2230.0, 'debug/num_tok_loss': 1591.0, 'debug/num_lat_total': 2230.0, 'debug/num_lat_loss': 1591.0, 'epoch': 3.0}

 30%|███       | 219/730 [19:29<42:41,  5.01s/it]
                                                 
{'train/ce_loss': 2.375, 'train/diffusion_loss': 0.462157279253006, 'epoch': 3.0}

 30%|███       | 219/730 [19:29<42:41,  5.01s/it]
                                                 
{'train/learning_rate_real': 2.0520673204063662e-05, 'epoch': 3.0}

 30%|███       | 219/730 [19:29<42:41,  5.01s/it]
                                                 
{'debug/num_tok_total': 2439.0, 'debug/num_tok_loss': 1800.0, 'debug/num_lat_total': 2439.0, 'debug/num_lat_loss': 1800.0, 'epoch': 3.0}

 30%|███       | 219/730 [19:30<42:41,  5.01s/it]
                                                 
{'train/ce_loss': 2.359375, 'train/diffusion_loss': 0.46685951948165894, 'epoch': 3.0}

 30%|███       | 219/730 [19:30<42:41,  5.01s/it]
                                                 
{'train/learning_rate_real': 2.0520673204063662e-05, 'epoch': 3.0}

 30%|███       | 219/730 [19:30<42:41,  5.01s/it]03/16/2026 07:07:03 - INFO - __main__ - LoRA debug step 220: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 30%|███       | 220/730 [19:31<43:47,  5.15s/it]
                                                 
{'loss': 2.7037, 'grad_norm': 1.6216073036193848, 'learning_rate': 2.0520673204063662e-05, 'epoch': 3.01}

 30%|███       | 220/730 [19:31<43:47,  5.15s/it]
                                                 
{'debug/num_tok_total': 2494.0, 'debug/num_tok_loss': 1633.0, 'debug/num_lat_total': 2494.0, 'debug/num_lat_loss': 1633.0, 'epoch': 3.01}

 30%|███       | 220/730 [19:32<43:47,  5.15s/it]
                                                 
{'train/ce_loss': 2.453125, 'train/diffusion_loss': 0.4172014892101288, 'epoch': 3.01}

 30%|███       | 220/730 [19:32<43:47,  5.15s/it]
                                                 
{'train/learning_rate_real': 2.0478052295560253e-05, 'epoch': 3.01}

 30%|███       | 220/730 [19:32<43:47,  5.15s/it]
                                                 
{'debug/num_tok_total': 2664.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2664.0, 'debug/num_lat_loss': 1786.0, 'epoch': 3.01}

 30%|███       | 220/730 [19:33<43:47,  5.15s/it]
                                                 
{'train/ce_loss': 2.265625, 'train/diffusion_loss': 0.39119380712509155, 'epoch': 3.01}

 30%|███       | 220/730 [19:33<43:47,  5.15s/it]
                                                 
{'train/learning_rate_real': 2.0478052295560253e-05, 'epoch': 3.01}

 30%|███       | 220/730 [19:33<43:47,  5.15s/it]
                                                 
{'debug/num_tok_total': 2433.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2433.0, 'debug/num_lat_loss': 1794.0, 'epoch': 3.01}

 30%|███       | 220/730 [19:34<43:47,  5.15s/it]
                                                 
{'train/ce_loss': 2.375, 'train/diffusion_loss': 0.4629015624523163, 'epoch': 3.01}

 30%|███       | 220/730 [19:34<43:47,  5.15s/it]
                                                 
{'train/learning_rate_real': 2.0478052295560253e-05, 'epoch': 3.01}

 30%|███       | 220/730 [19:34<43:47,  5.15s/it]
                                                 
{'debug/num_tok_total': 2620.0, 'debug/num_tok_loss': 1753.0, 'debug/num_lat_total': 2620.0, 'debug/num_lat_loss': 1753.0, 'epoch': 3.01}

 30%|███       | 220/730 [19:35<43:47,  5.15s/it]
                                                 
{'train/ce_loss': 2.4375, 'train/diffusion_loss': 0.40112146735191345, 'epoch': 3.01}

 30%|███       | 220/730 [19:35<43:47,  5.15s/it]
                                                 
{'train/learning_rate_real': 2.0478052295560253e-05, 'epoch': 3.01}

 30%|███       | 220/730 [19:35<43:47,  5.15s/it]
 30%|███       | 221/730 [19:36<43:57,  5.18s/it]
                                                 
{'debug/num_tok_total': 2418.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2418.0, 'debug/num_lat_loss': 1783.0, 'epoch': 3.03}

 30%|███       | 221/730 [19:37<43:57,  5.18s/it]
                                                 
{'train/ce_loss': 2.390625, 'train/diffusion_loss': 0.4611798822879791, 'epoch': 3.03}

 30%|███       | 221/730 [19:37<43:57,  5.18s/it]
                                                 
{'train/learning_rate_real': 2.0435274303981154e-05, 'epoch': 3.03}

 30%|███       | 221/730 [19:37<43:57,  5.18s/it]
                                                 
{'debug/num_tok_total': 3345.0, 'debug/num_tok_loss': 1812.0, 'debug/num_lat_total': 3345.0, 'debug/num_lat_loss': 1812.0, 'epoch': 3.03}

 30%|███       | 221/730 [19:38<43:57,  5.18s/it]
                                                 
{'train/ce_loss': 2.578125, 'train/diffusion_loss': 0.2868610918521881, 'epoch': 3.03}

 30%|███       | 221/730 [19:38<43:57,  5.18s/it]
                                                 
{'train/learning_rate_real': 2.0435274303981154e-05, 'epoch': 3.03}

 30%|███       | 221/730 [19:38<43:57,  5.18s/it]
                                                 
{'debug/num_tok_total': 2405.0, 'debug/num_tok_loss': 1762.0, 'debug/num_lat_total': 2405.0, 'debug/num_lat_loss': 1762.0, 'epoch': 3.03}

 30%|███       | 221/730 [19:40<43:57,  5.18s/it]
                                                 
{'train/ce_loss': 2.453125, 'train/diffusion_loss': 0.4683093726634979, 'epoch': 3.03}

 30%|███       | 221/730 [19:40<43:57,  5.18s/it]
                                                 
{'train/learning_rate_real': 2.0435274303981154e-05, 'epoch': 3.03}

 30%|███       | 221/730 [19:40<43:57,  5.18s/it]
                                                 
{'debug/num_tok_total': 3055.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 3055.0, 'debug/num_lat_loss': 1768.0, 'epoch': 3.03}

 30%|███       | 221/730 [19:41<43:57,  5.18s/it]
                                                 
{'train/ce_loss': 2.515625, 'train/diffusion_loss': 0.3245576322078705, 'epoch': 3.03}

 30%|███       | 221/730 [19:41<43:57,  5.18s/it]
                                                 
{'train/learning_rate_real': 2.0435274303981154e-05, 'epoch': 3.03}

 30%|███       | 221/730 [19:41<43:57,  5.18s/it]
 30%|███       | 222/730 [19:41<44:45,  5.29s/it]
                                                 
{'debug/num_tok_total': 2910.0, 'debug/num_tok_loss': 1693.0, 'debug/num_lat_total': 2910.0, 'debug/num_lat_loss': 1693.0, 'epoch': 3.04}

 30%|███       | 222/730 [19:42<44:45,  5.29s/it]
                                                 
{'train/ce_loss': 2.328125, 'train/diffusion_loss': 0.32742902636528015, 'epoch': 3.04}

 30%|███       | 222/730 [19:42<44:45,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.0392340071599418e-05, 'epoch': 3.04}

 30%|███       | 222/730 [19:42<44:45,  5.29s/it]
                                                 
{'debug/num_tok_total': 2383.0, 'debug/num_tok_loss': 1656.0, 'debug/num_lat_total': 2383.0, 'debug/num_lat_loss': 1656.0, 'epoch': 3.04}

 30%|███       | 222/730 [19:44<44:45,  5.29s/it]
                                                 
{'train/ce_loss': 2.515625, 'train/diffusion_loss': 0.4179707467556, 'epoch': 3.04}

 30%|███       | 222/730 [19:44<44:45,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.0392340071599418e-05, 'epoch': 3.04}

 30%|███       | 222/730 [19:44<44:45,  5.29s/it]
                                                 
{'debug/num_tok_total': 2634.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2634.0, 'debug/num_lat_loss': 1770.0, 'epoch': 3.04}

 30%|███       | 222/730 [19:45<44:45,  5.29s/it]
                                                 
{'train/ce_loss': 2.46875, 'train/diffusion_loss': 0.40202271938323975, 'epoch': 3.04}

 30%|███       | 222/730 [19:45<44:45,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.0392340071599418e-05, 'epoch': 3.04}

 30%|███       | 222/730 [19:45<44:45,  5.29s/it]
                                                 
{'debug/num_tok_total': 2481.0, 'debug/num_tok_loss': 1826.0, 'debug/num_lat_total': 2481.0, 'debug/num_lat_loss': 1826.0, 'epoch': 3.04}

 30%|███       | 222/730 [19:46<44:45,  5.29s/it]
                                                 
{'train/ce_loss': 2.640625, 'train/diffusion_loss': 0.4611271023750305, 'epoch': 3.04}

 30%|███       | 222/730 [19:46<44:45,  5.29s/it]
                                                 
{'train/learning_rate_real': 2.0392340071599418e-05, 'epoch': 3.04}

 30%|███       | 222/730 [19:46<44:45,  5.29s/it]
 31%|███       | 223/730 [19:47<44:30,  5.27s/it]
                                                 
{'debug/num_tok_total': 2004.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2004.0, 'debug/num_lat_loss': 1782.0, 'epoch': 3.05}

 31%|███       | 223/730 [19:48<44:30,  5.27s/it]
                                                 
{'train/ce_loss': 2.328125, 'train/diffusion_loss': 0.5542777180671692, 'epoch': 3.05}

 31%|███       | 223/730 [19:48<44:30,  5.27s/it]
                                                 
{'train/learning_rate_real': 2.03492504437644e-05, 'epoch': 3.05}

 31%|███       | 223/730 [19:48<44:30,  5.27s/it]
                                                 
{'debug/num_tok_total': 2015.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 2015.0, 'debug/num_lat_loss': 1797.0, 'epoch': 3.05}

 31%|███       | 223/730 [19:49<44:30,  5.27s/it]
                                                 
{'train/ce_loss': 2.484375, 'train/diffusion_loss': 0.5250970125198364, 'epoch': 3.05}

 31%|███       | 223/730 [19:49<44:30,  5.27s/it]
                                                 
{'train/learning_rate_real': 2.03492504437644e-05, 'epoch': 3.05}

 31%|███       | 223/730 [19:49<44:30,  5.27s/it]
                                                 
{'debug/num_tok_total': 1998.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 1998.0, 'debug/num_lat_loss': 1788.0, 'epoch': 3.05}

 31%|███       | 223/730 [19:50<44:30,  5.27s/it]
                                                 
{'train/ce_loss': 2.3125, 'train/diffusion_loss': 0.5293837785720825, 'epoch': 3.05}

 31%|███       | 223/730 [19:50<44:30,  5.27s/it]
                                                 
{'train/learning_rate_real': 2.03492504437644e-05, 'epoch': 3.05}

 31%|███       | 223/730 [19:50<44:30,  5.27s/it]
                                                 
{'debug/num_tok_total': 2439.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2439.0, 'debug/num_lat_loss': 1791.0, 'epoch': 3.05}

 31%|███       | 223/730 [19:51<44:30,  5.27s/it]
                                                 
{'train/ce_loss': 2.359375, 'train/diffusion_loss': 0.43268001079559326, 'epoch': 3.05}

 31%|███       | 223/730 [19:51<44:30,  5.27s/it]
                                                 
{'train/learning_rate_real': 2.03492504437644e-05, 'epoch': 3.05}

 31%|███       | 223/730 [19:51<44:30,  5.27s/it]
 31%|███       | 224/730 [19:51<42:57,  5.09s/it]
                                                 
{'debug/num_tok_total': 2434.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2434.0, 'debug/num_lat_loss': 1795.0, 'epoch': 3.07}

 31%|███       | 224/730 [19:52<42:57,  5.09s/it]
                                                 
{'train/ce_loss': 2.453125, 'train/diffusion_loss': 0.4283905625343323, 'epoch': 3.07}

 31%|███       | 224/730 [19:52<42:57,  5.09s/it]
                                                 
{'train/learning_rate_real': 2.0306006268885074e-05, 'epoch': 3.07}

 31%|███       | 224/730 [19:52<42:57,  5.09s/it]
                                                 
{'debug/num_tok_total': 2432.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2432.0, 'debug/num_lat_loss': 1786.0, 'epoch': 3.07}

 31%|███       | 224/730 [19:53<42:57,  5.09s/it]
                                                 
{'train/ce_loss': 2.359375, 'train/diffusion_loss': 0.4826100170612335, 'epoch': 3.07}

 31%|███       | 224/730 [19:53<42:57,  5.09s/it]
                                                 
{'train/learning_rate_real': 2.0306006268885074e-05, 'epoch': 3.07}

 31%|███       | 224/730 [19:53<42:57,  5.09s/it]
                                                 
{'debug/num_tok_total': 2845.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2845.0, 'debug/num_lat_loss': 1774.0, 'epoch': 3.07}

 31%|███       | 224/730 [19:55<42:57,  5.09s/it]
                                                 
{'train/ce_loss': 2.421875, 'train/diffusion_loss': 0.3821423351764679, 'epoch': 3.07}

 31%|███       | 224/730 [19:55<42:57,  5.09s/it]
                                                 
{'train/learning_rate_real': 2.0306006268885074e-05, 'epoch': 3.07}

 31%|███       | 224/730 [19:55<42:57,  5.09s/it]
                                                 
{'debug/num_tok_total': 2458.0, 'debug/num_tok_loss': 1803.0, 'debug/num_lat_total': 2458.0, 'debug/num_lat_loss': 1803.0, 'epoch': 3.07}

 31%|███       | 224/730 [19:56<42:57,  5.09s/it]
                                                 
{'train/ce_loss': 2.25, 'train/diffusion_loss': 0.4636325240135193, 'epoch': 3.07}

 31%|███       | 224/730 [19:56<42:57,  5.09s/it]
                                                 
{'train/learning_rate_real': 2.0306006268885074e-05, 'epoch': 3.07}

 31%|███       | 224/730 [19:56<42:57,  5.09s/it]
 31%|███       | 225/730 [19:56<42:51,  5.09s/it]
                                                 
{'debug/num_tok_total': 2739.0, 'debug/num_tok_loss': 1724.0, 'debug/num_lat_total': 2739.0, 'debug/num_lat_loss': 1724.0, 'epoch': 3.08}

 31%|███       | 225/730 [19:58<42:51,  5.09s/it]
                                                 
{'train/ce_loss': 2.34375, 'train/diffusion_loss': 0.3534652590751648, 'epoch': 3.08}

 31%|███       | 225/730 [19:58<42:51,  5.09s/it]
                                                 
{'train/learning_rate_real': 2.0262608398413375e-05, 'epoch': 3.08}

 31%|███       | 225/730 [19:58<42:51,  5.09s/it]
                                                 
{'debug/num_tok_total': 2422.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2422.0, 'debug/num_lat_loss': 1781.0, 'epoch': 3.08}

 31%|███       | 225/730 [19:59<42:51,  5.09s/it]
                                                 
{'train/ce_loss': 2.296875, 'train/diffusion_loss': 0.4696789085865021, 'epoch': 3.08}

 31%|███       | 225/730 [19:59<42:51,  5.09s/it]
                                                 
{'train/learning_rate_real': 2.0262608398413375e-05, 'epoch': 3.08}

 31%|███       | 225/730 [19:59<42:51,  5.09s/it]
                                                 
{'debug/num_tok_total': 2639.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2639.0, 'debug/num_lat_loss': 1784.0, 'epoch': 3.08}

 31%|███       | 225/730 [20:00<42:51,  5.09s/it]
                                                 
{'train/ce_loss': 2.296875, 'train/diffusion_loss': 0.4197196960449219, 'epoch': 3.08}

 31%|███       | 225/730 [20:00<42:51,  5.09s/it]
                                                 
{'train/learning_rate_real': 2.0262608398413375e-05, 'epoch': 3.08}

 31%|███       | 225/730 [20:00<42:51,  5.09s/it]
                                                 
{'debug/num_tok_total': 2431.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2431.0, 'debug/num_lat_loss': 1783.0, 'epoch': 3.08}

 31%|███       | 225/730 [20:01<42:51,  5.09s/it]
                                                 
{'train/ce_loss': 2.25, 'train/diffusion_loss': 0.45736926794052124, 'epoch': 3.08}

 31%|███       | 225/730 [20:01<42:51,  5.09s/it]
                                                 
{'train/learning_rate_real': 2.0262608398413375e-05, 'epoch': 3.08}

 31%|███       | 225/730 [20:01<42:51,  5.09s/it]
 31%|███       | 226/730 [20:02<43:11,  5.14s/it]
                                                 
{'debug/num_tok_total': 2853.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2853.0, 'debug/num_lat_loss': 1776.0, 'epoch': 3.1}

 31%|███       | 226/730 [20:03<43:11,  5.14s/it]
                                                 
{'train/ce_loss': 2.4375, 'train/diffusion_loss': 0.34873902797698975, 'epoch': 3.1}

 31%|███       | 226/730 [20:03<43:11,  5.14s/it]
                                                 
{'train/learning_rate_real': 2.02190576868274e-05, 'epoch': 3.1}

 31%|███       | 226/730 [20:03<43:11,  5.14s/it]
                                                 
{'debug/num_tok_total': 2466.0, 'debug/num_tok_loss': 1384.0, 'debug/num_lat_total': 2466.0, 'debug/num_lat_loss': 1384.0, 'epoch': 3.1}

 31%|███       | 226/730 [20:04<43:11,  5.14s/it]
                                                 
{'train/ce_loss': 2.1875, 'train/diffusion_loss': 0.2928074300289154, 'epoch': 3.1}

 31%|███       | 226/730 [20:04<43:11,  5.14s/it]
                                                 
{'train/learning_rate_real': 2.02190576868274e-05, 'epoch': 3.1}

 31%|███       | 226/730 [20:04<43:11,  5.14s/it]
                                                 
{'debug/num_tok_total': 2903.0, 'debug/num_tok_loss': 1808.0, 'debug/num_lat_total': 2903.0, 'debug/num_lat_loss': 1808.0, 'epoch': 3.1}

 31%|███       | 226/730 [20:05<43:11,  5.14s/it]
                                                 
{'train/ce_loss': 2.4375, 'train/diffusion_loss': 0.3605611324310303, 'epoch': 3.1}

 31%|███       | 226/730 [20:05<43:11,  5.14s/it]
                                                 
{'train/learning_rate_real': 2.02190576868274e-05, 'epoch': 3.1}

 31%|███       | 226/730 [20:05<43:11,  5.14s/it]
                                                 
{'debug/num_tok_total': 2245.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 2245.0, 'debug/num_lat_loss': 1801.0, 'epoch': 3.1}

 31%|███       | 226/730 [20:06<43:11,  5.14s/it]
                                                 
{'train/ce_loss': 2.390625, 'train/diffusion_loss': 0.5095975399017334, 'epoch': 3.1}

 31%|███       | 226/730 [20:07<43:11,  5.14s/it]
                                                 
{'train/learning_rate_real': 2.02190576868274e-05, 'epoch': 3.1}

 31%|███       | 226/730 [20:07<43:11,  5.14s/it]
 31%|███       | 227/730 [20:07<43:37,  5.20s/it]
                                                 
{'debug/num_tok_total': 2201.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 2201.0, 'debug/num_lat_loss': 1769.0, 'epoch': 3.11}

 31%|███       | 227/730 [20:08<43:37,  5.20s/it]
                                                 
{'train/ce_loss': 2.453125, 'train/diffusion_loss': 0.47939595580101013, 'epoch': 3.11}

 31%|███       | 227/730 [20:08<43:37,  5.20s/it]
                                                 
{'train/learning_rate_real': 2.01753549916146e-05, 'epoch': 3.11}

 31%|███       | 227/730 [20:08<43:37,  5.20s/it]
                                                 
{'debug/num_tok_total': 2665.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2665.0, 'debug/num_lat_loss': 1794.0, 'epoch': 3.11}

 31%|███       | 227/730 [20:09<43:37,  5.20s/it]
                                                 
{'train/ce_loss': 2.421875, 'train/diffusion_loss': 0.4275529086589813, 'epoch': 3.11}

 31%|███       | 227/730 [20:09<43:37,  5.20s/it]
                                                 
{'train/learning_rate_real': 2.01753549916146e-05, 'epoch': 3.11}

 31%|███       | 227/730 [20:09<43:37,  5.20s/it]
                                                 
{'debug/num_tok_total': 3038.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 3038.0, 'debug/num_lat_loss': 1765.0, 'epoch': 3.11}

 31%|███       | 227/730 [20:10<43:37,  5.20s/it]
                                                 
{'train/ce_loss': 2.296875, 'train/diffusion_loss': 0.30026838183403015, 'epoch': 3.11}

 31%|███       | 227/730 [20:10<43:37,  5.20s/it]
                                                 
{'train/learning_rate_real': 2.01753549916146e-05, 'epoch': 3.11}

 31%|███       | 227/730 [20:10<43:37,  5.20s/it]
                                                 
{'debug/num_tok_total': 2440.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2440.0, 'debug/num_lat_loss': 1792.0, 'epoch': 3.11}

 31%|███       | 227/730 [20:12<43:37,  5.20s/it]
                                                 
{'train/ce_loss': 2.390625, 'train/diffusion_loss': 0.4443340599536896, 'epoch': 3.11}

 31%|███       | 227/730 [20:12<43:37,  5.20s/it]
                                                 
{'train/learning_rate_real': 2.01753549916146e-05, 'epoch': 3.11}

 31%|███       | 227/730 [20:12<43:37,  5.20s/it]
 31%|███       | 228/730 [20:12<43:38,  5.22s/it]
                                                 
{'debug/num_tok_total': 2706.0, 'debug/num_tok_loss': 1601.0, 'debug/num_lat_total': 2706.0, 'debug/num_lat_loss': 1601.0, 'epoch': 3.12}

 31%|███       | 228/730 [20:13<43:38,  5.22s/it]
                                                 
{'train/ce_loss': 2.25, 'train/diffusion_loss': 0.2934297025203705, 'epoch': 3.12}

 31%|███       | 228/730 [20:13<43:38,  5.22s/it]
                                                 
{'train/learning_rate_real': 2.0131501173254895e-05, 'epoch': 3.12}

 31%|███       | 228/730 [20:13<43:38,  5.22s/it]
                                                 
{'debug/num_tok_total': 2850.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2850.0, 'debug/num_lat_loss': 1785.0, 'epoch': 3.12}

 31%|███       | 228/730 [20:15<43:38,  5.22s/it]
                                                 
{'train/ce_loss': 2.28125, 'train/diffusion_loss': 0.3723796010017395, 'epoch': 3.12}

 31%|███       | 228/730 [20:15<43:38,  5.22s/it]
                                                 
{'train/learning_rate_real': 2.0131501173254895e-05, 'epoch': 3.12}

 31%|███       | 228/730 [20:15<43:38,  5.22s/it]
                                                 
{'debug/num_tok_total': 2232.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2232.0, 'debug/num_lat_loss': 1785.0, 'epoch': 3.12}

 31%|███       | 228/730 [20:16<43:38,  5.22s/it]
                                                 
{'train/ce_loss': 2.34375, 'train/diffusion_loss': 0.4982549846172333, 'epoch': 3.12}

 31%|███       | 228/730 [20:16<43:38,  5.22s/it]
                                                 
{'train/learning_rate_real': 2.0131501173254895e-05, 'epoch': 3.12}

 31%|███       | 228/730 [20:16<43:38,  5.22s/it]
                                                 
{'debug/num_tok_total': 2419.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2419.0, 'debug/num_lat_loss': 1772.0, 'epoch': 3.12}

 31%|███       | 228/730 [20:17<43:38,  5.22s/it]
                                                 
{'train/ce_loss': 2.28125, 'train/diffusion_loss': 0.433102548122406, 'epoch': 3.12}

 31%|███       | 228/730 [20:17<43:38,  5.22s/it]
                                                 
{'train/learning_rate_real': 2.0131501173254895e-05, 'epoch': 3.12}

 31%|███       | 228/730 [20:17<43:38,  5.22s/it]
 31%|███▏      | 229/730 [20:17<43:47,  5.24s/it]
                                                 
{'debug/num_tok_total': 2213.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2213.0, 'debug/num_lat_loss': 1791.0, 'epoch': 3.14}

 31%|███▏      | 229/730 [20:19<43:47,  5.24s/it]
                                                 
{'train/ce_loss': 2.109375, 'train/diffusion_loss': 0.48188430070877075, 'epoch': 3.14}

 31%|███▏      | 229/730 [20:19<43:47,  5.24s/it]
                                                 
{'train/learning_rate_real': 2.0087497095203724e-05, 'epoch': 3.14}

 31%|███▏      | 229/730 [20:19<43:47,  5.24s/it]
                                                 
{'debug/num_tok_total': 2152.0, 'debug/num_tok_loss': 1725.0, 'debug/num_lat_total': 2152.0, 'debug/num_lat_loss': 1725.0, 'epoch': 3.14}

 31%|███▏      | 229/730 [20:20<43:47,  5.24s/it]
                                                 
{'train/ce_loss': 2.515625, 'train/diffusion_loss': 0.5034520030021667, 'epoch': 3.14}

 31%|███▏      | 229/730 [20:20<43:47,  5.24s/it]
                                                 
{'train/learning_rate_real': 2.0087497095203724e-05, 'epoch': 3.14}

 31%|███▏      | 229/730 [20:20<43:47,  5.24s/it]
                                                 
{'debug/num_tok_total': 3262.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 3262.0, 'debug/num_lat_loss': 1766.0, 'epoch': 3.14}

 31%|███▏      | 229/730 [20:21<43:47,  5.24s/it]
                                                 
{'train/ce_loss': 2.234375, 'train/diffusion_loss': 0.26973801851272583, 'epoch': 3.14}

 31%|███▏      | 229/730 [20:21<43:47,  5.24s/it]
                                                 
{'train/learning_rate_real': 2.0087497095203724e-05, 'epoch': 3.14}

 31%|███▏      | 229/730 [20:21<43:47,  5.24s/it]
                                                 
{'debug/num_tok_total': 2179.0, 'debug/num_tok_loss': 1756.0, 'debug/num_lat_total': 2179.0, 'debug/num_lat_loss': 1756.0, 'epoch': 3.14}

 31%|███▏      | 229/730 [20:22<43:47,  5.24s/it]
                                                 
{'train/ce_loss': 2.40625, 'train/diffusion_loss': 0.5100567936897278, 'epoch': 3.14}

 31%|███▏      | 229/730 [20:22<43:47,  5.24s/it]
                                                 
{'train/learning_rate_real': 2.0087497095203724e-05, 'epoch': 3.14}

 31%|███▏      | 229/730 [20:22<43:47,  5.24s/it]03/16/2026 07:07:55 - INFO - __main__ - LoRA debug step 230: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 32%|███▏      | 230/730 [20:23<43:30,  5.22s/it]
                                                 
{'loss': 2.7385, 'grad_norm': 1.9824601411819458, 'learning_rate': 2.0087497095203724e-05, 'epoch': 3.15}

 32%|███▏      | 230/730 [20:23<43:30,  5.22s/it]
                                                 
{'debug/num_tok_total': 2430.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2430.0, 'debug/num_lat_loss': 1787.0, 'epoch': 3.15}

 32%|███▏      | 230/730 [20:24<43:30,  5.22s/it]
                                                 
{'train/ce_loss': 2.34375, 'train/diffusion_loss': 0.4837017357349396, 'epoch': 3.15}

 32%|███▏      | 230/730 [20:24<43:30,  5.22s/it]
                                                 
{'train/learning_rate_real': 2.004334362387505e-05, 'epoch': 3.15}

 32%|███▏      | 230/730 [20:24<43:30,  5.22s/it]
                                                 
{'debug/num_tok_total': 2625.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2625.0, 'debug/num_lat_loss': 1774.0, 'epoch': 3.15}

 32%|███▏      | 230/730 [20:25<43:30,  5.22s/it]
                                                 
{'train/ce_loss': 2.390625, 'train/diffusion_loss': 0.3880443274974823, 'epoch': 3.15}

 32%|███▏      | 230/730 [20:25<43:30,  5.22s/it]
                                                 
{'train/learning_rate_real': 2.004334362387505e-05, 'epoch': 3.15}

 32%|███▏      | 230/730 [20:25<43:30,  5.22s/it]
                                                 
{'debug/num_tok_total': 2656.0, 'debug/num_tok_loss': 1798.0, 'debug/num_lat_total': 2656.0, 'debug/num_lat_loss': 1798.0, 'epoch': 3.15}

 32%|███▏      | 230/730 [20:26<43:30,  5.22s/it]
                                                 
{'train/ce_loss': 2.25, 'train/diffusion_loss': 0.40594106912612915, 'epoch': 3.15}

 32%|███▏      | 230/730 [20:26<43:30,  5.22s/it]
                                                 
{'train/learning_rate_real': 2.004334362387505e-05, 'epoch': 3.15}

 32%|███▏      | 230/730 [20:26<43:30,  5.22s/it]
                                                 
{'debug/num_tok_total': 2609.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2609.0, 'debug/num_lat_loss': 1766.0, 'epoch': 3.15}

 32%|███▏      | 230/730 [20:27<43:30,  5.22s/it]
                                                 
{'train/ce_loss': 2.421875, 'train/diffusion_loss': 0.4162349998950958, 'epoch': 3.15}

 32%|███▏      | 230/730 [20:27<43:30,  5.22s/it]
                                                 
{'train/learning_rate_real': 2.004334362387505e-05, 'epoch': 3.15}

 32%|███▏      | 230/730 [20:27<43:30,  5.22s/it]
 32%|███▏      | 231/730 [20:28<43:28,  5.23s/it]
                                                 
{'debug/num_tok_total': 2194.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2194.0, 'debug/num_lat_loss': 1776.0, 'epoch': 3.16}

 32%|███▏      | 231/730 [20:29<43:28,  5.23s/it]
                                                 
{'train/ce_loss': 2.359375, 'train/diffusion_loss': 0.5041800141334534, 'epoch': 3.16}

 32%|███▏      | 231/730 [20:29<43:28,  5.23s/it]
                                                 
{'train/learning_rate_real': 1.9999041628624304e-05, 'epoch': 3.16}

 32%|███▏      | 231/730 [20:29<43:28,  5.23s/it]
                                                 
{'debug/num_tok_total': 2626.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2626.0, 'debug/num_lat_loss': 1764.0, 'epoch': 3.16}

 32%|███▏      | 231/730 [20:30<43:28,  5.23s/it]
                                                 
{'train/ce_loss': 2.34375, 'train/diffusion_loss': 0.4167364835739136, 'epoch': 3.16}

 32%|███▏      | 231/730 [20:30<43:28,  5.23s/it]
                                                 
{'train/learning_rate_real': 1.9999041628624304e-05, 'epoch': 3.16}

 32%|███▏      | 231/730 [20:30<43:28,  5.23s/it]
                                                 
{'debug/num_tok_total': 2878.0, 'debug/num_tok_loss': 1798.0, 'debug/num_lat_total': 2878.0, 'debug/num_lat_loss': 1798.0, 'epoch': 3.16}

 32%|███▏      | 231/730 [20:32<43:28,  5.23s/it]
                                                 
{'train/ce_loss': 2.3125, 'train/diffusion_loss': 0.35796892642974854, 'epoch': 3.16}

 32%|███▏      | 231/730 [20:32<43:28,  5.23s/it]
                                                 
{'train/learning_rate_real': 1.9999041628624304e-05, 'epoch': 3.16}

 32%|███▏      | 231/730 [20:32<43:28,  5.23s/it]
                                                 
{'debug/num_tok_total': 2670.0, 'debug/num_tok_loss': 1803.0, 'debug/num_lat_total': 2670.0, 'debug/num_lat_loss': 1803.0, 'epoch': 3.16}

 32%|███▏      | 231/730 [20:33<43:28,  5.23s/it]
                                                 
{'train/ce_loss': 2.46875, 'train/diffusion_loss': 0.42231622338294983, 'epoch': 3.16}

 32%|███▏      | 231/730 [20:33<43:28,  5.23s/it]
                                                 
{'train/learning_rate_real': 1.9999041628624304e-05, 'epoch': 3.16}

 32%|███▏      | 231/730 [20:33<43:28,  5.23s/it]
 32%|███▏      | 232/730 [20:33<43:44,  5.27s/it]
                                                 
{'debug/num_tok_total': 2884.0, 'debug/num_tok_loss': 1800.0, 'debug/num_lat_total': 2884.0, 'debug/num_lat_loss': 1800.0, 'epoch': 3.18}

 32%|███▏      | 232/730 [20:34<43:44,  5.27s/it]
                                                 
{'train/ce_loss': 2.140625, 'train/diffusion_loss': 0.3421187996864319, 'epoch': 3.18}

 32%|███▏      | 232/730 [20:34<43:44,  5.27s/it]
                                                 
{'train/learning_rate_real': 1.995459198173127e-05, 'epoch': 3.18}

 32%|███▏      | 232/730 [20:34<43:44,  5.27s/it]
                                                 
{'debug/num_tok_total': 2199.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2199.0, 'debug/num_lat_loss': 1780.0, 'epoch': 3.18}

 32%|███▏      | 232/730 [20:36<43:44,  5.27s/it]
                                                 
{'train/ce_loss': 2.34375, 'train/diffusion_loss': 0.4964865744113922, 'epoch': 3.18}

 32%|███▏      | 232/730 [20:36<43:44,  5.27s/it]
                                                 
{'train/learning_rate_real': 1.995459198173127e-05, 'epoch': 3.18}

 32%|███▏      | 232/730 [20:36<43:44,  5.27s/it]
                                                 
{'debug/num_tok_total': 2631.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2631.0, 'debug/num_lat_loss': 1774.0, 'epoch': 3.18}

 32%|███▏      | 232/730 [20:37<43:44,  5.27s/it]
                                                 
{'train/ce_loss': 2.84375, 'train/diffusion_loss': 0.4140637218952179, 'epoch': 3.18}

 32%|███▏      | 232/730 [20:37<43:44,  5.27s/it]
                                                 
{'train/learning_rate_real': 1.995459198173127e-05, 'epoch': 3.18}

 32%|███▏      | 232/730 [20:37<43:44,  5.27s/it]
                                                 
{'debug/num_tok_total': 2224.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2224.0, 'debug/num_lat_loss': 1783.0, 'epoch': 3.18}

 32%|███▏      | 232/730 [20:38<43:44,  5.27s/it]
                                                 
{'train/ce_loss': 2.390625, 'train/diffusion_loss': 0.5294895172119141, 'epoch': 3.18}

 32%|███▏      | 232/730 [20:38<43:44,  5.27s/it]
                                                 
{'train/learning_rate_real': 1.995459198173127e-05, 'epoch': 3.18}

 32%|███▏      | 232/730 [20:38<43:44,  5.27s/it]
 32%|███▏      | 233/730 [20:38<43:06,  5.21s/it]
                                                 
{'debug/num_tok_total': 2461.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 2461.0, 'debug/num_lat_loss': 1801.0, 'epoch': 3.19}

 32%|███▏      | 233/730 [20:39<43:06,  5.21s/it]
                                                 
{'train/ce_loss': 2.3125, 'train/diffusion_loss': 0.4414790868759155, 'epoch': 3.19}

 32%|███▏      | 233/730 [20:39<43:06,  5.21s/it]
                                                 
{'train/learning_rate_real': 1.9909995558382886e-05, 'epoch': 3.19}

 32%|███▏      | 233/730 [20:39<43:06,  5.21s/it]
                                                 
{'debug/num_tok_total': 2615.0, 'debug/num_tok_loss': 1758.0, 'debug/num_lat_total': 2615.0, 'debug/num_lat_loss': 1758.0, 'epoch': 3.19}

 32%|███▏      | 233/730 [20:41<43:06,  5.21s/it]
                                                 
{'train/ce_loss': 2.234375, 'train/diffusion_loss': 0.404286652803421, 'epoch': 3.19}

 32%|███▏      | 233/730 [20:41<43:06,  5.21s/it]
                                                 
{'train/learning_rate_real': 1.9909995558382886e-05, 'epoch': 3.19}

 32%|███▏      | 233/730 [20:41<43:06,  5.21s/it]
                                                 
{'debug/num_tok_total': 2865.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2865.0, 'debug/num_lat_loss': 1787.0, 'epoch': 3.19}

 32%|███▏      | 233/730 [20:42<43:06,  5.21s/it]
                                                 
{'train/ce_loss': 2.375, 'train/diffusion_loss': 0.3791257441043854, 'epoch': 3.19}

 32%|███▏      | 233/730 [20:42<43:06,  5.21s/it]
                                                 
{'train/learning_rate_real': 1.9909995558382886e-05, 'epoch': 3.19}

 32%|███▏      | 233/730 [20:42<43:06,  5.21s/it]
                                                 
{'debug/num_tok_total': 2839.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2839.0, 'debug/num_lat_loss': 1777.0, 'epoch': 3.19}

 32%|███▏      | 233/730 [20:43<43:06,  5.21s/it]
                                                 
{'train/ce_loss': 2.296875, 'train/diffusion_loss': 0.357394814491272, 'epoch': 3.19}

 32%|███▏      | 233/730 [20:43<43:06,  5.21s/it]
                                                 
{'train/learning_rate_real': 1.9909995558382886e-05, 'epoch': 3.19}

 32%|███▏      | 233/730 [20:43<43:06,  5.21s/it]
 32%|███▏      | 234/730 [20:44<43:42,  5.29s/it]
                                                 
{'debug/num_tok_total': 2249.0, 'debug/num_tok_loss': 1622.0, 'debug/num_lat_total': 2249.0, 'debug/num_lat_loss': 1622.0, 'epoch': 3.21}

 32%|███▏      | 234/730 [20:45<43:42,  5.29s/it]
                                                 
{'train/ce_loss': 2.4375, 'train/diffusion_loss': 0.4440324306488037, 'epoch': 3.21}

 32%|███▏      | 234/730 [20:45<43:42,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.9865253236656044e-05, 'epoch': 3.21}

 32%|███▏      | 234/730 [20:45<43:42,  5.29s/it]
                                                 
{'debug/num_tok_total': 3095.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 3095.0, 'debug/num_lat_loss': 1794.0, 'epoch': 3.21}

 32%|███▏      | 234/730 [20:46<43:42,  5.29s/it]
                                                 
{'train/ce_loss': 2.328125, 'train/diffusion_loss': 0.2975127100944519, 'epoch': 3.21}

 32%|███▏      | 234/730 [20:46<43:42,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.9865253236656044e-05, 'epoch': 3.21}

 32%|███▏      | 234/730 [20:46<43:42,  5.29s/it]
                                                 
{'debug/num_tok_total': 2670.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2670.0, 'debug/num_lat_loss': 1790.0, 'epoch': 3.21}

 32%|███▏      | 234/730 [20:47<43:42,  5.29s/it]
                                                 
{'train/ce_loss': 2.25, 'train/diffusion_loss': 0.4166703522205353, 'epoch': 3.21}

 32%|███▏      | 234/730 [20:47<43:42,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.9865253236656044e-05, 'epoch': 3.21}

 32%|███▏      | 234/730 [20:47<43:42,  5.29s/it]
                                                 
{'debug/num_tok_total': 2644.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2644.0, 'debug/num_lat_loss': 1784.0, 'epoch': 3.21}

 32%|███▏      | 234/730 [20:49<43:42,  5.29s/it]
                                                 
{'train/ce_loss': 2.40625, 'train/diffusion_loss': 0.3953518271446228, 'epoch': 3.21}

 32%|███▏      | 234/730 [20:49<43:42,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.9865253236656044e-05, 'epoch': 3.21}

 32%|███▏      | 234/730 [20:49<43:42,  5.29s/it]
 32%|███▏      | 235/730 [20:49<43:45,  5.30s/it]
                                                 
{'debug/num_tok_total': 2202.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2202.0, 'debug/num_lat_loss': 1775.0, 'epoch': 3.22}

 32%|███▏      | 235/730 [20:50<43:45,  5.30s/it]
                                                 
{'train/ce_loss': 2.234375, 'train/diffusion_loss': 0.4948474168777466, 'epoch': 3.22}

 32%|███▏      | 235/730 [20:50<43:45,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.9820365897500294e-05, 'epoch': 3.22}

 32%|███▏      | 235/730 [20:50<43:45,  5.30s/it]
                                                 
{'debug/num_tok_total': 2849.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2849.0, 'debug/num_lat_loss': 1774.0, 'epoch': 3.22}

 32%|███▏      | 235/730 [20:52<43:45,  5.30s/it]
                                                 
{'train/ce_loss': 2.34375, 'train/diffusion_loss': 0.3483217656612396, 'epoch': 3.22}

 32%|███▏      | 235/730 [20:52<43:45,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.9820365897500294e-05, 'epoch': 3.22}

 32%|███▏      | 235/730 [20:52<43:45,  5.30s/it]
                                                 
{'debug/num_tok_total': 2191.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2191.0, 'debug/num_lat_loss': 1763.0, 'epoch': 3.22}

 32%|███▏      | 235/730 [20:53<43:45,  5.30s/it]
                                                 
{'train/ce_loss': 2.359375, 'train/diffusion_loss': 0.535424530506134, 'epoch': 3.22}

 32%|███▏      | 235/730 [20:53<43:45,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.9820365897500294e-05, 'epoch': 3.22}

 32%|███▏      | 235/730 [20:53<43:45,  5.30s/it]
                                                 
{'debug/num_tok_total': 2083.0, 'debug/num_tok_loss': 1715.0, 'debug/num_lat_total': 2083.0, 'debug/num_lat_loss': 1715.0, 'epoch': 3.22}

 32%|███▏      | 235/730 [20:54<43:45,  5.30s/it]
                                                 
{'train/ce_loss': 2.375, 'train/diffusion_loss': 0.5188778042793274, 'epoch': 3.22}

 32%|███▏      | 235/730 [20:54<43:45,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.9820365897500294e-05, 'epoch': 3.22}

 32%|███▏      | 235/730 [20:54<43:45,  5.30s/it]
 32%|███▏      | 236/730 [20:54<43:05,  5.23s/it]
                                                 
{'debug/num_tok_total': 2891.0, 'debug/num_tok_loss': 1806.0, 'debug/num_lat_total': 2891.0, 'debug/num_lat_loss': 1806.0, 'epoch': 3.23}

 32%|███▏      | 236/730 [20:55<43:05,  5.23s/it]
                                                 
{'train/ce_loss': 2.375, 'train/diffusion_loss': 0.34874579310417175, 'epoch': 3.23}

 32%|███▏      | 236/730 [20:55<43:05,  5.23s/it]
                                                 
{'train/learning_rate_real': 1.977533442472047e-05, 'epoch': 3.23}

 32%|███▏      | 236/730 [20:55<43:05,  5.23s/it]
                                                 
{'debug/num_tok_total': 2208.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2208.0, 'debug/num_lat_loss': 1766.0, 'epoch': 3.23}

 32%|███▏      | 236/730 [20:57<43:05,  5.23s/it]
                                                 
{'train/ce_loss': 2.234375, 'train/diffusion_loss': 0.5000365972518921, 'epoch': 3.23}

 32%|███▏      | 236/730 [20:57<43:05,  5.23s/it]
                                                 
{'train/learning_rate_real': 1.977533442472047e-05, 'epoch': 3.23}

 32%|███▏      | 236/730 [20:57<43:05,  5.23s/it]
                                                 
{'debug/num_tok_total': 2935.0, 'debug/num_tok_loss': 1632.0, 'debug/num_lat_total': 2935.0, 'debug/num_lat_loss': 1632.0, 'epoch': 3.23}

 32%|███▏      | 236/730 [20:58<43:05,  5.23s/it]
                                                 
{'train/ce_loss': 2.546875, 'train/diffusion_loss': 0.2920040786266327, 'epoch': 3.23}

 32%|███▏      | 236/730 [20:58<43:05,  5.23s/it]
                                                 
{'train/learning_rate_real': 1.977533442472047e-05, 'epoch': 3.23}

 32%|███▏      | 236/730 [20:58<43:05,  5.23s/it]
                                                 
{'debug/num_tok_total': 2382.0, 'debug/num_tok_loss': 1652.0, 'debug/num_lat_total': 2382.0, 'debug/num_lat_loss': 1652.0, 'epoch': 3.23}

 32%|███▏      | 236/730 [20:59<43:05,  5.23s/it]
                                                 
{'train/ce_loss': 2.40625, 'train/diffusion_loss': 0.392981618642807, 'epoch': 3.23}

 32%|███▏      | 236/730 [20:59<43:05,  5.23s/it]
                                                 
{'train/learning_rate_real': 1.977533442472047e-05, 'epoch': 3.23}

 32%|███▏      | 236/730 [20:59<43:05,  5.23s/it]
 32%|███▏      | 237/730 [21:00<43:48,  5.33s/it]
                                                 
{'debug/num_tok_total': 2626.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2626.0, 'debug/num_lat_loss': 1771.0, 'epoch': 3.25}

 32%|███▏      | 237/730 [21:01<43:48,  5.33s/it]
                                                 
{'train/ce_loss': 2.109375, 'train/diffusion_loss': 0.42288437485694885, 'epoch': 3.25}

 32%|███▏      | 237/730 [21:01<43:48,  5.33s/it]
                                                 
{'train/learning_rate_real': 1.973015970495934e-05, 'epoch': 3.25}

 32%|███▏      | 237/730 [21:01<43:48,  5.33s/it]
                                                 
{'debug/num_tok_total': 2887.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 2887.0, 'debug/num_lat_loss': 1801.0, 'epoch': 3.25}

 32%|███▏      | 237/730 [21:02<43:48,  5.33s/it]
                                                 
{'train/ce_loss': 2.28125, 'train/diffusion_loss': 0.3476720452308655, 'epoch': 3.25}

 32%|███▏      | 237/730 [21:02<43:48,  5.33s/it]
                                                 
{'train/learning_rate_real': 1.973015970495934e-05, 'epoch': 3.25}

 32%|███▏      | 237/730 [21:02<43:48,  5.33s/it]
                                                 
{'debug/num_tok_total': 2652.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2652.0, 'debug/num_lat_loss': 1789.0, 'epoch': 3.25}

 32%|███▏      | 237/730 [21:04<43:48,  5.33s/it]
                                                 
{'train/ce_loss': 2.328125, 'train/diffusion_loss': 0.4026311933994293, 'epoch': 3.25}

 32%|███▏      | 237/730 [21:04<43:48,  5.33s/it]
                                                 
{'train/learning_rate_real': 1.973015970495934e-05, 'epoch': 3.25}

 32%|███▏      | 237/730 [21:04<43:48,  5.33s/it]
                                                 
{'debug/num_tok_total': 2657.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2657.0, 'debug/num_lat_loss': 1788.0, 'epoch': 3.25}

 32%|███▏      | 237/730 [21:05<43:48,  5.33s/it]
                                                 
{'train/ce_loss': 2.265625, 'train/diffusion_loss': 0.39144831895828247, 'epoch': 3.25}

 32%|███▏      | 237/730 [21:05<43:48,  5.33s/it]
                                                 
{'train/learning_rate_real': 1.973015970495934e-05, 'epoch': 3.25}

 32%|███▏      | 237/730 [21:05<43:48,  5.33s/it]
 33%|███▎      | 238/730 [21:05<43:55,  5.36s/it]
                                                 
{'debug/num_tok_total': 2621.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2621.0, 'debug/num_lat_loss': 1764.0, 'epoch': 3.26}

 33%|███▎      | 238/730 [21:06<43:55,  5.36s/it]
                                                 
{'train/ce_loss': 2.453125, 'train/diffusion_loss': 0.3839724361896515, 'epoch': 3.26}

 33%|███▎      | 238/730 [21:06<43:55,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.9684842627680088e-05, 'epoch': 3.26}

 33%|███▎      | 238/730 [21:06<43:55,  5.36s/it]
                                                 
{'debug/num_tok_total': 3045.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 3045.0, 'debug/num_lat_loss': 1767.0, 'epoch': 3.26}

 33%|███▎      | 238/730 [21:08<43:55,  5.36s/it]
                                                 
{'train/ce_loss': 2.328125, 'train/diffusion_loss': 0.31859639286994934, 'epoch': 3.26}

 33%|███▎      | 238/730 [21:08<43:55,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.9684842627680088e-05, 'epoch': 3.26}

 33%|███▎      | 238/730 [21:08<43:55,  5.36s/it]
                                                 
{'debug/num_tok_total': 2437.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2437.0, 'debug/num_lat_loss': 1786.0, 'epoch': 3.26}

 33%|███▎      | 238/730 [21:09<43:55,  5.36s/it]
                                                 
{'train/ce_loss': 2.265625, 'train/diffusion_loss': 0.4462120532989502, 'epoch': 3.26}

 33%|███▎      | 238/730 [21:09<43:55,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.9684842627680088e-05, 'epoch': 3.26}

 33%|███▎      | 238/730 [21:09<43:55,  5.36s/it]
                                                 
{'debug/num_tok_total': 2214.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2214.0, 'debug/num_lat_loss': 1783.0, 'epoch': 3.26}

 33%|███▎      | 238/730 [21:10<43:55,  5.36s/it]
                                                 
{'train/ce_loss': 2.21875, 'train/diffusion_loss': 0.4857374131679535, 'epoch': 3.26}

 33%|███▎      | 238/730 [21:10<43:55,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.9684842627680088e-05, 'epoch': 3.26}

 33%|███▎      | 238/730 [21:10<43:55,  5.36s/it]
 33%|███▎      | 239/730 [21:10<43:30,  5.32s/it]
                                                 
{'debug/num_tok_total': 2263.0, 'debug/num_tok_loss': 1612.0, 'debug/num_lat_total': 2263.0, 'debug/num_lat_loss': 1612.0, 'epoch': 3.27}

 33%|███▎      | 239/730 [21:12<43:30,  5.32s/it]
                                                 
{'train/ce_loss': 2.21875, 'train/diffusion_loss': 0.41768062114715576, 'epoch': 3.27}

 33%|███▎      | 239/730 [21:12<43:30,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.963938408514886e-05, 'epoch': 3.27}

 33%|███▎      | 239/730 [21:12<43:30,  5.32s/it]
                                                 
{'debug/num_tok_total': 2321.0, 'debug/num_tok_loss': 1678.0, 'debug/num_lat_total': 2321.0, 'debug/num_lat_loss': 1678.0, 'epoch': 3.27}

 33%|███▎      | 239/730 [21:13<43:30,  5.32s/it]
                                                 
{'train/ce_loss': 2.1875, 'train/diffusion_loss': 0.4716084599494934, 'epoch': 3.27}

 33%|███▎      | 239/730 [21:13<43:30,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.963938408514886e-05, 'epoch': 3.27}

 33%|███▎      | 239/730 [21:13<43:30,  5.32s/it]
                                                 
{'debug/num_tok_total': 3149.0, 'debug/num_tok_loss': 1821.0, 'debug/num_lat_total': 3149.0, 'debug/num_lat_loss': 1821.0, 'epoch': 3.27}

 33%|███▎      | 239/730 [21:14<43:30,  5.32s/it]
                                                 
{'train/ce_loss': 2.3125, 'train/diffusion_loss': 0.3284153938293457, 'epoch': 3.27}

 33%|███▎      | 239/730 [21:14<43:30,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.963938408514886e-05, 'epoch': 3.27}

 33%|███▎      | 239/730 [21:14<43:30,  5.32s/it]
                                                 
{'debug/num_tok_total': 2454.0, 'debug/num_tok_loss': 1808.0, 'debug/num_lat_total': 2454.0, 'debug/num_lat_loss': 1808.0, 'epoch': 3.27}

 33%|███▎      | 239/730 [21:15<43:30,  5.32s/it]
                                                 
{'train/ce_loss': 2.21875, 'train/diffusion_loss': 0.45075204968452454, 'epoch': 3.27}

 33%|███▎      | 239/730 [21:15<43:30,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.963938408514886e-05, 'epoch': 3.27}

 33%|███▎      | 239/730 [21:15<43:30,  5.32s/it]03/16/2026 07:08:48 - INFO - __main__ - LoRA debug step 240: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 33%|███▎      | 240/730 [21:16<43:18,  5.30s/it]
                                                 
{'loss': 2.699, 'grad_norm': 1.4538198709487915, 'learning_rate': 1.963938408514886e-05, 'epoch': 3.29}

 33%|███▎      | 240/730 [21:16<43:18,  5.30s/it]
                                                 
{'debug/num_tok_total': 2900.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 2900.0, 'debug/num_lat_loss': 1801.0, 'epoch': 3.29}

 33%|███▎      | 240/730 [21:17<43:18,  5.30s/it]
                                                 
{'train/ce_loss': 2.09375, 'train/diffusion_loss': 0.340977281332016, 'epoch': 3.29}

 33%|███▎      | 240/730 [21:17<43:18,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.959378497241715e-05, 'epoch': 3.29}

 33%|███▎      | 240/730 [21:17<43:18,  5.30s/it]
                                                 
{'debug/num_tok_total': 2838.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2838.0, 'debug/num_lat_loss': 1766.0, 'epoch': 3.29}

 33%|███▎      | 240/730 [21:18<43:18,  5.30s/it]
                                                 
{'train/ce_loss': 2.265625, 'train/diffusion_loss': 0.38470348715782166, 'epoch': 3.29}

 33%|███▎      | 240/730 [21:18<43:18,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.959378497241715e-05, 'epoch': 3.29}

 33%|███▎      | 240/730 [21:18<43:18,  5.30s/it]
                                                 
{'debug/num_tok_total': 2607.0, 'debug/num_tok_loss': 1760.0, 'debug/num_lat_total': 2607.0, 'debug/num_lat_loss': 1760.0, 'epoch': 3.29}

 33%|███▎      | 240/730 [21:19<43:18,  5.30s/it]
                                                 
{'train/ce_loss': 2.328125, 'train/diffusion_loss': 0.3994775712490082, 'epoch': 3.29}

 33%|███▎      | 240/730 [21:19<43:18,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.959378497241715e-05, 'epoch': 3.29}

 33%|███▎      | 240/730 [21:19<43:18,  5.30s/it]
                                                 
{'debug/num_tok_total': 2628.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 2628.0, 'debug/num_lat_loss': 1767.0, 'epoch': 3.29}

 33%|███▎      | 240/730 [21:21<43:18,  5.30s/it]
                                                 
{'train/ce_loss': 2.5625, 'train/diffusion_loss': 0.40093958377838135, 'epoch': 3.29}

 33%|███▎      | 240/730 [21:21<43:18,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.959378497241715e-05, 'epoch': 3.29}

 33%|███▎      | 240/730 [21:21<43:18,  5.30s/it]
 33%|███▎      | 241/730 [21:21<43:45,  5.37s/it]
                                                 
{'debug/num_tok_total': 2614.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2614.0, 'debug/num_lat_loss': 1766.0, 'epoch': 3.3}

 33%|███▎      | 241/730 [21:22<43:45,  5.37s/it]
                                                 
{'train/ce_loss': 2.296875, 'train/diffusion_loss': 0.4155711829662323, 'epoch': 3.3}

 33%|███▎      | 241/730 [21:22<43:45,  5.37s/it]
                                                 
{'train/learning_rate_real': 1.95480461873042e-05, 'epoch': 3.3}

 33%|███▎      | 241/730 [21:22<43:45,  5.37s/it]
                                                 
{'debug/num_tok_total': 2688.0, 'debug/num_tok_loss': 1814.0, 'debug/num_lat_total': 2688.0, 'debug/num_lat_loss': 1814.0, 'epoch': 3.3}

 33%|███▎      | 241/730 [21:24<43:45,  5.37s/it]
                                                 
{'train/ce_loss': 2.328125, 'train/diffusion_loss': 0.4016936421394348, 'epoch': 3.3}

 33%|███▎      | 241/730 [21:24<43:45,  5.37s/it]
                                                 
{'train/learning_rate_real': 1.95480461873042e-05, 'epoch': 3.3}

 33%|███▎      | 241/730 [21:24<43:45,  5.37s/it]
                                                 
{'debug/num_tok_total': 2338.0, 'debug/num_tok_loss': 1707.0, 'debug/num_lat_total': 2338.0, 'debug/num_lat_loss': 1707.0, 'epoch': 3.3}

 33%|███▎      | 241/730 [21:25<43:45,  5.37s/it]
                                                 
{'train/ce_loss': 2.21875, 'train/diffusion_loss': 0.4285438656806946, 'epoch': 3.3}

 33%|███▎      | 241/730 [21:25<43:45,  5.37s/it]
                                                 
{'train/learning_rate_real': 1.95480461873042e-05, 'epoch': 3.3}

 33%|███▎      | 241/730 [21:25<43:45,  5.37s/it]
                                                 
{'debug/num_tok_total': 2637.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2637.0, 'debug/num_lat_loss': 1778.0, 'epoch': 3.3}

 33%|███▎      | 241/730 [21:26<43:45,  5.37s/it]
                                                 
{'train/ce_loss': 2.296875, 'train/diffusion_loss': 0.4184286594390869, 'epoch': 3.3}

 33%|███▎      | 241/730 [21:26<43:45,  5.37s/it]
                                                 
{'train/learning_rate_real': 1.95480461873042e-05, 'epoch': 3.3}

 33%|███▎      | 241/730 [21:26<43:45,  5.37s/it]
 33%|███▎      | 242/730 [21:27<43:36,  5.36s/it]
                                                 
{'debug/num_tok_total': 2450.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2450.0, 'debug/num_lat_loss': 1791.0, 'epoch': 3.32}

 33%|███▎      | 242/730 [21:28<43:36,  5.36s/it]
                                                 
{'train/ce_loss': 2.3125, 'train/diffusion_loss': 0.46178358793258667, 'epoch': 3.32}

 33%|███▎      | 242/730 [21:28<43:36,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.9502168630379324e-05, 'epoch': 3.32}

 33%|███▎      | 242/730 [21:28<43:36,  5.36s/it]
                                                 
{'debug/num_tok_total': 2459.0, 'debug/num_tok_loss': 1805.0, 'debug/num_lat_total': 2459.0, 'debug/num_lat_loss': 1805.0, 'epoch': 3.32}

 33%|███▎      | 242/730 [21:29<43:36,  5.36s/it]
                                                 
{'train/ce_loss': 2.203125, 'train/diffusion_loss': 0.44902271032333374, 'epoch': 3.32}

 33%|███▎      | 242/730 [21:29<43:36,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.9502168630379324e-05, 'epoch': 3.32}

 33%|███▎      | 242/730 [21:29<43:36,  5.36s/it]
                                                 
{'debug/num_tok_total': 2481.0, 'debug/num_tok_loss': 1701.0, 'debug/num_lat_total': 2481.0, 'debug/num_lat_loss': 1701.0, 'epoch': 3.32}

 33%|███▎      | 242/730 [21:30<43:36,  5.36s/it]
                                                 
{'train/ce_loss': 2.265625, 'train/diffusion_loss': 0.43946367502212524, 'epoch': 3.32}

 33%|███▎      | 242/730 [21:30<43:36,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.9502168630379324e-05, 'epoch': 3.32}

 33%|███▎      | 242/730 [21:30<43:36,  5.36s/it]
                                                 
{'debug/num_tok_total': 2530.0, 'debug/num_tok_loss': 1681.0, 'debug/num_lat_total': 2530.0, 'debug/num_lat_loss': 1681.0, 'epoch': 3.32}

 33%|███▎      | 242/730 [21:31<43:36,  5.36s/it]
                                                 
{'train/ce_loss': 2.328125, 'train/diffusion_loss': 0.37286049127578735, 'epoch': 3.32}

 33%|███▎      | 242/730 [21:31<43:36,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.9502168630379324e-05, 'epoch': 3.32}

 33%|███▎      | 242/730 [21:31<43:36,  5.36s/it]
 33%|███▎      | 243/730 [21:32<42:41,  5.26s/it]
                                                 
{'debug/num_tok_total': 2653.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2653.0, 'debug/num_lat_loss': 1791.0, 'epoch': 3.33}

 33%|███▎      | 243/730 [21:33<42:41,  5.26s/it]
                                                 
{'train/ce_loss': 2.203125, 'train/diffusion_loss': 0.3862497806549072, 'epoch': 3.33}

 33%|███▎      | 243/730 [21:33<42:41,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.9456153204944172e-05, 'epoch': 3.33}

 33%|███▎      | 243/730 [21:33<42:41,  5.26s/it]
                                                 
{'debug/num_tok_total': 2199.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2199.0, 'debug/num_lat_loss': 1764.0, 'epoch': 3.33}

 33%|███▎      | 243/730 [21:34<42:41,  5.26s/it]
                                                 
{'train/ce_loss': 2.25, 'train/diffusion_loss': 0.4877125918865204, 'epoch': 3.33}

 33%|███▎      | 243/730 [21:34<42:41,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.9456153204944172e-05, 'epoch': 3.33}

 33%|███▎      | 243/730 [21:34<42:41,  5.26s/it]
                                                 
{'debug/num_tok_total': 1988.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 1988.0, 'debug/num_lat_loss': 1779.0, 'epoch': 3.33}

 33%|███▎      | 243/730 [21:35<42:41,  5.26s/it]
                                                 
{'train/ce_loss': 2.234375, 'train/diffusion_loss': 0.5290765166282654, 'epoch': 3.33}

 33%|███▎      | 243/730 [21:35<42:41,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.9456153204944172e-05, 'epoch': 3.33}

 33%|███▎      | 243/730 [21:35<42:41,  5.26s/it]
                                                 
{'debug/num_tok_total': 2226.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2226.0, 'debug/num_lat_loss': 1783.0, 'epoch': 3.33}

 33%|███▎      | 243/730 [21:36<42:41,  5.26s/it]
                                                 
{'train/ce_loss': 2.25, 'train/diffusion_loss': 0.47163859009742737, 'epoch': 3.33}

 33%|███▎      | 243/730 [21:36<42:41,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.9456153204944172e-05, 'epoch': 3.33}

 33%|███▎      | 243/730 [21:36<42:41,  5.26s/it]
 33%|███▎      | 244/730 [21:36<41:30,  5.12s/it]
                                                 
{'debug/num_tok_total': 3078.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 3078.0, 'debug/num_lat_loss': 1784.0, 'epoch': 3.34}

 33%|███▎      | 244/730 [21:38<41:30,  5.12s/it]
                                                 
{'train/ce_loss': 2.328125, 'train/diffusion_loss': 0.31024348735809326, 'epoch': 3.34}

 33%|███▎      | 244/730 [21:38<41:30,  5.12s/it]
                                                 
{'train/learning_rate_real': 1.941000081701492e-05, 'epoch': 3.34}

 33%|███▎      | 244/730 [21:38<41:30,  5.12s/it]
                                                 
{'debug/num_tok_total': 3372.0, 'debug/num_tok_loss': 1828.0, 'debug/num_lat_total': 3372.0, 'debug/num_lat_loss': 1828.0, 'epoch': 3.34}

 33%|███▎      | 244/730 [21:39<41:30,  5.12s/it]
                                                 
{'train/ce_loss': 2.28125, 'train/diffusion_loss': 0.2503184974193573, 'epoch': 3.34}

 33%|███▎      | 244/730 [21:39<41:30,  5.12s/it]
                                                 
{'train/learning_rate_real': 1.941000081701492e-05, 'epoch': 3.34}

 33%|███▎      | 244/730 [21:39<41:30,  5.12s/it]
                                                 
{'debug/num_tok_total': 2654.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2654.0, 'debug/num_lat_loss': 1784.0, 'epoch': 3.34}

 33%|███▎      | 244/730 [21:40<41:30,  5.12s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.4020564556121826, 'epoch': 3.34}

 33%|███▎      | 244/730 [21:40<41:30,  5.12s/it]
                                                 
{'train/learning_rate_real': 1.941000081701492e-05, 'epoch': 3.34}

 33%|███▎      | 244/730 [21:40<41:30,  5.12s/it]
                                                 
{'debug/num_tok_total': 2814.0, 'debug/num_tok_loss': 1761.0, 'debug/num_lat_total': 2814.0, 'debug/num_lat_loss': 1761.0, 'epoch': 3.34}

 33%|███▎      | 244/730 [21:42<41:30,  5.12s/it]
                                                 
{'train/ce_loss': 2.359375, 'train/diffusion_loss': 0.35802125930786133, 'epoch': 3.34}

 33%|███▎      | 244/730 [21:42<41:30,  5.12s/it]
                                                 
{'train/learning_rate_real': 1.941000081701492e-05, 'epoch': 3.34}

 33%|███▎      | 244/730 [21:42<41:30,  5.12s/it]
 34%|███▎      | 245/730 [21:42<42:56,  5.31s/it]
                                                 
{'debug/num_tok_total': 2435.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2435.0, 'debug/num_lat_loss': 1788.0, 'epoch': 3.36}

 34%|███▎      | 245/730 [21:43<42:56,  5.31s/it]
                                                 
{'train/ce_loss': 2.265625, 'train/diffusion_loss': 0.45023059844970703, 'epoch': 3.36}

 34%|███▎      | 245/730 [21:43<42:56,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.9363712375304478e-05, 'epoch': 3.36}

 34%|███▎      | 245/730 [21:43<42:56,  5.31s/it]
                                                 
{'debug/num_tok_total': 2583.0, 'debug/num_tok_loss': 1655.0, 'debug/num_lat_total': 2583.0, 'debug/num_lat_loss': 1655.0, 'epoch': 3.36}

 34%|███▎      | 245/730 [21:45<42:56,  5.31s/it]
                                                 
{'train/ce_loss': 2.25, 'train/diffusion_loss': 0.39521628618240356, 'epoch': 3.36}

 34%|███▎      | 245/730 [21:45<42:56,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.9363712375304478e-05, 'epoch': 3.36}

 34%|███▎      | 245/730 [21:45<42:56,  5.31s/it]
                                                 
{'debug/num_tok_total': 2674.0, 'debug/num_tok_loss': 1610.0, 'debug/num_lat_total': 2674.0, 'debug/num_lat_loss': 1610.0, 'epoch': 3.36}

 34%|███▎      | 245/730 [21:46<42:56,  5.31s/it]
                                                 
{'train/ce_loss': 2.140625, 'train/diffusion_loss': 0.3283180594444275, 'epoch': 3.36}

 34%|███▎      | 245/730 [21:46<42:56,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.9363712375304478e-05, 'epoch': 3.36}

 34%|███▎      | 245/730 [21:46<42:56,  5.31s/it]
                                                 
{'debug/num_tok_total': 3026.0, 'debug/num_tok_loss': 1762.0, 'debug/num_lat_total': 3026.0, 'debug/num_lat_loss': 1762.0, 'epoch': 3.36}

 34%|███▎      | 245/730 [21:47<42:56,  5.31s/it]
                                                 
{'train/ce_loss': 2.28125, 'train/diffusion_loss': 0.29643863439559937, 'epoch': 3.36}

 34%|███▎      | 245/730 [21:47<42:56,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.9363712375304478e-05, 'epoch': 3.36}

 34%|███▎      | 245/730 [21:47<42:56,  5.31s/it]
 34%|███▎      | 246/730 [21:48<43:14,  5.36s/it]
                                                 
{'debug/num_tok_total': 1897.0, 'debug/num_tok_loss': 1623.0, 'debug/num_lat_total': 1897.0, 'debug/num_lat_loss': 1623.0, 'epoch': 3.37}

 34%|███▎      | 246/730 [21:49<43:14,  5.36s/it]
                                                 
{'train/ce_loss': 2.1875, 'train/diffusion_loss': 0.5394054055213928, 'epoch': 3.37}

 34%|███▎      | 246/730 [21:49<43:14,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.931728879120456e-05, 'epoch': 3.37}

 34%|███▎      | 246/730 [21:49<43:14,  5.36s/it]
                                                 
{'debug/num_tok_total': 2409.0, 'debug/num_tok_loss': 1760.0, 'debug/num_lat_total': 2409.0, 'debug/num_lat_loss': 1760.0, 'epoch': 3.37}

 34%|███▎      | 246/730 [21:50<43:14,  5.36s/it]
                                                 
{'train/ce_loss': 2.21875, 'train/diffusion_loss': 0.4408913254737854, 'epoch': 3.37}

 34%|███▎      | 246/730 [21:50<43:14,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.931728879120456e-05, 'epoch': 3.37}

 34%|███▎      | 246/730 [21:50<43:14,  5.36s/it]
                                                 
{'debug/num_tok_total': 2205.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2205.0, 'debug/num_lat_loss': 1776.0, 'epoch': 3.37}

 34%|███▎      | 246/730 [21:51<43:14,  5.36s/it]
                                                 
{'train/ce_loss': 2.140625, 'train/diffusion_loss': 0.4855363368988037, 'epoch': 3.37}

 34%|███▎      | 246/730 [21:51<43:14,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.931728879120456e-05, 'epoch': 3.37}

 34%|███▎      | 246/730 [21:51<43:14,  5.36s/it]
                                                 
{'debug/num_tok_total': 2417.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2417.0, 'debug/num_lat_loss': 1782.0, 'epoch': 3.37}

 34%|███▎      | 246/730 [21:52<43:14,  5.36s/it]
                                                 
{'train/ce_loss': 2.234375, 'train/diffusion_loss': 0.4715399444103241, 'epoch': 3.37}

 34%|███▎      | 246/730 [21:52<43:14,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.931728879120456e-05, 'epoch': 3.37}

 34%|███▎      | 246/730 [21:52<43:14,  5.36s/it]
 34%|███▍      | 247/730 [21:52<41:51,  5.20s/it]
                                                 
{'debug/num_tok_total': 3078.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 3078.0, 'debug/num_lat_loss': 1780.0, 'epoch': 3.38}

 34%|███▍      | 247/730 [21:54<41:51,  5.20s/it]
                                                 
{'train/ce_loss': 2.1875, 'train/diffusion_loss': 0.3111913204193115, 'epoch': 3.38}

 34%|███▍      | 247/730 [21:54<41:51,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.9270730978767764e-05, 'epoch': 3.38}

 34%|███▍      | 247/730 [21:54<41:51,  5.20s/it]
                                                 
{'debug/num_tok_total': 2691.0, 'debug/num_tok_loss': 1818.0, 'debug/num_lat_total': 2691.0, 'debug/num_lat_loss': 1818.0, 'epoch': 3.38}

 34%|███▍      | 247/730 [21:55<41:51,  5.20s/it]
                                                 
{'train/ce_loss': 2.21875, 'train/diffusion_loss': 0.3947170674800873, 'epoch': 3.38}

 34%|███▍      | 247/730 [21:55<41:51,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.9270730978767764e-05, 'epoch': 3.38}

 34%|███▍      | 247/730 [21:55<41:51,  5.20s/it]
                                                 
{'debug/num_tok_total': 1974.0, 'debug/num_tok_loss': 1759.0, 'debug/num_lat_total': 1974.0, 'debug/num_lat_loss': 1759.0, 'epoch': 3.38}

 34%|███▍      | 247/730 [21:56<41:51,  5.20s/it]
                                                 
{'train/ce_loss': 2.234375, 'train/diffusion_loss': 0.5423211455345154, 'epoch': 3.38}

 34%|███▍      | 247/730 [21:56<41:51,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.9270730978767764e-05, 'epoch': 3.38}

 34%|███▍      | 247/730 [21:56<41:51,  5.20s/it]
                                                 
{'debug/num_tok_total': 2899.0, 'debug/num_tok_loss': 1811.0, 'debug/num_lat_total': 2899.0, 'debug/num_lat_loss': 1811.0, 'epoch': 3.38}

 34%|███▍      | 247/730 [21:57<41:51,  5.20s/it]
                                                 
{'train/ce_loss': 2.234375, 'train/diffusion_loss': 0.4050007164478302, 'epoch': 3.38}

 34%|███▍      | 247/730 [21:57<41:51,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.9270730978767764e-05, 'epoch': 3.38}

 34%|███▍      | 247/730 [21:57<41:51,  5.20s/it]
 34%|███▍      | 248/730 [21:58<41:42,  5.19s/it]
                                                 
{'debug/num_tok_total': 2637.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2637.0, 'debug/num_lat_loss': 1771.0, 'epoch': 3.4}

 34%|███▍      | 248/730 [21:59<41:42,  5.19s/it]
                                                 
{'train/ce_loss': 2.25, 'train/diffusion_loss': 0.4002581834793091, 'epoch': 3.4}

 34%|███▍      | 248/730 [21:59<41:42,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.922403985468955e-05, 'epoch': 3.4}

 34%|███▍      | 248/730 [21:59<41:42,  5.19s/it]
                                                 
{'debug/num_tok_total': 2821.0, 'debug/num_tok_loss': 1755.0, 'debug/num_lat_total': 2821.0, 'debug/num_lat_loss': 1755.0, 'epoch': 3.4}

 34%|███▍      | 248/730 [22:00<41:42,  5.19s/it]
                                                 
{'train/ce_loss': 2.25, 'train/diffusion_loss': 0.3785499334335327, 'epoch': 3.4}

 34%|███▍      | 248/730 [22:00<41:42,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.922403985468955e-05, 'epoch': 3.4}

 34%|███▍      | 248/730 [22:00<41:42,  5.19s/it]
                                                 
{'debug/num_tok_total': 3077.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 3077.0, 'debug/num_lat_loss': 1784.0, 'epoch': 3.4}

 34%|███▍      | 248/730 [22:01<41:42,  5.19s/it]
                                                 
{'train/ce_loss': 2.21875, 'train/diffusion_loss': 0.33860430121421814, 'epoch': 3.4}

 34%|███▍      | 248/730 [22:01<41:42,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.922403985468955e-05, 'epoch': 3.4}

 34%|███▍      | 248/730 [22:01<41:42,  5.19s/it]
                                                 
{'debug/num_tok_total': 2688.0, 'debug/num_tok_loss': 1611.0, 'debug/num_lat_total': 2688.0, 'debug/num_lat_loss': 1611.0, 'epoch': 3.4}

 34%|███▍      | 248/730 [22:03<41:42,  5.19s/it]
                                                 
{'train/ce_loss': 2.203125, 'train/diffusion_loss': 0.33459946513175964, 'epoch': 3.4}

 34%|███▍      | 248/730 [22:03<41:42,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.922403985468955e-05, 'epoch': 3.4}

 34%|███▍      | 248/730 [22:03<41:42,  5.19s/it]
 34%|███▍      | 249/730 [22:03<42:25,  5.29s/it]
                                                 
{'debug/num_tok_total': 3101.0, 'debug/num_tok_loss': 1805.0, 'debug/num_lat_total': 3101.0, 'debug/num_lat_loss': 1805.0, 'epoch': 3.41}

 34%|███▍      | 249/730 [22:04<42:25,  5.29s/it]
                                                 
{'train/ce_loss': 2.265625, 'train/diffusion_loss': 0.33131060004234314, 'epoch': 3.41}

 34%|███▍      | 249/730 [22:04<42:25,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.9177216338290214e-05, 'epoch': 3.41}

 34%|███▍      | 249/730 [22:04<42:25,  5.29s/it]
                                                 
{'debug/num_tok_total': 2604.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2604.0, 'debug/num_lat_loss': 1766.0, 'epoch': 3.41}

 34%|███▍      | 249/730 [22:06<42:25,  5.29s/it]
                                                 
{'train/ce_loss': 2.140625, 'train/diffusion_loss': 0.4083004295825958, 'epoch': 3.41}

 34%|███▍      | 249/730 [22:06<42:25,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.9177216338290214e-05, 'epoch': 3.41}

 34%|███▍      | 249/730 [22:06<42:25,  5.29s/it]
                                                 
{'debug/num_tok_total': 2679.0, 'debug/num_tok_loss': 1805.0, 'debug/num_lat_total': 2679.0, 'debug/num_lat_loss': 1805.0, 'epoch': 3.41}

 34%|███▍      | 249/730 [22:07<42:25,  5.29s/it]
                                                 
{'train/ce_loss': 2.265625, 'train/diffusion_loss': 0.3885483145713806, 'epoch': 3.41}

 34%|███▍      | 249/730 [22:07<42:25,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.9177216338290214e-05, 'epoch': 3.41}

 34%|███▍      | 249/730 [22:07<42:25,  5.29s/it]
                                                 
{'debug/num_tok_total': 2629.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2629.0, 'debug/num_lat_loss': 1780.0, 'epoch': 3.41}

 34%|███▍      | 249/730 [22:08<42:25,  5.29s/it]
                                                 
{'train/ce_loss': 2.203125, 'train/diffusion_loss': 0.4085504710674286, 'epoch': 3.41}

 34%|███▍      | 249/730 [22:08<42:25,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.9177216338290214e-05, 'epoch': 3.41}

 34%|███▍      | 249/730 [22:08<42:25,  5.29s/it]03/16/2026 07:09:41 - INFO - __main__ - LoRA debug step 250: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 34%|███▍      | 250/730 [22:09<42:42,  5.34s/it]
                                                 
{'loss': 2.6215, 'grad_norm': 1.3147482872009277, 'learning_rate': 1.9177216338290214e-05, 'epoch': 3.42}

 34%|███▍      | 250/730 [22:09<42:42,  5.34s/it]
                                                 
{'debug/num_tok_total': 2662.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2662.0, 'debug/num_lat_loss': 1792.0, 'epoch': 3.42}

 34%|███▍      | 250/730 [22:10<42:42,  5.34s/it]
                                                 
{'train/ce_loss': 2.25, 'train/diffusion_loss': 0.3955235183238983, 'epoch': 3.42}

 34%|███▍      | 250/730 [22:10<42:42,  5.34s/it]
                                                 
{'train/learning_rate_real': 1.913026135149678e-05, 'epoch': 3.42}

 34%|███▍      | 250/730 [22:10<42:42,  5.34s/it]
                                                 
{'debug/num_tok_total': 2162.0, 'debug/num_tok_loss': 1649.0, 'debug/num_lat_total': 2162.0, 'debug/num_lat_loss': 1649.0, 'epoch': 3.42}

 34%|███▍      | 250/730 [22:11<42:42,  5.34s/it]
                                                 
{'train/ce_loss': 2.234375, 'train/diffusion_loss': 0.4867064654827118, 'epoch': 3.42}

 34%|███▍      | 250/730 [22:11<42:42,  5.34s/it]
                                                 
{'train/learning_rate_real': 1.913026135149678e-05, 'epoch': 3.42}

 34%|███▍      | 250/730 [22:11<42:42,  5.34s/it]
                                                 
{'debug/num_tok_total': 2884.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2884.0, 'debug/num_lat_loss': 1790.0, 'epoch': 3.42}

 34%|███▍      | 250/730 [22:12<42:42,  5.34s/it]
                                                 
{'train/ce_loss': 2.328125, 'train/diffusion_loss': 0.36458444595336914, 'epoch': 3.42}

 34%|███▍      | 250/730 [22:12<42:42,  5.34s/it]
                                                 
{'train/learning_rate_real': 1.913026135149678e-05, 'epoch': 3.42}

 34%|███▍      | 250/730 [22:12<42:42,  5.34s/it]
                                                 
{'debug/num_tok_total': 2216.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2216.0, 'debug/num_lat_loss': 1785.0, 'epoch': 3.42}

 34%|███▍      | 250/730 [22:13<42:42,  5.34s/it]
                                                 
{'train/ce_loss': 2.359375, 'train/diffusion_loss': 0.5149307250976562, 'epoch': 3.42}

 34%|███▍      | 250/730 [22:13<42:42,  5.34s/it]
                                                 
{'train/learning_rate_real': 1.913026135149678e-05, 'epoch': 3.42}

 34%|███▍      | 250/730 [22:13<42:42,  5.34s/it]
 34%|███▍      | 251/730 [22:14<42:06,  5.27s/it]
                                                 
{'debug/num_tok_total': 2664.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2664.0, 'debug/num_lat_loss': 1786.0, 'epoch': 3.44}

 34%|███▍      | 251/730 [22:15<42:06,  5.27s/it]
                                                 
{'train/ce_loss': 2.21875, 'train/diffusion_loss': 0.4316481351852417, 'epoch': 3.44}

 34%|███▍      | 251/730 [22:15<42:06,  5.27s/it]
                                                 
{'train/learning_rate_real': 1.9083175818824838e-05, 'epoch': 3.44}

 34%|███▍      | 251/730 [22:15<42:06,  5.27s/it]
                                                 
{'debug/num_tok_total': 2607.0, 'debug/num_tok_loss': 1748.0, 'debug/num_lat_total': 2607.0, 'debug/num_lat_loss': 1748.0, 'epoch': 3.44}

 34%|███▍      | 251/730 [22:16<42:06,  5.27s/it]
                                                 
{'train/ce_loss': 2.109375, 'train/diffusion_loss': 0.4034005403518677, 'epoch': 3.44}

 34%|███▍      | 251/730 [22:16<42:06,  5.27s/it]
                                                 
{'train/learning_rate_real': 1.9083175818824838e-05, 'epoch': 3.44}

 34%|███▍      | 251/730 [22:16<42:06,  5.27s/it]
                                                 
{'debug/num_tok_total': 2887.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 2887.0, 'debug/num_lat_loss': 1801.0, 'epoch': 3.44}

 34%|███▍      | 251/730 [22:17<42:06,  5.27s/it]
                                                 
{'train/ce_loss': 2.09375, 'train/diffusion_loss': 0.35007497668266296, 'epoch': 3.44}

 34%|███▍      | 251/730 [22:17<42:06,  5.27s/it]
                                                 
{'train/learning_rate_real': 1.9083175818824838e-05, 'epoch': 3.44}

 34%|███▍      | 251/730 [22:17<42:06,  5.27s/it]
                                                 
{'debug/num_tok_total': 2418.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 2418.0, 'debug/num_lat_loss': 1769.0, 'epoch': 3.44}

 34%|███▍      | 251/730 [22:19<42:06,  5.27s/it]
                                                 
{'train/ce_loss': 2.15625, 'train/diffusion_loss': 0.4340359568595886, 'epoch': 3.44}

 34%|███▍      | 251/730 [22:19<42:06,  5.27s/it]
                                                 
{'train/learning_rate_real': 1.9083175818824838e-05, 'epoch': 3.44}

 34%|███▍      | 251/730 [22:19<42:06,  5.27s/it]
 35%|███▍      | 252/730 [22:19<42:14,  5.30s/it]
                                                 
{'debug/num_tok_total': 2995.0, 'debug/num_tok_loss': 1707.0, 'debug/num_lat_total': 2995.0, 'debug/num_lat_loss': 1707.0, 'epoch': 3.45}

 35%|███▍      | 252/730 [22:20<42:14,  5.30s/it]
                                                 
{'train/ce_loss': 2.34375, 'train/diffusion_loss': 0.2904086709022522, 'epoch': 3.45}

 35%|███▍      | 252/730 [22:20<42:14,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.9035960667360348e-05, 'epoch': 3.45}

 35%|███▍      | 252/730 [22:20<42:14,  5.30s/it]
                                                 
{'debug/num_tok_total': 2293.0, 'debug/num_tok_loss': 1664.0, 'debug/num_lat_total': 2293.0, 'debug/num_lat_loss': 1664.0, 'epoch': 3.45}

 35%|███▍      | 252/730 [22:22<42:14,  5.30s/it]
                                                 
{'train/ce_loss': 2.546875, 'train/diffusion_loss': 0.4232002794742584, 'epoch': 3.45}

 35%|███▍      | 252/730 [22:22<42:14,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.9035960667360348e-05, 'epoch': 3.45}

 35%|███▍      | 252/730 [22:22<42:14,  5.30s/it]
                                                 
{'debug/num_tok_total': 2390.0, 'debug/num_tok_loss': 1761.0, 'debug/num_lat_total': 2390.0, 'debug/num_lat_loss': 1761.0, 'epoch': 3.45}

 35%|███▍      | 252/730 [22:23<42:14,  5.30s/it]
                                                 
{'train/ce_loss': 2.140625, 'train/diffusion_loss': 0.4461905360221863, 'epoch': 3.45}

 35%|███▍      | 252/730 [22:23<42:14,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.9035960667360348e-05, 'epoch': 3.45}

 35%|███▍      | 252/730 [22:23<42:14,  5.30s/it]
                                                 
{'debug/num_tok_total': 2956.0, 'debug/num_tok_loss': 1717.0, 'debug/num_lat_total': 2956.0, 'debug/num_lat_loss': 1717.0, 'epoch': 3.45}

 35%|███▍      | 252/730 [22:24<42:14,  5.30s/it]
                                                 
{'train/ce_loss': 2.171875, 'train/diffusion_loss': 0.32390090823173523, 'epoch': 3.45}

 35%|███▍      | 252/730 [22:24<42:14,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.9035960667360348e-05, 'epoch': 3.45}

 35%|███▍      | 252/730 [22:24<42:14,  5.30s/it]
 35%|███▍      | 253/730 [22:25<42:31,  5.35s/it]
                                                 
{'debug/num_tok_total': 2489.0, 'debug/num_tok_loss': 1600.0, 'debug/num_lat_total': 2489.0, 'debug/num_lat_loss': 1600.0, 'epoch': 3.47}

 35%|███▍      | 253/730 [22:26<42:31,  5.35s/it]
                                                 
{'train/ce_loss': 2.25, 'train/diffusion_loss': 0.37611258029937744, 'epoch': 3.47}

 35%|███▍      | 253/730 [22:26<42:31,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.8988616826741386e-05, 'epoch': 3.47}

 35%|███▍      | 253/730 [22:26<42:31,  5.35s/it]
                                                 
{'debug/num_tok_total': 2665.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2665.0, 'debug/num_lat_loss': 1794.0, 'epoch': 3.47}

 35%|███▍      | 253/730 [22:27<42:31,  5.35s/it]
                                                 
{'train/ce_loss': 2.125, 'train/diffusion_loss': 0.39329659938812256, 'epoch': 3.47}

 35%|███▍      | 253/730 [22:27<42:31,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.8988616826741386e-05, 'epoch': 3.47}

 35%|███▍      | 253/730 [22:27<42:31,  5.35s/it]
                                                 
{'debug/num_tok_total': 2415.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2415.0, 'debug/num_lat_loss': 1775.0, 'epoch': 3.47}

 35%|███▍      | 253/730 [22:28<42:31,  5.35s/it]
                                                 
{'train/ce_loss': 2.3125, 'train/diffusion_loss': 0.43804121017456055, 'epoch': 3.47}

 35%|███▍      | 253/730 [22:28<42:31,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.8988616826741386e-05, 'epoch': 3.47}

 35%|███▍      | 253/730 [22:28<42:31,  5.35s/it]
                                                 
{'debug/num_tok_total': 2610.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2610.0, 'debug/num_lat_loss': 1771.0, 'epoch': 3.47}

 35%|███▍      | 253/730 [22:29<42:31,  5.35s/it]
                                                 
{'train/ce_loss': 2.40625, 'train/diffusion_loss': 0.4059012830257416, 'epoch': 3.47}

 35%|███▍      | 253/730 [22:29<42:31,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.8988616826741386e-05, 'epoch': 3.47}

 35%|███▍      | 253/730 [22:29<42:31,  5.35s/it]
 35%|███▍      | 254/730 [22:30<42:27,  5.35s/it]
                                                 
{'debug/num_tok_total': 3099.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 3099.0, 'debug/num_lat_loss': 1785.0, 'epoch': 3.48}

 35%|███▍      | 254/730 [22:31<42:27,  5.35s/it]
                                                 
{'train/ce_loss': 2.1875, 'train/diffusion_loss': 0.30580952763557434, 'epoch': 3.48}

 35%|███▍      | 254/730 [22:31<42:27,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.894114522913985e-05, 'epoch': 3.48}

 35%|███▍      | 254/730 [22:31<42:27,  5.35s/it]
                                                 
{'debug/num_tok_total': 3007.0, 'debug/num_tok_loss': 1745.0, 'debug/num_lat_total': 3007.0, 'debug/num_lat_loss': 1745.0, 'epoch': 3.48}

 35%|███▍      | 254/730 [22:32<42:27,  5.35s/it]
                                                 
{'train/ce_loss': 2.15625, 'train/diffusion_loss': 0.30429425835609436, 'epoch': 3.48}

 35%|███▍      | 254/730 [22:32<42:27,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.894114522913985e-05, 'epoch': 3.48}

 35%|███▍      | 254/730 [22:32<42:27,  5.35s/it]
                                                 
{'debug/num_tok_total': 2658.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2658.0, 'debug/num_lat_loss': 1781.0, 'epoch': 3.48}

 35%|███▍      | 254/730 [22:34<42:27,  5.35s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.3724602460861206, 'epoch': 3.48}

 35%|███▍      | 254/730 [22:34<42:27,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.894114522913985e-05, 'epoch': 3.48}

 35%|███▍      | 254/730 [22:34<42:27,  5.35s/it]
                                                 
{'debug/num_tok_total': 2506.0, 'debug/num_tok_loss': 1659.0, 'debug/num_lat_total': 2506.0, 'debug/num_lat_loss': 1659.0, 'epoch': 3.48}

 35%|███▍      | 254/730 [22:35<42:27,  5.35s/it]
                                                 
{'train/ce_loss': 2.09375, 'train/diffusion_loss': 0.3866651952266693, 'epoch': 3.48}

 35%|███▍      | 254/730 [22:35<42:27,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.894114522913985e-05, 'epoch': 3.48}

 35%|███▍      | 254/730 [22:35<42:27,  5.35s/it]
 35%|███▍      | 255/730 [22:35<42:36,  5.38s/it]
                                                 
{'debug/num_tok_total': 3073.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 3073.0, 'debug/num_lat_loss': 1779.0, 'epoch': 3.49}

 35%|███▍      | 255/730 [22:37<42:36,  5.38s/it]
                                                 
{'train/ce_loss': 2.28125, 'train/diffusion_loss': 0.3223687708377838, 'epoch': 3.49}

 35%|███▍      | 255/730 [22:37<42:36,  5.38s/it]
                                                 
{'train/learning_rate_real': 1.8893546809243084e-05, 'epoch': 3.49}

 35%|███▍      | 255/730 [22:37<42:36,  5.38s/it]
                                                 
{'debug/num_tok_total': 2290.0, 'debug/num_tok_loss': 1517.0, 'debug/num_lat_total': 2290.0, 'debug/num_lat_loss': 1517.0, 'epoch': 3.49}

 35%|███▍      | 255/730 [22:38<42:36,  5.38s/it]
                                                 
{'train/ce_loss': 2.15625, 'train/diffusion_loss': 0.40321141481399536, 'epoch': 3.49}

 35%|███▍      | 255/730 [22:38<42:36,  5.38s/it]
                                                 
{'train/learning_rate_real': 1.8893546809243084e-05, 'epoch': 3.49}

 35%|███▍      | 255/730 [22:38<42:36,  5.38s/it]
                                                 
{'debug/num_tok_total': 2819.0, 'debug/num_tok_loss': 1756.0, 'debug/num_lat_total': 2819.0, 'debug/num_lat_loss': 1756.0, 'epoch': 3.49}

 35%|███▍      | 255/730 [22:39<42:36,  5.38s/it]
                                                 
{'train/ce_loss': 2.25, 'train/diffusion_loss': 0.37000900506973267, 'epoch': 3.49}

 35%|███▍      | 255/730 [22:39<42:36,  5.38s/it]
                                                 
{'train/learning_rate_real': 1.8893546809243084e-05, 'epoch': 3.49}

 35%|███▍      | 255/730 [22:39<42:36,  5.38s/it]
                                                 
{'debug/num_tok_total': 2637.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2637.0, 'debug/num_lat_loss': 1782.0, 'epoch': 3.49}

 35%|███▍      | 255/730 [22:41<42:36,  5.38s/it]
                                                 
{'train/ce_loss': 2.171875, 'train/diffusion_loss': 0.44469863176345825, 'epoch': 3.49}

 35%|███▍      | 255/730 [22:41<42:36,  5.38s/it]
                                                 
{'train/learning_rate_real': 1.8893546809243084e-05, 'epoch': 3.49}

 35%|███▍      | 255/730 [22:41<42:36,  5.38s/it]
 35%|███▌      | 256/730 [22:41<43:04,  5.45s/it]
                                                 
{'debug/num_tok_total': 2870.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2870.0, 'debug/num_lat_loss': 1790.0, 'epoch': 3.51}

 35%|███▌      | 256/730 [22:42<43:04,  5.45s/it]
                                                 
{'train/ce_loss': 2.15625, 'train/diffusion_loss': 0.3512857258319855, 'epoch': 3.51}

 35%|███▌      | 256/730 [22:42<43:04,  5.45s/it]
                                                 
{'train/learning_rate_real': 1.88458225042355e-05, 'epoch': 3.51}

 35%|███▌      | 256/730 [22:42<43:04,  5.45s/it]
                                                 
{'debug/num_tok_total': 2643.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2643.0, 'debug/num_lat_loss': 1794.0, 'epoch': 3.51}

 35%|███▌      | 256/730 [22:43<43:04,  5.45s/it]
                                                 
{'train/ce_loss': 2.453125, 'train/diffusion_loss': 0.4314759075641632, 'epoch': 3.51}

 35%|███▌      | 256/730 [22:43<43:04,  5.45s/it]
                                                 
{'train/learning_rate_real': 1.88458225042355e-05, 'epoch': 3.51}

 35%|███▌      | 256/730 [22:43<43:04,  5.45s/it]
                                                 
{'debug/num_tok_total': 2746.0, 'debug/num_tok_loss': 1616.0, 'debug/num_lat_total': 2746.0, 'debug/num_lat_loss': 1616.0, 'epoch': 3.51}

 35%|███▌      | 256/730 [22:45<43:04,  5.45s/it]
                                                 
{'train/ce_loss': 2.203125, 'train/diffusion_loss': 0.3479451537132263, 'epoch': 3.51}

 35%|███▌      | 256/730 [22:45<43:04,  5.45s/it]
                                                 
{'train/learning_rate_real': 1.88458225042355e-05, 'epoch': 3.51}

 35%|███▌      | 256/730 [22:45<43:04,  5.45s/it]
                                                 
{'debug/num_tok_total': 2696.0, 'debug/num_tok_loss': 1814.0, 'debug/num_lat_total': 2696.0, 'debug/num_lat_loss': 1814.0, 'epoch': 3.51}

 35%|███▌      | 256/730 [22:46<43:04,  5.45s/it]
                                                 
{'train/ce_loss': 2.15625, 'train/diffusion_loss': 0.43738123774528503, 'epoch': 3.51}

 35%|███▌      | 256/730 [22:46<43:04,  5.45s/it]
                                                 
{'train/learning_rate_real': 1.88458225042355e-05, 'epoch': 3.51}

 35%|███▌      | 256/730 [22:46<43:04,  5.45s/it]
 35%|███▌      | 257/730 [22:46<43:10,  5.48s/it]
                                                 
{'debug/num_tok_total': 2861.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2861.0, 'debug/num_lat_loss': 1781.0, 'epoch': 3.52}

 35%|███▌      | 257/730 [22:48<43:10,  5.48s/it]
                                                 
{'train/ce_loss': 2.1875, 'train/diffusion_loss': 0.3625929653644562, 'epoch': 3.52}

 35%|███▌      | 257/730 [22:48<43:10,  5.48s/it]
                                                 
{'train/learning_rate_real': 1.8797973253780105e-05, 'epoch': 3.52}

 35%|███▌      | 257/730 [22:48<43:10,  5.48s/it]
                                                 
{'debug/num_tok_total': 2453.0, 'debug/num_tok_loss': 1685.0, 'debug/num_lat_total': 2453.0, 'debug/num_lat_loss': 1685.0, 'epoch': 3.52}

 35%|███▌      | 257/730 [22:49<43:10,  5.48s/it]
                                                 
{'train/ce_loss': 2.15625, 'train/diffusion_loss': 0.409828782081604, 'epoch': 3.52}

 35%|███▌      | 257/730 [22:49<43:10,  5.48s/it]
                                                 
{'train/learning_rate_real': 1.8797973253780105e-05, 'epoch': 3.52}

 35%|███▌      | 257/730 [22:49<43:10,  5.48s/it]
                                                 
{'debug/num_tok_total': 2574.0, 'debug/num_tok_loss': 1641.0, 'debug/num_lat_total': 2574.0, 'debug/num_lat_loss': 1641.0, 'epoch': 3.52}

 35%|███▌      | 257/730 [22:50<43:10,  5.48s/it]
                                                 
{'train/ce_loss': 2.171875, 'train/diffusion_loss': 0.3821832537651062, 'epoch': 3.52}

 35%|███▌      | 257/730 [22:50<43:10,  5.48s/it]
                                                 
{'train/learning_rate_real': 1.8797973253780105e-05, 'epoch': 3.52}

 35%|███▌      | 257/730 [22:50<43:10,  5.48s/it]
                                                 
{'debug/num_tok_total': 2733.0, 'debug/num_tok_loss': 1612.0, 'debug/num_lat_total': 2733.0, 'debug/num_lat_loss': 1612.0, 'epoch': 3.52}

 35%|███▌      | 257/730 [22:51<43:10,  5.48s/it]
                                                 
{'train/ce_loss': 2.171875, 'train/diffusion_loss': 0.3593869209289551, 'epoch': 3.52}

 35%|███▌      | 257/730 [22:51<43:10,  5.48s/it]
                                                 
{'train/learning_rate_real': 1.8797973253780105e-05, 'epoch': 3.52}

 35%|███▌      | 257/730 [22:51<43:10,  5.48s/it]
 35%|███▌      | 258/730 [22:52<42:55,  5.46s/it]
                                                 
{'debug/num_tok_total': 2408.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2408.0, 'debug/num_lat_loss': 1766.0, 'epoch': 3.53}

 35%|███▌      | 258/730 [22:53<42:55,  5.46s/it]
                                                 
{'train/ce_loss': 2.25, 'train/diffusion_loss': 0.4660320281982422, 'epoch': 3.53}

 35%|███▌      | 258/730 [22:53<42:55,  5.46s/it]
                                                 
{'train/learning_rate_real': 1.8750000000000002e-05, 'epoch': 3.53}

 35%|███▌      | 258/730 [22:53<42:55,  5.46s/it]
                                                 
{'debug/num_tok_total': 3292.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 3292.0, 'debug/num_lat_loss': 1785.0, 'epoch': 3.53}

 35%|███▌      | 258/730 [22:54<42:55,  5.46s/it]
                                                 
{'train/ce_loss': 2.125, 'train/diffusion_loss': 0.3105265200138092, 'epoch': 3.53}

 35%|███▌      | 258/730 [22:54<42:55,  5.46s/it]
                                                 
{'train/learning_rate_real': 1.8750000000000002e-05, 'epoch': 3.53}

 35%|███▌      | 258/730 [22:54<42:55,  5.46s/it]
                                                 
{'debug/num_tok_total': 2452.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2452.0, 'debug/num_lat_loss': 1795.0, 'epoch': 3.53}

 35%|███▌      | 258/730 [22:56<42:55,  5.46s/it]
                                                 
{'train/ce_loss': 2.171875, 'train/diffusion_loss': 0.41989848017692566, 'epoch': 3.53}

 35%|███▌      | 258/730 [22:56<42:55,  5.46s/it]
                                                 
{'train/learning_rate_real': 1.8750000000000002e-05, 'epoch': 3.53}

 35%|███▌      | 258/730 [22:56<42:55,  5.46s/it]
                                                 
{'debug/num_tok_total': 2661.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2661.0, 'debug/num_lat_loss': 1790.0, 'epoch': 3.53}

 35%|███▌      | 258/730 [22:57<42:55,  5.46s/it]
                                                 
{'train/ce_loss': 2.171875, 'train/diffusion_loss': 0.4183150827884674, 'epoch': 3.53}

 35%|███▌      | 258/730 [22:57<42:55,  5.46s/it]
                                                 
{'train/learning_rate_real': 1.8750000000000002e-05, 'epoch': 3.53}

 35%|███▌      | 258/730 [22:57<42:55,  5.46s/it]
 35%|███▌      | 259/730 [22:57<42:33,  5.42s/it]
                                                 
{'debug/num_tok_total': 2475.0, 'debug/num_tok_loss': 1706.0, 'debug/num_lat_total': 2475.0, 'debug/num_lat_loss': 1706.0, 'epoch': 3.55}

 35%|███▌      | 259/730 [22:58<42:33,  5.42s/it]
                                                 
{'train/ce_loss': 2.125, 'train/diffusion_loss': 0.4180438816547394, 'epoch': 3.55}

 35%|███▌      | 259/730 [22:58<42:33,  5.42s/it]
                                                 
{'train/learning_rate_real': 1.8701903687459858e-05, 'epoch': 3.55}

 35%|███▌      | 259/730 [22:58<42:33,  5.42s/it]
                                                 
{'debug/num_tok_total': 3245.0, 'debug/num_tok_loss': 1758.0, 'debug/num_lat_total': 3245.0, 'debug/num_lat_loss': 1758.0, 'epoch': 3.55}

 35%|███▌      | 259/730 [23:00<42:33,  5.42s/it]
                                                 
{'train/ce_loss': 2.125, 'train/diffusion_loss': 0.2874610424041748, 'epoch': 3.55}

 35%|███▌      | 259/730 [23:00<42:33,  5.42s/it]
                                                 
{'train/learning_rate_real': 1.8701903687459858e-05, 'epoch': 3.55}

 35%|███▌      | 259/730 [23:00<42:33,  5.42s/it]
                                                 
{'debug/num_tok_total': 2750.0, 'debug/num_tok_loss': 1731.0, 'debug/num_lat_total': 2750.0, 'debug/num_lat_loss': 1731.0, 'epoch': 3.55}

 35%|███▌      | 259/730 [23:01<42:33,  5.42s/it]
                                                 
{'train/ce_loss': 2.40625, 'train/diffusion_loss': 0.3924647271633148, 'epoch': 3.55}

 35%|███▌      | 259/730 [23:01<42:33,  5.42s/it]
                                                 
{'train/learning_rate_real': 1.8701903687459858e-05, 'epoch': 3.55}

 35%|███▌      | 259/730 [23:01<42:33,  5.42s/it]
                                                 
{'debug/num_tok_total': 2381.0, 'debug/num_tok_loss': 1752.0, 'debug/num_lat_total': 2381.0, 'debug/num_lat_loss': 1752.0, 'epoch': 3.55}

 35%|███▌      | 259/730 [23:02<42:33,  5.42s/it]
                                                 
{'train/ce_loss': 2.09375, 'train/diffusion_loss': 0.4388349950313568, 'epoch': 3.55}

 35%|███▌      | 259/730 [23:02<42:33,  5.42s/it]
                                                 
{'train/learning_rate_real': 1.8701903687459858e-05, 'epoch': 3.55}

 35%|███▌      | 259/730 [23:02<42:33,  5.42s/it]03/16/2026 07:10:35 - INFO - __main__ - LoRA debug step 260: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 36%|███▌      | 260/730 [23:03<42:28,  5.42s/it]
                                                 
{'loss': 2.5412, 'grad_norm': 1.548999309539795, 'learning_rate': 1.8701903687459858e-05, 'epoch': 3.56}

 36%|███▌      | 260/730 [23:03<42:28,  5.42s/it]
                                                 
{'debug/num_tok_total': 2591.0, 'debug/num_tok_loss': 1756.0, 'debug/num_lat_total': 2591.0, 'debug/num_lat_loss': 1756.0, 'epoch': 3.56}

 36%|███▌      | 260/730 [23:04<42:28,  5.42s/it]
                                                 
{'train/ce_loss': 2.046875, 'train/diffusion_loss': 0.4313473403453827, 'epoch': 3.56}

 36%|███▌      | 260/730 [23:04<42:28,  5.42s/it]
                                                 
{'train/learning_rate_real': 1.8653685263147278e-05, 'epoch': 3.56}

 36%|███▌      | 260/730 [23:04<42:28,  5.42s/it]
                                                 
{'debug/num_tok_total': 3259.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 3259.0, 'debug/num_lat_loss': 1767.0, 'epoch': 3.56}

 36%|███▌      | 260/730 [23:05<42:28,  5.42s/it]
                                                 
{'train/ce_loss': 2.15625, 'train/diffusion_loss': 0.26685136556625366, 'epoch': 3.56}

 36%|███▌      | 260/730 [23:05<42:28,  5.42s/it]
                                                 
{'train/learning_rate_real': 1.8653685263147278e-05, 'epoch': 3.56}

 36%|███▌      | 260/730 [23:05<42:28,  5.42s/it]
                                                 
{'debug/num_tok_total': 2464.0, 'debug/num_tok_loss': 1804.0, 'debug/num_lat_total': 2464.0, 'debug/num_lat_loss': 1804.0, 'epoch': 3.56}

 36%|███▌      | 260/730 [23:06<42:28,  5.42s/it]
                                                 
{'train/ce_loss': 2.125, 'train/diffusion_loss': 0.4555809795856476, 'epoch': 3.56}

 36%|███▌      | 260/730 [23:06<42:28,  5.42s/it]
                                                 
{'train/learning_rate_real': 1.8653685263147278e-05, 'epoch': 3.56}

 36%|███▌      | 260/730 [23:06<42:28,  5.42s/it]
                                                 
{'debug/num_tok_total': 2659.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2659.0, 'debug/num_lat_loss': 1784.0, 'epoch': 3.56}

 36%|███▌      | 260/730 [23:08<42:28,  5.42s/it]
                                                 
{'train/ce_loss': 2.28125, 'train/diffusion_loss': 0.4081794023513794, 'epoch': 3.56}

 36%|███▌      | 260/730 [23:08<42:28,  5.42s/it]
                                                 
{'train/learning_rate_real': 1.8653685263147278e-05, 'epoch': 3.56}

 36%|███▌      | 260/730 [23:08<42:28,  5.42s/it]
 36%|███▌      | 261/730 [23:08<42:15,  5.41s/it]
                                                 
{'debug/num_tok_total': 1826.0, 'debug/num_tok_loss': 1617.0, 'debug/num_lat_total': 1826.0, 'debug/num_lat_loss': 1617.0, 'epoch': 3.58}

 36%|███▌      | 261/730 [23:09<42:15,  5.41s/it]
                                                 
{'train/ce_loss': 2.109375, 'train/diffusion_loss': 0.558996856212616, 'epoch': 3.58}

 36%|███▌      | 261/730 [23:09<42:15,  5.41s/it]
                                                 
{'train/learning_rate_real': 1.860534567645419e-05, 'epoch': 3.58}

 36%|███▌      | 261/730 [23:09<42:15,  5.41s/it]
                                                 
{'debug/num_tok_total': 2823.0, 'debug/num_tok_loss': 1759.0, 'debug/num_lat_total': 2823.0, 'debug/num_lat_loss': 1759.0, 'epoch': 3.58}

 36%|███▌      | 261/730 [23:10<42:15,  5.41s/it]
                                                 
{'train/ce_loss': 2.1875, 'train/diffusion_loss': 0.36656317114830017, 'epoch': 3.58}

 36%|███▌      | 261/730 [23:10<42:15,  5.41s/it]
                                                 
{'train/learning_rate_real': 1.860534567645419e-05, 'epoch': 3.58}

 36%|███▌      | 261/730 [23:10<42:15,  5.41s/it]
                                                 
{'debug/num_tok_total': 2679.0, 'debug/num_tok_loss': 1804.0, 'debug/num_lat_total': 2679.0, 'debug/num_lat_loss': 1804.0, 'epoch': 3.58}

 36%|███▌      | 261/730 [23:12<42:15,  5.41s/it]
                                                 
{'train/ce_loss': 2.625, 'train/diffusion_loss': 0.4010484516620636, 'epoch': 3.58}

 36%|███▌      | 261/730 [23:12<42:15,  5.41s/it]
                                                 
{'train/learning_rate_real': 1.860534567645419e-05, 'epoch': 3.58}

 36%|███▌      | 261/730 [23:12<42:15,  5.41s/it]
                                                 
{'debug/num_tok_total': 2360.0, 'debug/num_tok_loss': 1717.0, 'debug/num_lat_total': 2360.0, 'debug/num_lat_loss': 1717.0, 'epoch': 3.58}

 36%|███▌      | 261/730 [23:13<42:15,  5.41s/it]
                                                 
{'train/ce_loss': 2.046875, 'train/diffusion_loss': 0.4341590404510498, 'epoch': 3.58}

 36%|███▌      | 261/730 [23:13<42:15,  5.41s/it]
                                                 
{'train/learning_rate_real': 1.860534567645419e-05, 'epoch': 3.58}

 36%|███▌      | 261/730 [23:13<42:15,  5.41s/it]
 36%|███▌      | 262/730 [23:13<41:39,  5.34s/it]
                                                 
{'debug/num_tok_total': 2086.0, 'debug/num_tok_loss': 1671.0, 'debug/num_lat_total': 2086.0, 'debug/num_lat_loss': 1671.0, 'epoch': 3.59}

 36%|███▌      | 262/730 [23:14<41:39,  5.34s/it]
                                                 
{'train/ce_loss': 2.25, 'train/diffusion_loss': 0.5023366808891296, 'epoch': 3.59}

 36%|███▌      | 262/730 [23:14<41:39,  5.34s/it]
                                                 
{'train/learning_rate_real': 1.855688587915813e-05, 'epoch': 3.59}

 36%|███▌      | 262/730 [23:14<41:39,  5.34s/it]
                                                 
{'debug/num_tok_total': 2659.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2659.0, 'debug/num_lat_loss': 1785.0, 'epoch': 3.59}

 36%|███▌      | 262/730 [23:15<41:39,  5.34s/it]
                                                 
{'train/ce_loss': 2.125, 'train/diffusion_loss': 0.3760988116264343, 'epoch': 3.59}

 36%|███▌      | 262/730 [23:15<41:39,  5.34s/it]
                                                 
{'train/learning_rate_real': 1.855688587915813e-05, 'epoch': 3.59}

 36%|███▌      | 262/730 [23:15<41:39,  5.34s/it]
                                                 
{'debug/num_tok_total': 3033.0, 'debug/num_tok_loss': 1754.0, 'debug/num_lat_total': 3033.0, 'debug/num_lat_loss': 1754.0, 'epoch': 3.59}

 36%|███▌      | 262/730 [23:17<41:39,  5.34s/it]
                                                 
{'train/ce_loss': 2.03125, 'train/diffusion_loss': 0.3116809129714966, 'epoch': 3.59}

 36%|███▌      | 262/730 [23:17<41:39,  5.34s/it]
                                                 
{'train/learning_rate_real': 1.855688587915813e-05, 'epoch': 3.59}

 36%|███▌      | 262/730 [23:17<41:39,  5.34s/it]
                                                 
{'debug/num_tok_total': 2578.0, 'debug/num_tok_loss': 1649.0, 'debug/num_lat_total': 2578.0, 'debug/num_lat_loss': 1649.0, 'epoch': 3.59}

 36%|███▌      | 262/730 [23:18<41:39,  5.34s/it]
                                                 
{'train/ce_loss': 2.171875, 'train/diffusion_loss': 0.388486385345459, 'epoch': 3.59}

 36%|███▌      | 262/730 [23:18<41:39,  5.34s/it]
                                                 
{'train/learning_rate_real': 1.855688587915813e-05, 'epoch': 3.59}

 36%|███▌      | 262/730 [23:18<41:39,  5.34s/it]
 36%|███▌      | 263/730 [23:18<41:23,  5.32s/it]
                                                 
{'debug/num_tok_total': 2426.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2426.0, 'debug/num_lat_loss': 1777.0, 'epoch': 3.6}

 36%|███▌      | 263/730 [23:20<41:23,  5.32s/it]
                                                 
{'train/ce_loss': 2.109375, 'train/diffusion_loss': 0.447339802980423, 'epoch': 3.6}

 36%|███▌      | 263/730 [23:20<41:23,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.8508306825403504e-05, 'epoch': 3.6}

 36%|███▌      | 263/730 [23:20<41:23,  5.32s/it]
                                                 
{'debug/num_tok_total': 2696.0, 'debug/num_tok_loss': 1590.0, 'debug/num_lat_total': 2696.0, 'debug/num_lat_loss': 1590.0, 'epoch': 3.6}

 36%|███▌      | 263/730 [23:21<41:23,  5.32s/it]
                                                 
{'train/ce_loss': 2.203125, 'train/diffusion_loss': 0.3028818666934967, 'epoch': 3.6}

 36%|███▌      | 263/730 [23:21<41:23,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.8508306825403504e-05, 'epoch': 3.6}

 36%|███▌      | 263/730 [23:21<41:23,  5.32s/it]
                                                 
{'debug/num_tok_total': 2880.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2880.0, 'debug/num_lat_loss': 1795.0, 'epoch': 3.6}

 36%|███▌      | 263/730 [23:22<41:23,  5.32s/it]
                                                 
{'train/ce_loss': 2.3125, 'train/diffusion_loss': 0.3490007519721985, 'epoch': 3.6}

 36%|███▌      | 263/730 [23:22<41:23,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.8508306825403504e-05, 'epoch': 3.6}

 36%|███▌      | 263/730 [23:22<41:23,  5.32s/it]
                                                 
{'debug/num_tok_total': 2440.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2440.0, 'debug/num_lat_loss': 1793.0, 'epoch': 3.6}

 36%|███▌      | 263/730 [23:24<41:23,  5.32s/it]
                                                 
{'train/ce_loss': 2.1875, 'train/diffusion_loss': 0.45419830083847046, 'epoch': 3.6}

 36%|███▌      | 263/730 [23:24<41:23,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.8508306825403504e-05, 'epoch': 3.6}

 36%|███▌      | 263/730 [23:24<41:23,  5.32s/it]
 36%|███▌      | 264/730 [23:24<41:41,  5.37s/it]
                                                 
{'debug/num_tok_total': 2809.0, 'debug/num_tok_loss': 1754.0, 'debug/num_lat_total': 2809.0, 'debug/num_lat_loss': 1754.0, 'epoch': 3.62}

 36%|███▌      | 264/730 [23:25<41:41,  5.37s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.35602056980133057, 'epoch': 3.62}

 36%|███▌      | 264/730 [23:25<41:41,  5.37s/it]
                                                 
{'train/learning_rate_real': 1.8459609471682816e-05, 'epoch': 3.62}

 36%|███▌      | 264/730 [23:25<41:41,  5.37s/it]
                                                 
{'debug/num_tok_total': 3066.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 3066.0, 'debug/num_lat_loss': 1776.0, 'epoch': 3.62}

 36%|███▌      | 264/730 [23:27<41:41,  5.37s/it]
                                                 
{'train/ce_loss': 2.125, 'train/diffusion_loss': 0.3372049629688263, 'epoch': 3.62}

 36%|███▌      | 264/730 [23:27<41:41,  5.37s/it]
                                                 
{'train/learning_rate_real': 1.8459609471682816e-05, 'epoch': 3.62}

 36%|███▌      | 264/730 [23:27<41:41,  5.37s/it]
                                                 
{'debug/num_tok_total': 3320.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 3320.0, 'debug/num_lat_loss': 1799.0, 'epoch': 3.62}

 36%|███▌      | 264/730 [23:28<41:41,  5.37s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.254958838224411, 'epoch': 3.62}

 36%|███▌      | 264/730 [23:28<41:41,  5.37s/it]
                                                 
{'train/learning_rate_real': 1.8459609471682816e-05, 'epoch': 3.62}

 36%|███▌      | 264/730 [23:28<41:41,  5.37s/it]
                                                 
{'debug/num_tok_total': 3562.0, 'debug/num_tok_loss': 1811.0, 'debug/num_lat_total': 3562.0, 'debug/num_lat_loss': 1811.0, 'epoch': 3.62}

 36%|███▌      | 264/730 [23:29<41:41,  5.37s/it]
                                                 
{'train/ce_loss': 2.203125, 'train/diffusion_loss': 0.2555168569087982, 'epoch': 3.62}

 36%|███▌      | 264/730 [23:29<41:41,  5.37s/it]
                                                 
{'train/learning_rate_real': 1.8459609471682816e-05, 'epoch': 3.62}

 36%|███▌      | 264/730 [23:29<41:41,  5.37s/it]
 36%|███▋      | 265/730 [23:30<42:54,  5.54s/it]
                                                 
{'debug/num_tok_total': 2649.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2649.0, 'debug/num_lat_loss': 1789.0, 'epoch': 3.63}

 36%|███▋      | 265/730 [23:31<42:54,  5.54s/it]
                                                 
{'train/ce_loss': 2.125, 'train/diffusion_loss': 0.3796527087688446, 'epoch': 3.63}

 36%|███▋      | 265/730 [23:31<42:54,  5.54s/it]
                                                 
{'train/learning_rate_real': 1.841079477681782e-05, 'epoch': 3.63}

 36%|███▋      | 265/730 [23:31<42:54,  5.54s/it]
                                                 
{'debug/num_tok_total': 2466.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2466.0, 'debug/num_lat_loss': 1799.0, 'epoch': 3.63}

 36%|███▋      | 265/730 [23:32<42:54,  5.54s/it]
                                                 
{'train/ce_loss': 2.109375, 'train/diffusion_loss': 0.43377572298049927, 'epoch': 3.63}

 36%|███▋      | 265/730 [23:32<42:54,  5.54s/it]
                                                 
{'train/learning_rate_real': 1.841079477681782e-05, 'epoch': 3.63}

 36%|███▋      | 265/730 [23:32<42:54,  5.54s/it]
                                                 
{'debug/num_tok_total': 2607.0, 'debug/num_tok_loss': 1757.0, 'debug/num_lat_total': 2607.0, 'debug/num_lat_loss': 1757.0, 'epoch': 3.63}

 36%|███▋      | 265/730 [23:33<42:54,  5.54s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.3975103199481964, 'epoch': 3.63}

 36%|███▋      | 265/730 [23:33<42:54,  5.54s/it]
                                                 
{'train/learning_rate_real': 1.841079477681782e-05, 'epoch': 3.63}

 36%|███▋      | 265/730 [23:33<42:54,  5.54s/it]
                                                 
{'debug/num_tok_total': 2435.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2435.0, 'debug/num_lat_loss': 1799.0, 'epoch': 3.63}

 36%|███▋      | 265/730 [23:35<42:54,  5.54s/it]
                                                 
{'train/ce_loss': 2.265625, 'train/diffusion_loss': 0.46887922286987305, 'epoch': 3.63}

 36%|███▋      | 265/730 [23:35<42:54,  5.54s/it]
                                                 
{'train/learning_rate_real': 1.841079477681782e-05, 'epoch': 3.63}

 36%|███▋      | 265/730 [23:35<42:54,  5.54s/it]
 36%|███▋      | 266/730 [23:35<41:54,  5.42s/it]
                                                 
{'debug/num_tok_total': 1820.0, 'debug/num_tok_loss': 1611.0, 'debug/num_lat_total': 1820.0, 'debug/num_lat_loss': 1611.0, 'epoch': 3.64}

 36%|███▋      | 266/730 [23:36<41:54,  5.42s/it]
                                                 
{'train/ce_loss': 2.3125, 'train/diffusion_loss': 0.5174155235290527, 'epoch': 3.64}

 36%|███▋      | 266/730 [23:36<41:54,  5.42s/it]
                                                 
{'train/learning_rate_real': 1.8361863701940642e-05, 'epoch': 3.64}

 36%|███▋      | 266/730 [23:36<41:54,  5.42s/it]
                                                 
{'debug/num_tok_total': 2195.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2195.0, 'debug/num_lat_loss': 1773.0, 'epoch': 3.64}

 36%|███▋      | 266/730 [23:37<41:54,  5.42s/it]
                                                 
{'train/ce_loss': 2.25, 'train/diffusion_loss': 0.49644601345062256, 'epoch': 3.64}

 36%|███▋      | 266/730 [23:37<41:54,  5.42s/it]
                                                 
{'train/learning_rate_real': 1.8361863701940642e-05, 'epoch': 3.64}

 36%|███▋      | 266/730 [23:37<41:54,  5.42s/it]
                                                 
{'debug/num_tok_total': 2215.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2215.0, 'debug/num_lat_loss': 1788.0, 'epoch': 3.64}

 36%|███▋      | 266/730 [23:38<41:54,  5.42s/it]
                                                 
{'train/ce_loss': 2.078125, 'train/diffusion_loss': 0.4839715361595154, 'epoch': 3.64}

 36%|███▋      | 266/730 [23:38<41:54,  5.42s/it]
                                                 
{'train/learning_rate_real': 1.8361863701940642e-05, 'epoch': 3.64}

 36%|███▋      | 266/730 [23:38<41:54,  5.42s/it]
                                                 
{'debug/num_tok_total': 2000.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2000.0, 'debug/num_lat_loss': 1780.0, 'epoch': 3.64}

 36%|███▋      | 266/730 [23:39<41:54,  5.42s/it]
                                                 
{'train/ce_loss': 2.25, 'train/diffusion_loss': 0.538507878780365, 'epoch': 3.64}

 36%|███▋      | 266/730 [23:39<41:54,  5.42s/it]
                                                 
{'train/learning_rate_real': 1.8361863701940642e-05, 'epoch': 3.64}

 36%|███▋      | 266/730 [23:39<41:54,  5.42s/it]
 37%|███▋      | 267/730 [23:40<40:17,  5.22s/it]
                                                 
{'debug/num_tok_total': 2395.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2395.0, 'debug/num_lat_loss': 1766.0, 'epoch': 3.66}

 37%|███▋      | 267/730 [23:41<40:17,  5.22s/it]
                                                 
{'train/ce_loss': 1.9921875, 'train/diffusion_loss': 0.4636920392513275, 'epoch': 3.66}

 37%|███▋      | 267/730 [23:41<40:17,  5.22s/it]
                                                 
{'train/learning_rate_real': 1.831281721047487e-05, 'epoch': 3.66}

 37%|███▋      | 267/730 [23:41<40:17,  5.22s/it]
                                                 
{'debug/num_tok_total': 3290.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 3290.0, 'debug/num_lat_loss': 1783.0, 'epoch': 3.66}

 37%|███▋      | 267/730 [23:42<40:17,  5.22s/it]
                                                 
{'train/ce_loss': 2.09375, 'train/diffusion_loss': 0.2822487950325012, 'epoch': 3.66}

 37%|███▋      | 267/730 [23:42<40:17,  5.22s/it]
                                                 
{'train/learning_rate_real': 1.831281721047487e-05, 'epoch': 3.66}

 37%|███▋      | 267/730 [23:42<40:17,  5.22s/it]
                                                 
{'debug/num_tok_total': 2268.0, 'debug/num_tok_loss': 1706.0, 'debug/num_lat_total': 2268.0, 'debug/num_lat_loss': 1706.0, 'epoch': 3.66}

 37%|███▋      | 267/730 [23:43<40:17,  5.22s/it]
                                                 
{'train/ce_loss': 2.1875, 'train/diffusion_loss': 0.45802122354507446, 'epoch': 3.66}

 37%|███▋      | 267/730 [23:43<40:17,  5.22s/it]
                                                 
{'train/learning_rate_real': 1.831281721047487e-05, 'epoch': 3.66}

 37%|███▋      | 267/730 [23:43<40:17,  5.22s/it]
                                                 
{'debug/num_tok_total': 2398.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2398.0, 'debug/num_lat_loss': 1770.0, 'epoch': 3.66}

 37%|███▋      | 267/730 [23:45<40:17,  5.22s/it]
                                                 
{'train/ce_loss': 2.234375, 'train/diffusion_loss': 0.44514697790145874, 'epoch': 3.66}

 37%|███▋      | 267/730 [23:45<40:17,  5.22s/it]
                                                 
{'train/learning_rate_real': 1.831281721047487e-05, 'epoch': 3.66}

 37%|███▋      | 267/730 [23:45<40:17,  5.22s/it]
 37%|███▋      | 268/730 [23:45<40:02,  5.20s/it]
                                                 
{'debug/num_tok_total': 2853.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2853.0, 'debug/num_lat_loss': 1777.0, 'epoch': 3.67}

 37%|███▋      | 268/730 [23:46<40:02,  5.20s/it]
                                                 
{'train/ce_loss': 2.125, 'train/diffusion_loss': 0.3616383671760559, 'epoch': 3.67}

 37%|███▋      | 268/730 [23:46<40:02,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.8263656268116576e-05, 'epoch': 3.67}

 37%|███▋      | 268/730 [23:46<40:02,  5.20s/it]
                                                 
{'debug/num_tok_total': 2850.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2850.0, 'debug/num_lat_loss': 1775.0, 'epoch': 3.67}

 37%|███▋      | 268/730 [23:47<40:02,  5.20s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.37581169605255127, 'epoch': 3.67}

 37%|███▋      | 268/730 [23:47<40:02,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.8263656268116576e-05, 'epoch': 3.67}

 37%|███▋      | 268/730 [23:47<40:02,  5.20s/it]
                                                 
{'debug/num_tok_total': 2891.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2891.0, 'debug/num_lat_loss': 1796.0, 'epoch': 3.67}

 37%|███▋      | 268/730 [23:49<40:02,  5.20s/it]
                                                 
{'train/ce_loss': 2.359375, 'train/diffusion_loss': 0.33970674872398376, 'epoch': 3.67}

 37%|███▋      | 268/730 [23:49<40:02,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.8263656268116576e-05, 'epoch': 3.67}

 37%|███▋      | 268/730 [23:49<40:02,  5.20s/it]
                                                 
{'debug/num_tok_total': 2232.0, 'debug/num_tok_loss': 1798.0, 'debug/num_lat_total': 2232.0, 'debug/num_lat_loss': 1798.0, 'epoch': 3.67}

 37%|███▋      | 268/730 [23:50<40:02,  5.20s/it]
                                                 
{'train/ce_loss': 2.078125, 'train/diffusion_loss': 0.4902125298976898, 'epoch': 3.67}

 37%|███▋      | 268/730 [23:50<40:02,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.8263656268116576e-05, 'epoch': 3.67}

 37%|███▋      | 268/730 [23:50<40:02,  5.20s/it]
 37%|███▋      | 269/730 [23:50<40:07,  5.22s/it]
                                                 
{'debug/num_tok_total': 2194.0, 'debug/num_tok_loss': 1762.0, 'debug/num_lat_total': 2194.0, 'debug/num_lat_loss': 1762.0, 'epoch': 3.68}

 37%|███▋      | 269/730 [23:51<40:07,  5.22s/it]
                                                 
{'train/ce_loss': 2.328125, 'train/diffusion_loss': 0.46788713335990906, 'epoch': 3.68}

 37%|███▋      | 269/730 [23:51<40:07,  5.22s/it]
                                                 
{'train/learning_rate_real': 1.8214381842815293e-05, 'epoch': 3.68}

 37%|███▋      | 269/730 [23:51<40:07,  5.22s/it]
                                                 
{'debug/num_tok_total': 2014.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2014.0, 'debug/num_lat_loss': 1795.0, 'epoch': 3.68}

 37%|███▋      | 269/730 [23:52<40:07,  5.22s/it]
                                                 
{'train/ce_loss': 2.125, 'train/diffusion_loss': 0.5531131625175476, 'epoch': 3.68}

 37%|███▋      | 269/730 [23:52<40:07,  5.22s/it]
                                                 
{'train/learning_rate_real': 1.8214381842815293e-05, 'epoch': 3.68}

 37%|███▋      | 269/730 [23:52<40:07,  5.22s/it]
                                                 
{'debug/num_tok_total': 2081.0, 'debug/num_tok_loss': 1646.0, 'debug/num_lat_total': 2081.0, 'debug/num_lat_loss': 1646.0, 'epoch': 3.68}

 37%|███▋      | 269/730 [23:54<40:07,  5.22s/it]
                                                 
{'train/ce_loss': 2.125, 'train/diffusion_loss': 0.4983122944831848, 'epoch': 3.68}

 37%|███▋      | 269/730 [23:54<40:07,  5.22s/it]
                                                 
{'train/learning_rate_real': 1.8214381842815293e-05, 'epoch': 3.68}

 37%|███▋      | 269/730 [23:54<40:07,  5.22s/it]
                                                 
{'debug/num_tok_total': 2674.0, 'debug/num_tok_loss': 1672.0, 'debug/num_lat_total': 2674.0, 'debug/num_lat_loss': 1672.0, 'epoch': 3.68}

 37%|███▋      | 269/730 [23:55<40:07,  5.22s/it]
                                                 
{'train/ce_loss': 2.28125, 'train/diffusion_loss': 0.3529197573661804, 'epoch': 3.68}

 37%|███▋      | 269/730 [23:55<40:07,  5.22s/it]
                                                 
{'train/learning_rate_real': 1.8214381842815293e-05, 'epoch': 3.68}

 37%|███▋      | 269/730 [23:55<40:07,  5.22s/it]03/16/2026 07:11:27 - INFO - __main__ - LoRA debug step 270: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 37%|███▋      | 270/730 [23:55<39:46,  5.19s/it]
                                                 
{'loss': 2.6522, 'grad_norm': 1.421845555305481, 'learning_rate': 1.8214381842815293e-05, 'epoch': 3.7}

 37%|███▋      | 270/730 [23:55<39:46,  5.19s/it]
                                                 
{'debug/num_tok_total': 2646.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2646.0, 'debug/num_lat_loss': 1783.0, 'epoch': 3.7}

 37%|███▋      | 270/730 [23:57<39:46,  5.19s/it]
                                                 
{'train/ce_loss': 2.140625, 'train/diffusion_loss': 0.41246217489242554, 'epoch': 3.7}

 37%|███▋      | 270/730 [23:57<39:46,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.8164994904754966e-05, 'epoch': 3.7}

 37%|███▋      | 270/730 [23:57<39:46,  5.19s/it]
                                                 
{'debug/num_tok_total': 2858.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2858.0, 'debug/num_lat_loss': 1787.0, 'epoch': 3.7}

 37%|███▋      | 270/730 [23:58<39:46,  5.19s/it]
                                                 
{'train/ce_loss': 2.078125, 'train/diffusion_loss': 0.3775860667228699, 'epoch': 3.7}

 37%|███▋      | 270/730 [23:58<39:46,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.8164994904754966e-05, 'epoch': 3.7}

 37%|███▋      | 270/730 [23:58<39:46,  5.19s/it]
                                                 
{'debug/num_tok_total': 2525.0, 'debug/num_tok_loss': 1659.0, 'debug/num_lat_total': 2525.0, 'debug/num_lat_loss': 1659.0, 'epoch': 3.7}

 37%|███▋      | 270/730 [23:59<39:46,  5.19s/it]
                                                 
{'train/ce_loss': 2.171875, 'train/diffusion_loss': 0.39445722103118896, 'epoch': 3.7}

 37%|███▋      | 270/730 [23:59<39:46,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.8164994904754966e-05, 'epoch': 3.7}

 37%|███▋      | 270/730 [23:59<39:46,  5.19s/it]
                                                 
{'debug/num_tok_total': 3281.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 3281.0, 'debug/num_lat_loss': 1777.0, 'epoch': 3.7}

 37%|███▋      | 270/730 [24:00<39:46,  5.19s/it]
                                                 
{'train/ce_loss': 2.046875, 'train/diffusion_loss': 0.27408111095428467, 'epoch': 3.7}

 37%|███▋      | 270/730 [24:00<39:46,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.8164994904754966e-05, 'epoch': 3.7}

 37%|███▋      | 270/730 [24:00<39:46,  5.19s/it]
 37%|███▋      | 271/730 [24:01<40:13,  5.26s/it]
                                                 
{'debug/num_tok_total': 2631.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2631.0, 'debug/num_lat_loss': 1784.0, 'epoch': 3.71}

 37%|███▋      | 271/730 [24:02<40:13,  5.26s/it]
                                                 
{'train/ce_loss': 2.125, 'train/diffusion_loss': 0.41432589292526245, 'epoch': 3.71}

 37%|███▋      | 271/730 [24:02<40:13,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.811549642633486e-05, 'epoch': 3.71}

 37%|███▋      | 271/730 [24:02<40:13,  5.26s/it]
                                                 
{'debug/num_tok_total': 2817.0, 'debug/num_tok_loss': 1651.0, 'debug/num_lat_total': 2817.0, 'debug/num_lat_loss': 1651.0, 'epoch': 3.71}

 37%|███▋      | 271/730 [24:03<40:13,  5.26s/it]
                                                 
{'train/ce_loss': 2.1875, 'train/diffusion_loss': 0.311215341091156, 'epoch': 3.71}

 37%|███▋      | 271/730 [24:03<40:13,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.811549642633486e-05, 'epoch': 3.71}

 37%|███▋      | 271/730 [24:03<40:13,  5.26s/it]
                                                 
{'debug/num_tok_total': 2315.0, 'debug/num_tok_loss': 1673.0, 'debug/num_lat_total': 2315.0, 'debug/num_lat_loss': 1673.0, 'epoch': 3.71}

 37%|███▋      | 271/730 [24:04<40:13,  5.26s/it]
                                                 
{'train/ce_loss': 2.171875, 'train/diffusion_loss': 0.41784414649009705, 'epoch': 3.71}

 37%|███▋      | 271/730 [24:04<40:13,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.811549642633486e-05, 'epoch': 3.71}

 37%|███▋      | 271/730 [24:04<40:13,  5.26s/it]
                                                 
{'debug/num_tok_total': 2860.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2860.0, 'debug/num_lat_loss': 1779.0, 'epoch': 3.71}

 37%|███▋      | 271/730 [24:06<40:13,  5.26s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.37519437074661255, 'epoch': 3.71}

 37%|███▋      | 271/730 [24:06<40:13,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.811549642633486e-05, 'epoch': 3.71}

 37%|███▋      | 271/730 [24:06<40:13,  5.26s/it]
 37%|███▋      | 272/730 [24:06<40:20,  5.29s/it]
                                                 
{'debug/num_tok_total': 2240.0, 'debug/num_tok_loss': 1809.0, 'debug/num_lat_total': 2240.0, 'debug/num_lat_loss': 1809.0, 'epoch': 3.73}

 37%|███▋      | 272/730 [24:07<40:20,  5.29s/it]
                                                 
{'train/ce_loss': 2.03125, 'train/diffusion_loss': 0.5138973593711853, 'epoch': 3.73}

 37%|███▋      | 272/730 [24:07<40:20,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.8065887382150394e-05, 'epoch': 3.73}

 37%|███▋      | 272/730 [24:07<40:20,  5.29s/it]
                                                 
{'debug/num_tok_total': 2850.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2850.0, 'debug/num_lat_loss': 1786.0, 'epoch': 3.73}

 37%|███▋      | 272/730 [24:08<40:20,  5.29s/it]
                                                 
{'train/ce_loss': 2.140625, 'train/diffusion_loss': 0.36435139179229736, 'epoch': 3.73}

 37%|███▋      | 272/730 [24:08<40:20,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.8065887382150394e-05, 'epoch': 3.73}

 37%|███▋      | 272/730 [24:08<40:20,  5.29s/it]
                                                 
{'debug/num_tok_total': 2483.0, 'debug/num_tok_loss': 1602.0, 'debug/num_lat_total': 2483.0, 'debug/num_lat_loss': 1602.0, 'epoch': 3.73}

 37%|███▋      | 272/730 [24:10<40:20,  5.29s/it]
                                                 
{'train/ce_loss': 2.1875, 'train/diffusion_loss': 0.36120906472206116, 'epoch': 3.73}

 37%|███▋      | 272/730 [24:10<40:20,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.8065887382150394e-05, 'epoch': 3.73}

 37%|███▋      | 272/730 [24:10<40:20,  5.29s/it]
                                                 
{'debug/num_tok_total': 2507.0, 'debug/num_tok_loss': 1656.0, 'debug/num_lat_total': 2507.0, 'debug/num_lat_loss': 1656.0, 'epoch': 3.73}

 37%|███▋      | 272/730 [24:11<40:20,  5.29s/it]
                                                 
{'train/ce_loss': 2.09375, 'train/diffusion_loss': 0.3839026093482971, 'epoch': 3.73}

 37%|███▋      | 272/730 [24:11<40:20,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.8065887382150394e-05, 'epoch': 3.73}

 37%|███▋      | 272/730 [24:11<40:20,  5.29s/it]
 37%|███▋      | 273/730 [24:11<39:43,  5.22s/it]
                                                 
{'debug/num_tok_total': 3089.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 3089.0, 'debug/num_lat_loss': 1794.0, 'epoch': 3.74}

 37%|███▋      | 273/730 [24:12<39:43,  5.22s/it]
                                                 
{'train/ce_loss': 2.171875, 'train/diffusion_loss': 0.28552016615867615, 'epoch': 3.74}

 37%|███▋      | 273/730 [24:12<39:43,  5.22s/it]
                                                 
{'train/learning_rate_real': 1.801616874897396e-05, 'epoch': 3.74}

 37%|███▋      | 273/730 [24:12<39:43,  5.22s/it]
                                                 
{'debug/num_tok_total': 3122.0, 'debug/num_tok_loss': 1807.0, 'debug/num_lat_total': 3122.0, 'debug/num_lat_loss': 1807.0, 'epoch': 3.74}

 37%|███▋      | 273/730 [24:14<39:43,  5.22s/it]
                                                 
{'train/ce_loss': 2.078125, 'train/diffusion_loss': 0.28701597452163696, 'epoch': 3.74}

 37%|███▋      | 273/730 [24:14<39:43,  5.22s/it]
                                                 
{'train/learning_rate_real': 1.801616874897396e-05, 'epoch': 3.74}

 37%|███▋      | 273/730 [24:14<39:43,  5.22s/it]
                                                 
{'debug/num_tok_total': 2696.0, 'debug/num_tok_loss': 1811.0, 'debug/num_lat_total': 2696.0, 'debug/num_lat_loss': 1811.0, 'epoch': 3.74}

 37%|███▋      | 273/730 [24:15<39:43,  5.22s/it]
                                                 
{'train/ce_loss': 2.25, 'train/diffusion_loss': 0.3732485771179199, 'epoch': 3.74}

 37%|███▋      | 273/730 [24:15<39:43,  5.22s/it]
                                                 
{'train/learning_rate_real': 1.801616874897396e-05, 'epoch': 3.74}

 37%|███▋      | 273/730 [24:15<39:43,  5.22s/it]
                                                 
{'debug/num_tok_total': 2415.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2415.0, 'debug/num_lat_loss': 1766.0, 'epoch': 3.74}

 37%|███▋      | 273/730 [24:16<39:43,  5.22s/it]
                                                 
{'train/ce_loss': 2.109375, 'train/diffusion_loss': 0.4355056583881378, 'epoch': 3.74}

 37%|███▋      | 273/730 [24:16<39:43,  5.22s/it]
                                                 
{'train/learning_rate_real': 1.801616874897396e-05, 'epoch': 3.74}

 37%|███▋      | 273/730 [24:16<39:43,  5.22s/it]
 38%|███▊      | 274/730 [24:17<40:00,  5.26s/it]
                                                 
{'debug/num_tok_total': 2215.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2215.0, 'debug/num_lat_loss': 1775.0, 'epoch': 3.75}

 38%|███▊      | 274/730 [24:18<40:00,  5.26s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.4883570373058319, 'epoch': 3.75}

 38%|███▊      | 274/730 [24:18<40:00,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.7966341505735695e-05, 'epoch': 3.75}

 38%|███▊      | 274/730 [24:18<40:00,  5.26s/it]
                                                 
{'debug/num_tok_total': 3005.0, 'debug/num_tok_loss': 1740.0, 'debug/num_lat_total': 3005.0, 'debug/num_lat_loss': 1740.0, 'epoch': 3.75}

 38%|███▊      | 274/730 [24:19<40:00,  5.26s/it]
                                                 
{'train/ce_loss': 2.03125, 'train/diffusion_loss': 0.2945389747619629, 'epoch': 3.75}

 38%|███▊      | 274/730 [24:19<40:00,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.7966341505735695e-05, 'epoch': 3.75}

 38%|███▊      | 274/730 [24:19<40:00,  5.26s/it]
                                                 
{'debug/num_tok_total': 2434.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2434.0, 'debug/num_lat_loss': 1779.0, 'epoch': 3.75}

 38%|███▊      | 274/730 [24:20<40:00,  5.26s/it]
                                                 
{'train/ce_loss': 2.234375, 'train/diffusion_loss': 0.4722411036491394, 'epoch': 3.75}

 38%|███▊      | 274/730 [24:20<40:00,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.7966341505735695e-05, 'epoch': 3.75}

 38%|███▊      | 274/730 [24:20<40:00,  5.26s/it]
                                                 
{'debug/num_tok_total': 2676.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2676.0, 'debug/num_lat_loss': 1789.0, 'epoch': 3.75}

 38%|███▊      | 274/730 [24:21<40:00,  5.26s/it]
                                                 
{'train/ce_loss': 2.234375, 'train/diffusion_loss': 0.3966304659843445, 'epoch': 3.75}

 38%|███▊      | 274/730 [24:21<40:00,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.7966341505735695e-05, 'epoch': 3.75}

 38%|███▊      | 274/730 [24:21<40:00,  5.26s/it]
 38%|███▊      | 275/730 [24:22<39:49,  5.25s/it]
                                                 
{'debug/num_tok_total': 2198.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2198.0, 'debug/num_lat_loss': 1781.0, 'epoch': 3.77}

 38%|███▊      | 275/730 [24:23<39:49,  5.25s/it]
                                                 
{'train/ce_loss': 2.1875, 'train/diffusion_loss': 0.5182521939277649, 'epoch': 3.77}

 38%|███▊      | 275/730 [24:23<39:49,  5.25s/it]
                                                 
{'train/learning_rate_real': 1.79164066335042e-05, 'epoch': 3.77}

 38%|███▊      | 275/730 [24:23<39:49,  5.25s/it]
                                                 
{'debug/num_tok_total': 2640.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2640.0, 'debug/num_lat_loss': 1771.0, 'epoch': 3.77}

 38%|███▊      | 275/730 [24:24<39:49,  5.25s/it]
                                                 
{'train/ce_loss': 2.046875, 'train/diffusion_loss': 0.35757267475128174, 'epoch': 3.77}

 38%|███▊      | 275/730 [24:24<39:49,  5.25s/it]
                                                 
{'train/learning_rate_real': 1.79164066335042e-05, 'epoch': 3.77}

 38%|███▊      | 275/730 [24:24<39:49,  5.25s/it]
                                                 
{'debug/num_tok_total': 2456.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2456.0, 'debug/num_lat_loss': 1796.0, 'epoch': 3.77}

 38%|███▊      | 275/730 [24:25<39:49,  5.25s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.43631818890571594, 'epoch': 3.77}

 38%|███▊      | 275/730 [24:25<39:49,  5.25s/it]
                                                 
{'train/learning_rate_real': 1.79164066335042e-05, 'epoch': 3.77}

 38%|███▊      | 275/730 [24:25<39:49,  5.25s/it]
                                                 
{'debug/num_tok_total': 2419.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2419.0, 'debug/num_lat_loss': 1763.0, 'epoch': 3.77}

 38%|███▊      | 275/730 [24:26<39:49,  5.25s/it]
                                                 
{'train/ce_loss': 2.359375, 'train/diffusion_loss': 0.4431777894496918, 'epoch': 3.77}

 38%|███▊      | 275/730 [24:26<39:49,  5.25s/it]
                                                 
{'train/learning_rate_real': 1.79164066335042e-05, 'epoch': 3.77}

 38%|███▊      | 275/730 [24:26<39:49,  5.25s/it]
 38%|███▊      | 276/730 [24:27<39:26,  5.21s/it]
                                                 
{'debug/num_tok_total': 3064.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 3064.0, 'debug/num_lat_loss': 1777.0, 'epoch': 3.78}

 38%|███▊      | 276/730 [24:28<39:26,  5.21s/it]
                                                 
{'train/ce_loss': 2.078125, 'train/diffusion_loss': 0.29707273840904236, 'epoch': 3.78}

 38%|███▊      | 276/730 [24:28<39:26,  5.21s/it]
                                                 
{'train/learning_rate_real': 1.7866365115467233e-05, 'epoch': 3.78}

 38%|███▊      | 276/730 [24:28<39:26,  5.21s/it]
                                                 
{'debug/num_tok_total': 2684.0, 'debug/num_tok_loss': 1617.0, 'debug/num_lat_total': 2684.0, 'debug/num_lat_loss': 1617.0, 'epoch': 3.78}

 38%|███▊      | 276/730 [24:29<39:26,  5.21s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.35491320490837097, 'epoch': 3.78}

 38%|███▊      | 276/730 [24:29<39:26,  5.21s/it]
                                                 
{'train/learning_rate_real': 1.7866365115467233e-05, 'epoch': 3.78}

 38%|███▊      | 276/730 [24:29<39:26,  5.21s/it]
                                                 
{'debug/num_tok_total': 2858.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2858.0, 'debug/num_lat_loss': 1780.0, 'epoch': 3.78}

 38%|███▊      | 276/730 [24:31<39:26,  5.21s/it]
                                                 
{'train/ce_loss': 2.15625, 'train/diffusion_loss': 0.340061217546463, 'epoch': 3.78}

 38%|███▊      | 276/730 [24:31<39:26,  5.21s/it]
                                                 
{'train/learning_rate_real': 1.7866365115467233e-05, 'epoch': 3.78}

 38%|███▊      | 276/730 [24:31<39:26,  5.21s/it]
                                                 
{'debug/num_tok_total': 2353.0, 'debug/num_tok_loss': 1633.0, 'debug/num_lat_total': 2353.0, 'debug/num_lat_loss': 1633.0, 'epoch': 3.78}

 38%|███▊      | 276/730 [24:32<39:26,  5.21s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.387778639793396, 'epoch': 3.78}

 38%|███▊      | 276/730 [24:32<39:26,  5.21s/it]
                                                 
{'train/learning_rate_real': 1.7866365115467233e-05, 'epoch': 3.78}

 38%|███▊      | 276/730 [24:32<39:26,  5.21s/it]
 38%|███▊      | 277/730 [24:32<40:01,  5.30s/it]
                                                 
{'debug/num_tok_total': 2425.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2425.0, 'debug/num_lat_loss': 1783.0, 'epoch': 3.79}

 38%|███▊      | 277/730 [24:33<40:01,  5.30s/it]
                                                 
{'train/ce_loss': 2.109375, 'train/diffusion_loss': 0.4345017671585083, 'epoch': 3.79}

 38%|███▊      | 277/730 [24:34<40:01,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.781621793691234e-05, 'epoch': 3.79}

 38%|███▊      | 277/730 [24:34<40:01,  5.30s/it]
                                                 
{'debug/num_tok_total': 2424.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2424.0, 'debug/num_lat_loss': 1780.0, 'epoch': 3.79}

 38%|███▊      | 277/730 [24:35<40:01,  5.30s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.4521588981151581, 'epoch': 3.79}

 38%|███▊      | 277/730 [24:35<40:01,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.781621793691234e-05, 'epoch': 3.79}

 38%|███▊      | 277/730 [24:35<40:01,  5.30s/it]
                                                 
{'debug/num_tok_total': 2387.0, 'debug/num_tok_loss': 1513.0, 'debug/num_lat_total': 2387.0, 'debug/num_lat_loss': 1513.0, 'epoch': 3.79}

 38%|███▊      | 277/730 [24:36<40:01,  5.30s/it]
                                                 
{'train/ce_loss': 2.328125, 'train/diffusion_loss': 0.3901711404323578, 'epoch': 3.79}

 38%|███▊      | 277/730 [24:36<40:01,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.781621793691234e-05, 'epoch': 3.79}

 38%|███▊      | 277/730 [24:36<40:01,  5.30s/it]
                                                 
{'debug/num_tok_total': 2876.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2876.0, 'debug/num_lat_loss': 1793.0, 'epoch': 3.79}

 38%|███▊      | 277/730 [24:37<40:01,  5.30s/it]
                                                 
{'train/ce_loss': 2.109375, 'train/diffusion_loss': 0.3412567973136902, 'epoch': 3.79}

 38%|███▊      | 277/730 [24:37<40:01,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.781621793691234e-05, 'epoch': 3.79}

 38%|███▊      | 277/730 [24:37<40:01,  5.30s/it]
 38%|███▊      | 278/730 [24:38<39:39,  5.26s/it]
                                                 
{'debug/num_tok_total': 2611.0, 'debug/num_tok_loss': 1662.0, 'debug/num_lat_total': 2611.0, 'debug/num_lat_loss': 1662.0, 'epoch': 3.81}

 38%|███▊      | 278/730 [24:39<39:39,  5.26s/it]
                                                 
{'train/ce_loss': 2.140625, 'train/diffusion_loss': 0.3693002462387085, 'epoch': 3.81}

 38%|███▊      | 278/730 [24:39<39:39,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.7765966085207448e-05, 'epoch': 3.81}

 38%|███▊      | 278/730 [24:39<39:39,  5.26s/it]
                                                 
{'debug/num_tok_total': 2808.0, 'debug/num_tok_loss': 1758.0, 'debug/num_lat_total': 2808.0, 'debug/num_lat_loss': 1758.0, 'epoch': 3.81}

 38%|███▊      | 278/730 [24:40<39:39,  5.26s/it]
                                                 
{'train/ce_loss': 2.09375, 'train/diffusion_loss': 0.35720962285995483, 'epoch': 3.81}

 38%|███▊      | 278/730 [24:40<39:39,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.7765966085207448e-05, 'epoch': 3.81}

 38%|███▊      | 278/730 [24:40<39:39,  5.26s/it]
                                                 
{'debug/num_tok_total': 2622.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 2622.0, 'debug/num_lat_loss': 1765.0, 'epoch': 3.81}

 38%|███▊      | 278/730 [24:41<39:39,  5.26s/it]
                                                 
{'train/ce_loss': 2.125, 'train/diffusion_loss': 0.42649921774864197, 'epoch': 3.81}

 38%|███▊      | 278/730 [24:41<39:39,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.7765966085207448e-05, 'epoch': 3.81}

 38%|███▊      | 278/730 [24:41<39:39,  5.26s/it]
                                                 
{'debug/num_tok_total': 2686.0, 'debug/num_tok_loss': 1810.0, 'debug/num_lat_total': 2686.0, 'debug/num_lat_loss': 1810.0, 'epoch': 3.81}

 38%|███▊      | 278/730 [24:43<39:39,  5.26s/it]
                                                 
{'train/ce_loss': 2.078125, 'train/diffusion_loss': 0.37720000743865967, 'epoch': 3.81}

 38%|███▊      | 278/730 [24:43<39:39,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.7765966085207448e-05, 'epoch': 3.81}

 38%|███▊      | 278/730 [24:43<39:39,  5.26s/it]
 38%|███▊      | 279/730 [24:43<39:58,  5.32s/it]
                                                 
{'debug/num_tok_total': 2464.0, 'debug/num_tok_loss': 1597.0, 'debug/num_lat_total': 2464.0, 'debug/num_lat_loss': 1597.0, 'epoch': 3.82}

 38%|███▊      | 279/730 [24:44<39:58,  5.32s/it]
                                                 
{'train/ce_loss': 2.46875, 'train/diffusion_loss': 0.40019384026527405, 'epoch': 3.82}

 38%|███▊      | 279/730 [24:44<39:58,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.7715610549781457e-05, 'epoch': 3.82}

 38%|███▊      | 279/730 [24:44<39:58,  5.32s/it]
                                                 
{'debug/num_tok_total': 2434.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2434.0, 'debug/num_lat_loss': 1775.0, 'epoch': 3.82}

 38%|███▊      | 279/730 [24:45<39:58,  5.32s/it]
                                                 
{'train/ce_loss': 1.9921875, 'train/diffusion_loss': 0.4430074989795685, 'epoch': 3.82}

 38%|███▊      | 279/730 [24:45<39:58,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.7715610549781457e-05, 'epoch': 3.82}

 38%|███▊      | 279/730 [24:45<39:58,  5.32s/it]
                                                 
{'debug/num_tok_total': 3096.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 3096.0, 'debug/num_lat_loss': 1795.0, 'epoch': 3.82}

 38%|███▊      | 279/730 [24:47<39:58,  5.32s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.34121590852737427, 'epoch': 3.82}

 38%|███▊      | 279/730 [24:47<39:58,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.7715610549781457e-05, 'epoch': 3.82}

 38%|███▊      | 279/730 [24:47<39:58,  5.32s/it]
                                                 
{'debug/num_tok_total': 3084.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 3084.0, 'debug/num_lat_loss': 1794.0, 'epoch': 3.82}

 38%|███▊      | 279/730 [24:48<39:58,  5.32s/it]
                                                 
{'train/ce_loss': 2.046875, 'train/diffusion_loss': 0.35795098543167114, 'epoch': 3.82}

 38%|███▊      | 279/730 [24:48<39:58,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.7715610549781457e-05, 'epoch': 3.82}

 38%|███▊      | 279/730 [24:48<39:58,  5.32s/it]03/16/2026 07:12:21 - INFO - __main__ - LoRA debug step 280: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 38%|███▊      | 280/730 [24:49<40:19,  5.38s/it]
                                                 
{'loss': 2.5047, 'grad_norm': 1.58546781539917, 'learning_rate': 1.7715610549781457e-05, 'epoch': 3.84}

 38%|███▊      | 280/730 [24:49<40:19,  5.38s/it]
                                                 
{'debug/num_tok_total': 2629.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2629.0, 'debug/num_lat_loss': 1777.0, 'epoch': 3.84}

 38%|███▊      | 280/730 [24:50<40:19,  5.38s/it]
                                                 
{'train/ce_loss': 2.1875, 'train/diffusion_loss': 0.40037307143211365, 'epoch': 3.84}

 38%|███▊      | 280/730 [24:50<40:19,  5.38s/it]
                                                 
{'train/learning_rate_real': 1.766515232210473e-05, 'epoch': 3.84}

 38%|███▊      | 280/730 [24:50<40:19,  5.38s/it]
                                                 
{'debug/num_tok_total': 2863.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2863.0, 'debug/num_lat_loss': 1789.0, 'epoch': 3.84}

 38%|███▊      | 280/730 [24:51<40:19,  5.38s/it]
                                                 
{'train/ce_loss': 2.3125, 'train/diffusion_loss': 0.36349400877952576, 'epoch': 3.84}

 38%|███▊      | 280/730 [24:51<40:19,  5.38s/it]
                                                 
{'train/learning_rate_real': 1.766515232210473e-05, 'epoch': 3.84}

 38%|███▊      | 280/730 [24:51<40:19,  5.38s/it]
                                                 
{'debug/num_tok_total': 2648.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2648.0, 'debug/num_lat_loss': 1780.0, 'epoch': 3.84}

 38%|███▊      | 280/730 [24:52<40:19,  5.38s/it]
                                                 
{'train/ce_loss': 2.046875, 'train/diffusion_loss': 0.3854428827762604, 'epoch': 3.84}

 38%|███▊      | 280/730 [24:52<40:19,  5.38s/it]
                                                 
{'train/learning_rate_real': 1.766515232210473e-05, 'epoch': 3.84}

 38%|███▊      | 280/730 [24:52<40:19,  5.38s/it]
                                                 
{'debug/num_tok_total': 2873.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2873.0, 'debug/num_lat_loss': 1796.0, 'epoch': 3.84}

 38%|███▊      | 280/730 [24:53<40:19,  5.38s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.34019672870635986, 'epoch': 3.84}

 38%|███▊      | 280/730 [24:53<40:19,  5.38s/it]
                                                 
{'train/learning_rate_real': 1.766515232210473e-05, 'epoch': 3.84}

 38%|███▊      | 280/730 [24:53<40:19,  5.38s/it]
 38%|███▊      | 281/730 [24:54<40:20,  5.39s/it]
                                                 
{'debug/num_tok_total': 2658.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2658.0, 'debug/num_lat_loss': 1793.0, 'epoch': 3.85}

 38%|███▊      | 281/730 [24:55<40:20,  5.39s/it]
                                                 
{'train/ce_loss': 2.40625, 'train/diffusion_loss': 0.4039023518562317, 'epoch': 3.85}

 38%|███▊      | 281/730 [24:55<40:20,  5.39s/it]
                                                 
{'train/learning_rate_real': 1.7614592395669568e-05, 'epoch': 3.85}

 38%|███▊      | 281/730 [24:55<40:20,  5.39s/it]
                                                 
{'debug/num_tok_total': 2388.0, 'debug/num_tok_loss': 1734.0, 'debug/num_lat_total': 2388.0, 'debug/num_lat_loss': 1734.0, 'epoch': 3.85}

 38%|███▊      | 281/730 [24:56<40:20,  5.39s/it]
                                                 
{'train/ce_loss': 2.078125, 'train/diffusion_loss': 0.4264967739582062, 'epoch': 3.85}

 38%|███▊      | 281/730 [24:56<40:20,  5.39s/it]
                                                 
{'train/learning_rate_real': 1.7614592395669568e-05, 'epoch': 3.85}

 38%|███▊      | 281/730 [24:56<40:20,  5.39s/it]
                                                 
{'debug/num_tok_total': 2652.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2652.0, 'debug/num_lat_loss': 1782.0, 'epoch': 3.85}

 38%|███▊      | 281/730 [24:57<40:20,  5.39s/it]
                                                 
{'train/ce_loss': 2.1875, 'train/diffusion_loss': 0.39664873480796814, 'epoch': 3.85}

 38%|███▊      | 281/730 [24:57<40:20,  5.39s/it]
                                                 
{'train/learning_rate_real': 1.7614592395669568e-05, 'epoch': 3.85}

 38%|███▊      | 281/730 [24:57<40:20,  5.39s/it]
                                                 
{'debug/num_tok_total': 2494.0, 'debug/num_tok_loss': 1625.0, 'debug/num_lat_total': 2494.0, 'debug/num_lat_loss': 1625.0, 'epoch': 3.85}

 38%|███▊      | 281/730 [24:59<40:20,  5.39s/it]
                                                 
{'train/ce_loss': 2.15625, 'train/diffusion_loss': 0.4028310775756836, 'epoch': 3.85}

 38%|███▊      | 281/730 [24:59<40:20,  5.39s/it]
                                                 
{'train/learning_rate_real': 1.7614592395669568e-05, 'epoch': 3.85}

 38%|███▊      | 281/730 [24:59<40:20,  5.39s/it]
 39%|███▊      | 282/730 [24:59<39:43,  5.32s/it]
                                                 
{'debug/num_tok_total': 3115.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 3115.0, 'debug/num_lat_loss': 1802.0, 'epoch': 3.86}

 39%|███▊      | 282/730 [25:00<39:43,  5.32s/it]
                                                 
{'train/ce_loss': 2.203125, 'train/diffusion_loss': 0.3257860243320465, 'epoch': 3.86}

 39%|███▊      | 282/730 [25:00<39:43,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.756393176597067e-05, 'epoch': 3.86}

 39%|███▊      | 282/730 [25:00<39:43,  5.32s/it]
                                                 
{'debug/num_tok_total': 2842.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2842.0, 'debug/num_lat_loss': 1772.0, 'epoch': 3.86}

 39%|███▊      | 282/730 [25:02<39:43,  5.32s/it]
                                                 
{'train/ce_loss': 2.125, 'train/diffusion_loss': 0.33505338430404663, 'epoch': 3.86}

 39%|███▊      | 282/730 [25:02<39:43,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.756393176597067e-05, 'epoch': 3.86}

 39%|███▊      | 282/730 [25:02<39:43,  5.32s/it]
                                                 
{'debug/num_tok_total': 2608.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2608.0, 'debug/num_lat_loss': 1774.0, 'epoch': 3.86}

 39%|███▊      | 282/730 [25:03<39:43,  5.32s/it]
                                                 
{'train/ce_loss': 2.078125, 'train/diffusion_loss': 0.39998874068260193, 'epoch': 3.86}

 39%|███▊      | 282/730 [25:03<39:43,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.756393176597067e-05, 'epoch': 3.86}

 39%|███▊      | 282/730 [25:03<39:43,  5.32s/it]
                                                 
{'debug/num_tok_total': 2870.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2870.0, 'debug/num_lat_loss': 1795.0, 'epoch': 3.86}

 39%|███▊      | 282/730 [25:04<39:43,  5.32s/it]
                                                 
{'train/ce_loss': 2.15625, 'train/diffusion_loss': 0.3786977231502533, 'epoch': 3.86}

 39%|███▊      | 282/730 [25:04<39:43,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.756393176597067e-05, 'epoch': 3.86}

 39%|███▊      | 282/730 [25:04<39:43,  5.32s/it]
 39%|███▉      | 283/730 [25:05<39:53,  5.35s/it]
                                                 
{'debug/num_tok_total': 2152.0, 'debug/num_tok_loss': 1739.0, 'debug/num_lat_total': 2152.0, 'debug/num_lat_loss': 1739.0, 'epoch': 3.88}

 39%|███▉      | 283/730 [25:06<39:53,  5.35s/it]
                                                 
{'train/ce_loss': 2.171875, 'train/diffusion_loss': 0.5135148763656616, 'epoch': 3.88}

 39%|███▉      | 283/730 [25:06<39:53,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.751317143048552e-05, 'epoch': 3.88}

 39%|███▉      | 283/730 [25:06<39:53,  5.35s/it]
                                                 
{'debug/num_tok_total': 2412.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2412.0, 'debug/num_lat_loss': 1786.0, 'epoch': 3.88}

 39%|███▉      | 283/730 [25:07<39:53,  5.35s/it]
                                                 
{'train/ce_loss': 2.3125, 'train/diffusion_loss': 0.43617475032806396, 'epoch': 3.88}

 39%|███▉      | 283/730 [25:07<39:53,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.751317143048552e-05, 'epoch': 3.88}

 39%|███▉      | 283/730 [25:07<39:53,  5.35s/it]
                                                 
{'debug/num_tok_total': 2612.0, 'debug/num_tok_loss': 1762.0, 'debug/num_lat_total': 2612.0, 'debug/num_lat_loss': 1762.0, 'epoch': 3.88}

 39%|███▉      | 283/730 [25:08<39:53,  5.35s/it]
                                                 
{'train/ce_loss': 2.140625, 'train/diffusion_loss': 0.3687136173248291, 'epoch': 3.88}

 39%|███▉      | 283/730 [25:08<39:53,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.751317143048552e-05, 'epoch': 3.88}

 39%|███▉      | 283/730 [25:08<39:53,  5.35s/it]
                                                 
{'debug/num_tok_total': 2563.0, 'debug/num_tok_loss': 1731.0, 'debug/num_lat_total': 2563.0, 'debug/num_lat_loss': 1731.0, 'epoch': 3.88}

 39%|███▉      | 283/730 [25:09<39:53,  5.35s/it]
                                                 
{'train/ce_loss': 2.1875, 'train/diffusion_loss': 0.4157203435897827, 'epoch': 3.88}

 39%|███▉      | 283/730 [25:09<39:53,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.751317143048552e-05, 'epoch': 3.88}

 39%|███▉      | 283/730 [25:09<39:53,  5.35s/it]
 39%|███▉      | 284/730 [25:10<38:59,  5.24s/it]
                                                 
{'debug/num_tok_total': 2787.0, 'debug/num_tok_loss': 1641.0, 'debug/num_lat_total': 2787.0, 'debug/num_lat_loss': 1641.0, 'epoch': 3.89}

 39%|███▉      | 284/730 [25:11<38:59,  5.24s/it]
                                                 
{'train/ce_loss': 2.453125, 'train/diffusion_loss': 0.35626301169395447, 'epoch': 3.89}

 39%|███▉      | 284/730 [25:11<38:59,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.7462312388654752e-05, 'epoch': 3.89}

 39%|███▉      | 284/730 [25:11<38:59,  5.24s/it]
                                                 
{'debug/num_tok_total': 1998.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 1998.0, 'debug/num_lat_loss': 1785.0, 'epoch': 3.89}

 39%|███▉      | 284/730 [25:12<38:59,  5.24s/it]
                                                 
{'train/ce_loss': 2.25, 'train/diffusion_loss': 0.5404520630836487, 'epoch': 3.89}

 39%|███▉      | 284/730 [25:12<38:59,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.7462312388654752e-05, 'epoch': 3.89}

 39%|███▉      | 284/730 [25:12<38:59,  5.24s/it]
                                                 
{'debug/num_tok_total': 2832.0, 'debug/num_tok_loss': 1759.0, 'debug/num_lat_total': 2832.0, 'debug/num_lat_loss': 1759.0, 'epoch': 3.89}

 39%|███▉      | 284/730 [25:13<38:59,  5.24s/it]
                                                 
{'train/ce_loss': 2.421875, 'train/diffusion_loss': 0.3632473945617676, 'epoch': 3.89}

 39%|███▉      | 284/730 [25:13<38:59,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.7462312388654752e-05, 'epoch': 3.89}

 39%|███▉      | 284/730 [25:13<38:59,  5.24s/it]
                                                 
{'debug/num_tok_total': 2420.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2420.0, 'debug/num_lat_loss': 1778.0, 'epoch': 3.89}

 39%|███▉      | 284/730 [25:14<38:59,  5.24s/it]
                                                 
{'train/ce_loss': 2.296875, 'train/diffusion_loss': 0.44463783502578735, 'epoch': 3.89}

 39%|███▉      | 284/730 [25:14<38:59,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.7462312388654752e-05, 'epoch': 3.89}

 39%|███▉      | 284/730 [25:14<38:59,  5.24s/it]
 39%|███▉      | 285/730 [25:15<38:34,  5.20s/it]
                                                 
{'debug/num_tok_total': 2039.0, 'debug/num_tok_loss': 1623.0, 'debug/num_lat_total': 2039.0, 'debug/num_lat_loss': 1623.0, 'epoch': 3.9}

 39%|███▉      | 285/730 [25:16<38:34,  5.20s/it]
                                                 
{'train/ce_loss': 2.078125, 'train/diffusion_loss': 0.4851874113082886, 'epoch': 3.9}

 39%|███▉      | 285/730 [25:16<38:34,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.7411355641862466e-05, 'epoch': 3.9}

 39%|███▉      | 285/730 [25:16<38:34,  5.20s/it]
                                                 
{'debug/num_tok_total': 2418.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2418.0, 'debug/num_lat_loss': 1776.0, 'epoch': 3.9}

 39%|███▉      | 285/730 [25:17<38:34,  5.20s/it]
                                                 
{'train/ce_loss': 2.109375, 'train/diffusion_loss': 0.4470239281654358, 'epoch': 3.9}

 39%|███▉      | 285/730 [25:17<38:34,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.7411355641862466e-05, 'epoch': 3.9}

 39%|███▉      | 285/730 [25:17<38:34,  5.20s/it]
                                                 
{'debug/num_tok_total': 2431.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2431.0, 'debug/num_lat_loss': 1787.0, 'epoch': 3.9}

 39%|███▉      | 285/730 [25:18<38:34,  5.20s/it]
                                                 
{'train/ce_loss': 2.140625, 'train/diffusion_loss': 0.4692850112915039, 'epoch': 3.9}

 39%|███▉      | 285/730 [25:18<38:34,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.7411355641862466e-05, 'epoch': 3.9}

 39%|███▉      | 285/730 [25:18<38:34,  5.20s/it]
                                                 
{'debug/num_tok_total': 2917.0, 'debug/num_tok_loss': 1823.0, 'debug/num_lat_total': 2917.0, 'debug/num_lat_loss': 1823.0, 'epoch': 3.9}

 39%|███▉      | 285/730 [25:19<38:34,  5.20s/it]
                                                 
{'train/ce_loss': 2.125, 'train/diffusion_loss': 0.34697669744491577, 'epoch': 3.9}

 39%|███▉      | 285/730 [25:19<38:34,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.7411355641862466e-05, 'epoch': 3.9}

 39%|███▉      | 285/730 [25:19<38:34,  5.20s/it]
 39%|███▉      | 286/730 [25:20<38:06,  5.15s/it]
                                                 
{'debug/num_tok_total': 2454.0, 'debug/num_tok_loss': 1807.0, 'debug/num_lat_total': 2454.0, 'debug/num_lat_loss': 1807.0, 'epoch': 3.92}

 39%|███▉      | 286/730 [25:21<38:06,  5.15s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.43287089467048645, 'epoch': 3.92}

 39%|███▉      | 286/730 [25:21<38:06,  5.15s/it]
                                                 
{'train/learning_rate_real': 1.736030219341651e-05, 'epoch': 3.92}

 39%|███▉      | 286/730 [25:21<38:06,  5.15s/it]
                                                 
{'debug/num_tok_total': 2659.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 2659.0, 'debug/num_lat_loss': 1797.0, 'epoch': 3.92}

 39%|███▉      | 286/730 [25:22<38:06,  5.15s/it]
                                                 
{'train/ce_loss': 2.1875, 'train/diffusion_loss': 0.3962542414665222, 'epoch': 3.92}

 39%|███▉      | 286/730 [25:22<38:06,  5.15s/it]
                                                 
{'train/learning_rate_real': 1.736030219341651e-05, 'epoch': 3.92}

 39%|███▉      | 286/730 [25:22<38:06,  5.15s/it]
                                                 
{'debug/num_tok_total': 2676.0, 'debug/num_tok_loss': 1686.0, 'debug/num_lat_total': 2676.0, 'debug/num_lat_loss': 1686.0, 'epoch': 3.92}

 39%|███▉      | 286/730 [25:23<38:06,  5.15s/it]
                                                 
{'train/ce_loss': 2.09375, 'train/diffusion_loss': 0.37192660570144653, 'epoch': 3.92}

 39%|███▉      | 286/730 [25:23<38:06,  5.15s/it]
                                                 
{'train/learning_rate_real': 1.736030219341651e-05, 'epoch': 3.92}

 39%|███▉      | 286/730 [25:23<38:06,  5.15s/it]
                                                 
{'debug/num_tok_total': 2491.0, 'debug/num_tok_loss': 1825.0, 'debug/num_lat_total': 2491.0, 'debug/num_lat_loss': 1825.0, 'epoch': 3.92}

 39%|███▉      | 286/730 [25:24<38:06,  5.15s/it]
                                                 
{'train/ce_loss': 2.109375, 'train/diffusion_loss': 0.43956512212753296, 'epoch': 3.92}

 39%|███▉      | 286/730 [25:24<38:06,  5.15s/it]
                                                 
{'train/learning_rate_real': 1.736030219341651e-05, 'epoch': 3.92}

 39%|███▉      | 286/730 [25:24<38:06,  5.15s/it]
 39%|███▉      | 287/730 [25:25<38:16,  5.18s/it]
                                                 
{'debug/num_tok_total': 2464.0, 'debug/num_tok_loss': 1805.0, 'debug/num_lat_total': 2464.0, 'debug/num_lat_loss': 1805.0, 'epoch': 3.93}

 39%|███▉      | 287/730 [25:26<38:16,  5.18s/it]
                                                 
{'train/ce_loss': 2.09375, 'train/diffusion_loss': 0.42794516682624817, 'epoch': 3.93}

 39%|███▉      | 287/730 [25:26<38:16,  5.18s/it]
                                                 
{'train/learning_rate_real': 1.7309153048528735e-05, 'epoch': 3.93}

 39%|███▉      | 287/730 [25:26<38:16,  5.18s/it]
                                                 
{'debug/num_tok_total': 2636.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2636.0, 'debug/num_lat_loss': 1780.0, 'epoch': 3.93}

 39%|███▉      | 287/730 [25:27<38:16,  5.18s/it]
                                                 
{'train/ce_loss': 2.015625, 'train/diffusion_loss': 0.3934583365917206, 'epoch': 3.93}

 39%|███▉      | 287/730 [25:27<38:16,  5.18s/it]
                                                 
{'train/learning_rate_real': 1.7309153048528735e-05, 'epoch': 3.93}

 39%|███▉      | 287/730 [25:27<38:16,  5.18s/it]
                                                 
{'debug/num_tok_total': 3072.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 3072.0, 'debug/num_lat_loss': 1779.0, 'epoch': 3.93}

 39%|███▉      | 287/730 [25:29<38:16,  5.18s/it]
                                                 
{'train/ce_loss': 1.984375, 'train/diffusion_loss': 0.2945156395435333, 'epoch': 3.93}

 39%|███▉      | 287/730 [25:29<38:16,  5.18s/it]
                                                 
{'train/learning_rate_real': 1.7309153048528735e-05, 'epoch': 3.93}

 39%|███▉      | 287/730 [25:29<38:16,  5.18s/it]
                                                 
{'debug/num_tok_total': 2416.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2416.0, 'debug/num_lat_loss': 1785.0, 'epoch': 3.93}

 39%|███▉      | 287/730 [25:30<38:16,  5.18s/it]
                                                 
{'train/ce_loss': 2.15625, 'train/diffusion_loss': 0.4588899314403534, 'epoch': 3.93}

 39%|███▉      | 287/730 [25:30<38:16,  5.18s/it]
                                                 
{'train/learning_rate_real': 1.7309153048528735e-05, 'epoch': 3.93}

 39%|███▉      | 287/730 [25:30<38:16,  5.18s/it]
 39%|███▉      | 288/730 [25:30<38:14,  5.19s/it]
                                                 
{'debug/num_tok_total': 2689.0, 'debug/num_tok_loss': 1812.0, 'debug/num_lat_total': 2689.0, 'debug/num_lat_loss': 1812.0, 'epoch': 3.95}

 39%|███▉      | 288/730 [25:31<38:14,  5.19s/it]
                                                 
{'train/ce_loss': 2.140625, 'train/diffusion_loss': 0.4178195893764496, 'epoch': 3.95}

 39%|███▉      | 288/730 [25:31<38:14,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.72579092142952e-05, 'epoch': 3.95}

 39%|███▉      | 288/730 [25:31<38:14,  5.19s/it]
                                                 
{'debug/num_tok_total': 3043.0, 'debug/num_tok_loss': 1753.0, 'debug/num_lat_total': 3043.0, 'debug/num_lat_loss': 1753.0, 'epoch': 3.95}

 39%|███▉      | 288/730 [25:33<38:14,  5.19s/it]
                                                 
{'train/ce_loss': 2.046875, 'train/diffusion_loss': 0.34568458795547485, 'epoch': 3.95}

 39%|███▉      | 288/730 [25:33<38:14,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.72579092142952e-05, 'epoch': 3.95}

 39%|███▉      | 288/730 [25:33<38:14,  5.19s/it]
                                                 
{'debug/num_tok_total': 2622.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2622.0, 'debug/num_lat_loss': 1778.0, 'epoch': 3.95}

 39%|███▉      | 288/730 [25:34<38:14,  5.19s/it]
                                                 
{'train/ce_loss': 2.109375, 'train/diffusion_loss': 0.3778199851512909, 'epoch': 3.95}

 39%|███▉      | 288/730 [25:34<38:14,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.72579092142952e-05, 'epoch': 3.95}

 39%|███▉      | 288/730 [25:34<38:14,  5.19s/it]
                                                 
{'debug/num_tok_total': 1749.0, 'debug/num_tok_loss': 1749.0, 'debug/num_lat_total': 1749.0, 'debug/num_lat_loss': 1749.0, 'epoch': 3.95}

 39%|███▉      | 288/730 [25:35<38:14,  5.19s/it]
                                                 
{'train/ce_loss': 2.390625, 'train/diffusion_loss': 0.5831629037857056, 'epoch': 3.95}

 39%|███▉      | 288/730 [25:35<38:14,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.72579092142952e-05, 'epoch': 3.95}

 39%|███▉      | 288/730 [25:35<38:14,  5.19s/it]
 40%|███▉      | 289/730 [25:35<37:50,  5.15s/it]
                                                 
{'debug/num_tok_total': 2861.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2861.0, 'debug/num_lat_loss': 1789.0, 'epoch': 3.96}

 40%|███▉      | 289/730 [25:36<37:50,  5.15s/it]
                                                 
{'train/ce_loss': 2.109375, 'train/diffusion_loss': 0.3550923764705658, 'epoch': 3.96}

 40%|███▉      | 289/730 [25:36<37:50,  5.15s/it]
                                                 
{'train/learning_rate_real': 1.720657169967633e-05, 'epoch': 3.96}

 40%|███▉      | 289/730 [25:36<37:50,  5.15s/it]
                                                 
{'debug/num_tok_total': 2450.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2450.0, 'debug/num_lat_loss': 1792.0, 'epoch': 3.96}

 40%|███▉      | 289/730 [25:38<37:50,  5.15s/it]
                                                 
{'train/ce_loss': 2.296875, 'train/diffusion_loss': 0.43373000621795654, 'epoch': 3.96}

 40%|███▉      | 289/730 [25:38<37:50,  5.15s/it]
                                                 
{'train/learning_rate_real': 1.720657169967633e-05, 'epoch': 3.96}

 40%|███▉      | 289/730 [25:38<37:50,  5.15s/it]
                                                 
{'debug/num_tok_total': 2404.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2404.0, 'debug/num_lat_loss': 1764.0, 'epoch': 3.96}

 40%|███▉      | 289/730 [25:39<37:50,  5.15s/it]
                                                 
{'train/ce_loss': 2.015625, 'train/diffusion_loss': 0.44662579894065857, 'epoch': 3.96}

 40%|███▉      | 289/730 [25:39<37:50,  5.15s/it]
                                                 
{'train/learning_rate_real': 1.720657169967633e-05, 'epoch': 3.96}

 40%|███▉      | 289/730 [25:39<37:50,  5.15s/it]
                                                 
{'debug/num_tok_total': 2840.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2840.0, 'debug/num_lat_loss': 1774.0, 'epoch': 3.96}

 40%|███▉      | 289/730 [25:40<37:50,  5.15s/it]
                                                 
{'train/ce_loss': 2.046875, 'train/diffusion_loss': 0.33653688430786133, 'epoch': 3.96}

 40%|███▉      | 289/730 [25:40<37:50,  5.15s/it]
                                                 
{'train/learning_rate_real': 1.720657169967633e-05, 'epoch': 3.96}

 40%|███▉      | 289/730 [25:40<37:50,  5.15s/it]03/16/2026 07:13:12 - INFO - __main__ - LoRA debug step 290: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 40%|███▉      | 290/730 [25:41<38:08,  5.20s/it]
                                                 
{'loss': 2.6218, 'grad_norm': 1.321590781211853, 'learning_rate': 1.720657169967633e-05, 'epoch': 3.97}

 40%|███▉      | 290/730 [25:41<38:08,  5.20s/it]
                                                 
{'debug/num_tok_total': 3055.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 3055.0, 'debug/num_lat_loss': 1765.0, 'epoch': 3.97}

 40%|███▉      | 290/730 [25:42<38:08,  5.20s/it]
                                                 
{'train/ce_loss': 2.109375, 'train/diffusion_loss': 0.3077630400657654, 'epoch': 3.97}

 40%|███▉      | 290/730 [25:42<38:08,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.7155141515477075e-05, 'epoch': 3.97}

 40%|███▉      | 290/730 [25:42<38:08,  5.20s/it]
                                                 
{'debug/num_tok_total': 3049.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 3049.0, 'debug/num_lat_loss': 1770.0, 'epoch': 3.97}

 40%|███▉      | 290/730 [25:43<38:08,  5.20s/it]
                                                 
{'train/ce_loss': 1.9453125, 'train/diffusion_loss': 0.32115334272384644, 'epoch': 3.97}

 40%|███▉      | 290/730 [25:43<38:08,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.7155141515477075e-05, 'epoch': 3.97}

 40%|███▉      | 290/730 [25:43<38:08,  5.20s/it]
                                                 
{'debug/num_tok_total': 2655.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2655.0, 'debug/num_lat_loss': 1793.0, 'epoch': 3.97}

 40%|███▉      | 290/730 [25:44<38:08,  5.20s/it]
                                                 
{'train/ce_loss': 2.3125, 'train/diffusion_loss': 0.3869441747665405, 'epoch': 3.97}

 40%|███▉      | 290/730 [25:44<38:08,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.7155141515477075e-05, 'epoch': 3.97}

 40%|███▉      | 290/730 [25:44<38:08,  5.20s/it]
                                                 
{'debug/num_tok_total': 2605.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 2605.0, 'debug/num_lat_loss': 1765.0, 'epoch': 3.97}

 40%|███▉      | 290/730 [25:46<38:08,  5.20s/it]
                                                 
{'train/ce_loss': 2.125, 'train/diffusion_loss': 0.4380507469177246, 'epoch': 3.97}

 40%|███▉      | 290/730 [25:46<38:08,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.7155141515477075e-05, 'epoch': 3.97}

 40%|███▉      | 290/730 [25:46<38:08,  5.20s/it]
 40%|███▉      | 291/730 [25:46<38:54,  5.32s/it]
                                                 
{'debug/num_tok_total': 2399.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 2399.0, 'debug/num_lat_loss': 1767.0, 'epoch': 3.99}

 40%|███▉      | 291/730 [25:47<38:54,  5.32s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.4650937616825104, 'epoch': 3.99}

 40%|███▉      | 291/730 [25:47<38:54,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.7103619674326984e-05, 'epoch': 3.99}

 40%|███▉      | 291/730 [25:47<38:54,  5.32s/it]
                                                 
{'debug/num_tok_total': 3365.0, 'debug/num_tok_loss': 1822.0, 'debug/num_lat_total': 3365.0, 'debug/num_lat_loss': 1822.0, 'epoch': 3.99}

 40%|███▉      | 291/730 [25:48<38:54,  5.32s/it]
                                                 
{'train/ce_loss': 2.078125, 'train/diffusion_loss': 0.2749086022377014, 'epoch': 3.99}

 40%|███▉      | 291/730 [25:48<38:54,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.7103619674326984e-05, 'epoch': 3.99}

 40%|███▉      | 291/730 [25:48<38:54,  5.32s/it]
                                                 
{'debug/num_tok_total': 2476.0, 'debug/num_tok_loss': 1824.0, 'debug/num_lat_total': 2476.0, 'debug/num_lat_loss': 1824.0, 'epoch': 3.99}

 40%|███▉      | 291/730 [25:50<38:54,  5.32s/it]
                                                 
{'train/ce_loss': 2.09375, 'train/diffusion_loss': 0.4461573362350464, 'epoch': 3.99}

 40%|███▉      | 291/730 [25:50<38:54,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.7103619674326984e-05, 'epoch': 3.99}

 40%|███▉      | 291/730 [25:50<38:54,  5.32s/it]
                                                 
{'debug/num_tok_total': 438.0, 'debug/num_tok_loss': 438.0, 'debug/num_lat_total': 438.0, 'debug/num_lat_loss': 438.0, 'epoch': 3.99}

 40%|███▉      | 291/730 [25:50<38:54,  5.32s/it]
                                                 
{'train/ce_loss': 1.9375, 'train/diffusion_loss': 0.6075020432472229, 'epoch': 3.99}

 40%|███▉      | 291/730 [25:50<38:54,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.7103619674326984e-05, 'epoch': 3.99}

 40%|███▉      | 291/730 [25:50<38:54,  5.32s/it]
 40%|████      | 292/730 [25:50<36:25,  4.99s/it]
                                                 
{'debug/num_tok_total': 2397.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2397.0, 'debug/num_lat_loss': 1764.0, 'epoch': 4.0}

 40%|████      | 292/730 [25:52<36:25,  4.99s/it]
                                                 
{'train/ce_loss': 1.9609375, 'train/diffusion_loss': 0.43957024812698364, 'epoch': 4.0}

 40%|████      | 292/730 [25:52<36:25,  4.99s/it]
                                                 
{'train/learning_rate_real': 1.705200719066028e-05, 'epoch': 4.0}

 40%|████      | 292/730 [25:52<36:25,  4.99s/it]
                                                 
{'debug/num_tok_total': 2442.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2442.0, 'debug/num_lat_loss': 1799.0, 'epoch': 4.0}

 40%|████      | 292/730 [25:53<36:25,  4.99s/it]
                                                 
{'train/ce_loss': 2.125, 'train/diffusion_loss': 0.43320709466934204, 'epoch': 4.0}

 40%|████      | 292/730 [25:53<36:25,  4.99s/it]
                                                 
{'train/learning_rate_real': 1.705200719066028e-05, 'epoch': 4.0}

 40%|████      | 292/730 [25:53<36:25,  4.99s/it]
                                                 
{'debug/num_tok_total': 2588.0, 'debug/num_tok_loss': 1608.0, 'debug/num_lat_total': 2588.0, 'debug/num_lat_loss': 1608.0, 'epoch': 4.0}

 40%|████      | 292/730 [25:54<36:25,  4.99s/it]
                                                 
{'train/ce_loss': 2.203125, 'train/diffusion_loss': 0.36547115445137024, 'epoch': 4.0}

 40%|████      | 292/730 [25:54<36:25,  4.99s/it]
                                                 
{'train/learning_rate_real': 1.705200719066028e-05, 'epoch': 4.0}

 40%|████      | 292/730 [25:54<36:25,  4.99s/it]
                                                 
{'debug/num_tok_total': 2672.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 2672.0, 'debug/num_lat_loss': 1801.0, 'epoch': 4.0}

 40%|████      | 292/730 [25:55<36:25,  4.99s/it]
                                                 
{'train/ce_loss': 2.03125, 'train/diffusion_loss': 0.3984234035015106, 'epoch': 4.0}

 40%|████      | 292/730 [25:55<36:25,  4.99s/it]
                                                 
{'train/learning_rate_real': 1.705200719066028e-05, 'epoch': 4.0}

 40%|████      | 292/730 [25:55<36:25,  4.99s/it]
 40%|████      | 293/730 [25:56<37:11,  5.11s/it]
                                                 
{'debug/num_tok_total': 2219.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2219.0, 'debug/num_lat_loss': 1788.0, 'epoch': 4.01}

 40%|████      | 293/730 [25:57<37:11,  5.11s/it]
                                                 
{'train/ce_loss': 2.1875, 'train/diffusion_loss': 0.47886258363723755, 'epoch': 4.01}

 40%|████      | 293/730 [25:57<37:11,  5.11s/it]
                                                 
{'train/learning_rate_real': 1.7000305080695878e-05, 'epoch': 4.01}

 40%|████      | 293/730 [25:57<37:11,  5.11s/it]
                                                 
{'debug/num_tok_total': 2270.0, 'debug/num_tok_loss': 1595.0, 'debug/num_lat_total': 2270.0, 'debug/num_lat_loss': 1595.0, 'epoch': 4.01}

 40%|████      | 293/730 [25:58<37:11,  5.11s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.3977213501930237, 'epoch': 4.01}

 40%|████      | 293/730 [25:58<37:11,  5.11s/it]
                                                 
{'train/learning_rate_real': 1.7000305080695878e-05, 'epoch': 4.01}

 40%|████      | 293/730 [25:58<37:11,  5.11s/it]
                                                 
{'debug/num_tok_total': 2594.0, 'debug/num_tok_loss': 1747.0, 'debug/num_lat_total': 2594.0, 'debug/num_lat_loss': 1747.0, 'epoch': 4.01}

 40%|████      | 293/730 [25:59<37:11,  5.11s/it]
                                                 
{'train/ce_loss': 2.015625, 'train/diffusion_loss': 0.3679512143135071, 'epoch': 4.01}

 40%|████      | 293/730 [25:59<37:11,  5.11s/it]
                                                 
{'train/learning_rate_real': 1.7000305080695878e-05, 'epoch': 4.01}

 40%|████      | 293/730 [25:59<37:11,  5.11s/it]
                                                 
{'debug/num_tok_total': 2877.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2877.0, 'debug/num_lat_loss': 1793.0, 'epoch': 4.01}

 40%|████      | 293/730 [26:00<37:11,  5.11s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.35806283354759216, 'epoch': 4.01}

 40%|████      | 293/730 [26:00<37:11,  5.11s/it]
                                                 
{'train/learning_rate_real': 1.7000305080695878e-05, 'epoch': 4.01}

 40%|████      | 293/730 [26:00<37:11,  5.11s/it]
 40%|████      | 294/730 [26:01<37:21,  5.14s/it]
                                                 
{'debug/num_tok_total': 2644.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2644.0, 'debug/num_lat_loss': 1779.0, 'epoch': 4.03}

 40%|████      | 294/730 [26:02<37:21,  5.14s/it]
                                                 
{'train/ce_loss': 2.046875, 'train/diffusion_loss': 0.3923610746860504, 'epoch': 4.03}

 40%|████      | 294/730 [26:02<37:21,  5.14s/it]
                                                 
{'train/learning_rate_real': 1.6948514362417385e-05, 'epoch': 4.03}

 40%|████      | 294/730 [26:02<37:21,  5.14s/it]
                                                 
{'debug/num_tok_total': 2811.0, 'debug/num_tok_loss': 1752.0, 'debug/num_lat_total': 2811.0, 'debug/num_lat_loss': 1752.0, 'epoch': 4.03}

 40%|████      | 294/730 [26:03<37:21,  5.14s/it]
                                                 
{'train/ce_loss': 1.9609375, 'train/diffusion_loss': 0.3480333983898163, 'epoch': 4.03}

 40%|████      | 294/730 [26:03<37:21,  5.14s/it]
                                                 
{'train/learning_rate_real': 1.6948514362417385e-05, 'epoch': 4.03}

 40%|████      | 294/730 [26:03<37:21,  5.14s/it]
                                                 
{'debug/num_tok_total': 2007.0, 'debug/num_tok_loss': 1577.0, 'debug/num_lat_total': 2007.0, 'debug/num_lat_loss': 1577.0, 'epoch': 4.03}

 40%|████      | 294/730 [26:04<37:21,  5.14s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.4790566563606262, 'epoch': 4.03}

 40%|████      | 294/730 [26:04<37:21,  5.14s/it]
                                                 
{'train/learning_rate_real': 1.6948514362417385e-05, 'epoch': 4.03}

 40%|████      | 294/730 [26:04<37:21,  5.14s/it]
                                                 
{'debug/num_tok_total': 2240.0, 'debug/num_tok_loss': 1594.0, 'debug/num_lat_total': 2240.0, 'debug/num_lat_loss': 1594.0, 'epoch': 4.03}

 40%|████      | 294/730 [26:06<37:21,  5.14s/it]
                                                 
{'train/ce_loss': 2.5, 'train/diffusion_loss': 0.42749252915382385, 'epoch': 4.03}

 40%|████      | 294/730 [26:06<37:21,  5.14s/it]
                                                 
{'train/learning_rate_real': 1.6948514362417385e-05, 'epoch': 4.03}

 40%|████      | 294/730 [26:06<37:21,  5.14s/it]
 40%|████      | 295/730 [26:06<37:06,  5.12s/it]
                                                 
{'debug/num_tok_total': 2838.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2838.0, 'debug/num_lat_loss': 1777.0, 'epoch': 4.04}

 40%|████      | 295/730 [26:07<37:06,  5.12s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.35384467244148254, 'epoch': 4.04}

 40%|████      | 295/730 [26:07<37:06,  5.12s/it]
                                                 
{'train/learning_rate_real': 1.689663605555306e-05, 'epoch': 4.04}

 40%|████      | 295/730 [26:07<37:06,  5.12s/it]
                                                 
{'debug/num_tok_total': 2467.0, 'debug/num_tok_loss': 1806.0, 'debug/num_lat_total': 2467.0, 'debug/num_lat_loss': 1806.0, 'epoch': 4.04}

 40%|████      | 295/730 [26:08<37:06,  5.12s/it]
                                                 
{'train/ce_loss': 2.046875, 'train/diffusion_loss': 0.4224318265914917, 'epoch': 4.04}

 40%|████      | 295/730 [26:08<37:06,  5.12s/it]
                                                 
{'train/learning_rate_real': 1.689663605555306e-05, 'epoch': 4.04}

 40%|████      | 295/730 [26:08<37:06,  5.12s/it]
                                                 
{'debug/num_tok_total': 2208.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2208.0, 'debug/num_lat_loss': 1774.0, 'epoch': 4.04}

 40%|████      | 295/730 [26:09<37:06,  5.12s/it]
                                                 
{'train/ce_loss': 2.078125, 'train/diffusion_loss': 0.47826892137527466, 'epoch': 4.04}

 40%|████      | 295/730 [26:10<37:06,  5.12s/it]
                                                 
{'train/learning_rate_real': 1.689663605555306e-05, 'epoch': 4.04}

 40%|████      | 295/730 [26:10<37:06,  5.12s/it]
                                                 
{'debug/num_tok_total': 3044.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 3044.0, 'debug/num_lat_loss': 1764.0, 'epoch': 4.04}

 40%|████      | 295/730 [26:11<37:06,  5.12s/it]
                                                 
{'train/ce_loss': 2.09375, 'train/diffusion_loss': 0.3062922954559326, 'epoch': 4.04}

 40%|████      | 295/730 [26:11<37:06,  5.12s/it]
                                                 
{'train/learning_rate_real': 1.689663605555306e-05, 'epoch': 4.04}

 40%|████      | 295/730 [26:11<37:06,  5.12s/it]
 41%|████      | 296/730 [26:11<37:20,  5.16s/it]
                                                 
{'debug/num_tok_total': 2660.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2660.0, 'debug/num_lat_loss': 1781.0, 'epoch': 4.05}

 41%|████      | 296/730 [26:12<37:20,  5.16s/it]
                                                 
{'train/ce_loss': 1.9921875, 'train/diffusion_loss': 0.393525630235672, 'epoch': 4.05}

 41%|████      | 296/730 [26:12<37:20,  5.16s/it]
                                                 
{'train/learning_rate_real': 1.6844671181555717e-05, 'epoch': 4.05}

 41%|████      | 296/730 [26:12<37:20,  5.16s/it]
                                                 
{'debug/num_tok_total': 2509.0, 'debug/num_tok_loss': 1724.0, 'debug/num_lat_total': 2509.0, 'debug/num_lat_loss': 1724.0, 'epoch': 4.05}

 41%|████      | 296/730 [26:14<37:20,  5.16s/it]
                                                 
{'train/ce_loss': 2.21875, 'train/diffusion_loss': 0.44487008452415466, 'epoch': 4.05}

 41%|████      | 296/730 [26:14<37:20,  5.16s/it]
                                                 
{'train/learning_rate_real': 1.6844671181555717e-05, 'epoch': 4.05}

 41%|████      | 296/730 [26:14<37:20,  5.16s/it]
                                                 
{'debug/num_tok_total': 2675.0, 'debug/num_tok_loss': 1798.0, 'debug/num_lat_total': 2675.0, 'debug/num_lat_loss': 1798.0, 'epoch': 4.05}

 41%|████      | 296/730 [26:15<37:20,  5.16s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.428099662065506, 'epoch': 4.05}

 41%|████      | 296/730 [26:15<37:20,  5.16s/it]
                                                 
{'train/learning_rate_real': 1.6844671181555717e-05, 'epoch': 4.05}

 41%|████      | 296/730 [26:15<37:20,  5.16s/it]
                                                 
{'debug/num_tok_total': 1990.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 1990.0, 'debug/num_lat_loss': 1784.0, 'epoch': 4.05}

 41%|████      | 296/730 [26:16<37:20,  5.16s/it]
                                                 
{'train/ce_loss': 2.15625, 'train/diffusion_loss': 0.5531203150749207, 'epoch': 4.05}

 41%|████      | 296/730 [26:16<37:20,  5.16s/it]
                                                 
{'train/learning_rate_real': 1.6844671181555717e-05, 'epoch': 4.05}

 41%|████      | 296/730 [26:16<37:20,  5.16s/it]
 41%|████      | 297/730 [26:16<37:04,  5.14s/it]
                                                 
{'debug/num_tok_total': 3108.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 3108.0, 'debug/num_lat_loss': 1799.0, 'epoch': 4.07}

 41%|████      | 297/730 [26:18<37:04,  5.14s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.3036717176437378, 'epoch': 4.07}

 41%|████      | 297/730 [26:18<37:04,  5.14s/it]
                                                 
{'train/learning_rate_real': 1.679262076358263e-05, 'epoch': 4.07}

 41%|████      | 297/730 [26:18<37:04,  5.14s/it]
                                                 
{'debug/num_tok_total': 2454.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2454.0, 'debug/num_lat_loss': 1794.0, 'epoch': 4.07}

 41%|████      | 297/730 [26:19<37:04,  5.14s/it]
                                                 
{'train/ce_loss': 2.015625, 'train/diffusion_loss': 0.45707595348358154, 'epoch': 4.07}

 41%|████      | 297/730 [26:19<37:04,  5.14s/it]
                                                 
{'train/learning_rate_real': 1.679262076358263e-05, 'epoch': 4.07}

 41%|████      | 297/730 [26:19<37:04,  5.14s/it]
                                                 
{'debug/num_tok_total': 2618.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2618.0, 'debug/num_lat_loss': 1771.0, 'epoch': 4.07}

 41%|████      | 297/730 [26:20<37:04,  5.14s/it]
                                                 
{'train/ce_loss': 1.953125, 'train/diffusion_loss': 0.3870525360107422, 'epoch': 4.07}

 41%|████      | 297/730 [26:20<37:04,  5.14s/it]
                                                 
{'train/learning_rate_real': 1.679262076358263e-05, 'epoch': 4.07}

 41%|████      | 297/730 [26:20<37:04,  5.14s/it]
                                                 
{'debug/num_tok_total': 2852.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2852.0, 'debug/num_lat_loss': 1775.0, 'epoch': 4.07}

 41%|████      | 297/730 [26:21<37:04,  5.14s/it]
                                                 
{'train/ce_loss': 2.203125, 'train/diffusion_loss': 0.34175747632980347, 'epoch': 4.07}

 41%|████      | 297/730 [26:21<37:04,  5.14s/it]
                                                 
{'train/learning_rate_real': 1.679262076358263e-05, 'epoch': 4.07}

 41%|████      | 297/730 [26:21<37:04,  5.14s/it]
 41%|████      | 298/730 [26:22<37:21,  5.19s/it]
                                                 
{'debug/num_tok_total': 2696.0, 'debug/num_tok_loss': 1809.0, 'debug/num_lat_total': 2696.0, 'debug/num_lat_loss': 1809.0, 'epoch': 4.08}

 41%|████      | 298/730 [26:23<37:21,  5.19s/it]
                                                 
{'train/ce_loss': 1.9765625, 'train/diffusion_loss': 0.41354861855506897, 'epoch': 4.08}

 41%|████      | 298/730 [26:23<37:21,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.674048582647538e-05, 'epoch': 4.08}

 41%|████      | 298/730 [26:23<37:21,  5.19s/it]
                                                 
{'debug/num_tok_total': 2156.0, 'debug/num_tok_loss': 1715.0, 'debug/num_lat_total': 2156.0, 'debug/num_lat_loss': 1715.0, 'epoch': 4.08}

 41%|████      | 298/730 [26:24<37:21,  5.19s/it]
                                                 
{'train/ce_loss': 2.03125, 'train/diffusion_loss': 0.4522693455219269, 'epoch': 4.08}

 41%|████      | 298/730 [26:24<37:21,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.674048582647538e-05, 'epoch': 4.08}

 41%|████      | 298/730 [26:24<37:21,  5.19s/it]
                                                 
{'debug/num_tok_total': 2651.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2651.0, 'debug/num_lat_loss': 1783.0, 'epoch': 4.08}

 41%|████      | 298/730 [26:25<37:21,  5.19s/it]
                                                 
{'train/ce_loss': 1.9765625, 'train/diffusion_loss': 0.40085849165916443, 'epoch': 4.08}

 41%|████      | 298/730 [26:25<37:21,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.674048582647538e-05, 'epoch': 4.08}

 41%|████      | 298/730 [26:25<37:21,  5.19s/it]
                                                 
{'debug/num_tok_total': 2849.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2849.0, 'debug/num_lat_loss': 1782.0, 'epoch': 4.08}

 41%|████      | 298/730 [26:27<37:21,  5.19s/it]
                                                 
{'train/ce_loss': 1.921875, 'train/diffusion_loss': 0.36254966259002686, 'epoch': 4.08}

 41%|████      | 298/730 [26:27<37:21,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.674048582647538e-05, 'epoch': 4.08}

 41%|████      | 298/730 [26:27<37:21,  5.19s/it]
 41%|████      | 299/730 [26:27<37:30,  5.22s/it]
                                                 
{'debug/num_tok_total': 2433.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2433.0, 'debug/num_lat_loss': 1785.0, 'epoch': 4.1}

 41%|████      | 299/730 [26:28<37:30,  5.22s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.4671931564807892, 'epoch': 4.1}

 41%|████      | 299/730 [26:28<37:30,  5.22s/it]
                                                 
{'train/learning_rate_real': 1.6688267396739686e-05, 'epoch': 4.1}

 41%|████      | 299/730 [26:28<37:30,  5.22s/it]
                                                 
{'debug/num_tok_total': 2682.0, 'debug/num_tok_loss': 1803.0, 'debug/num_lat_total': 2682.0, 'debug/num_lat_loss': 1803.0, 'epoch': 4.1}

 41%|████      | 299/730 [26:29<37:30,  5.22s/it]
                                                 
{'train/ce_loss': 1.9375, 'train/diffusion_loss': 0.408421128988266, 'epoch': 4.1}

 41%|████      | 299/730 [26:29<37:30,  5.22s/it]
                                                 
{'train/learning_rate_real': 1.6688267396739686e-05, 'epoch': 4.1}

 41%|████      | 299/730 [26:29<37:30,  5.22s/it]
                                                 
{'debug/num_tok_total': 2388.0, 'debug/num_tok_loss': 1753.0, 'debug/num_lat_total': 2388.0, 'debug/num_lat_loss': 1753.0, 'epoch': 4.1}

 41%|████      | 299/730 [26:30<37:30,  5.22s/it]
                                                 
{'train/ce_loss': 2.046875, 'train/diffusion_loss': 0.43699023127555847, 'epoch': 4.1}

 41%|████      | 299/730 [26:30<37:30,  5.22s/it]
                                                 
{'train/learning_rate_real': 1.6688267396739686e-05, 'epoch': 4.1}

 41%|████      | 299/730 [26:30<37:30,  5.22s/it]
                                                 
{'debug/num_tok_total': 2423.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2423.0, 'debug/num_lat_loss': 1777.0, 'epoch': 4.1}

 41%|████      | 299/730 [26:32<37:30,  5.22s/it]
                                                 
{'train/ce_loss': 1.8984375, 'train/diffusion_loss': 0.4659460484981537, 'epoch': 4.1}

 41%|████      | 299/730 [26:32<37:30,  5.22s/it]
                                                 
{'train/learning_rate_real': 1.6688267396739686e-05, 'epoch': 4.1}

 41%|████      | 299/730 [26:32<37:30,  5.22s/it]03/16/2026 07:14:04 - INFO - __main__ - LoRA debug step 300: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 41%|████      | 300/730 [26:32<37:12,  5.19s/it]
                                                 
{'loss': 2.6268, 'grad_norm': 1.2555819749832153, 'learning_rate': 1.6688267396739686e-05, 'epoch': 4.11}

 41%|████      | 300/730 [26:32<37:12,  5.19s/it]
                                                 
{'debug/num_tok_total': 2662.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2662.0, 'debug/num_lat_loss': 1784.0, 'epoch': 4.11}

 41%|████      | 300/730 [26:33<37:12,  5.19s/it]
                                                 
{'train/ce_loss': 2.078125, 'train/diffusion_loss': 0.39393725991249084, 'epoch': 4.11}

 41%|████      | 300/730 [26:33<37:12,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.6635966502525174e-05, 'epoch': 4.11}

 41%|████      | 300/730 [26:33<37:12,  5.19s/it]
                                                 
{'debug/num_tok_total': 2202.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 2202.0, 'debug/num_lat_loss': 1768.0, 'epoch': 4.11}

 41%|████      | 300/730 [26:34<37:12,  5.19s/it]
                                                 
{'train/ce_loss': 2.25, 'train/diffusion_loss': 0.5026405453681946, 'epoch': 4.11}

 41%|████      | 300/730 [26:34<37:12,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.6635966502525174e-05, 'epoch': 4.11}

 41%|████      | 300/730 [26:34<37:12,  5.19s/it]
                                                 
{'debug/num_tok_total': 2420.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2420.0, 'debug/num_lat_loss': 1778.0, 'epoch': 4.11}

 41%|████      | 300/730 [26:35<37:12,  5.19s/it]
                                                 
{'train/ce_loss': 2.109375, 'train/diffusion_loss': 0.4620644152164459, 'epoch': 4.11}

 41%|████      | 300/730 [26:35<37:12,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.6635966502525174e-05, 'epoch': 4.11}

 41%|████      | 300/730 [26:35<37:12,  5.19s/it]
                                                 
{'debug/num_tok_total': 2869.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2869.0, 'debug/num_lat_loss': 1787.0, 'epoch': 4.11}

 41%|████      | 300/730 [26:37<37:12,  5.19s/it]
                                                 
{'train/ce_loss': 2.296875, 'train/diffusion_loss': 0.3546100854873657, 'epoch': 4.11}

 41%|████      | 300/730 [26:37<37:12,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.6635966502525174e-05, 'epoch': 4.11}

 41%|████      | 300/730 [26:37<37:12,  5.19s/it]
 41%|████      | 301/730 [26:37<36:53,  5.16s/it]
                                                 
{'debug/num_tok_total': 1691.0, 'debug/num_tok_loss': 1473.0, 'debug/num_lat_total': 1691.0, 'debug/num_lat_loss': 1473.0, 'epoch': 4.12}

 41%|████      | 301/730 [26:38<36:53,  5.16s/it]
                                                 
{'train/ce_loss': 1.890625, 'train/diffusion_loss': 0.5275248289108276, 'epoch': 4.12}

 41%|████      | 301/730 [26:38<36:53,  5.16s/it]
                                                 
{'train/learning_rate_real': 1.6583584173605164e-05, 'epoch': 4.12}

 41%|████      | 301/730 [26:38<36:53,  5.16s/it]
                                                 
{'debug/num_tok_total': 2630.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2630.0, 'debug/num_lat_loss': 1777.0, 'epoch': 4.12}

 41%|████      | 301/730 [26:39<36:53,  5.16s/it]
                                                 
{'train/ce_loss': 2.1875, 'train/diffusion_loss': 0.37552115321159363, 'epoch': 4.12}

 41%|████      | 301/730 [26:39<36:53,  5.16s/it]
                                                 
{'train/learning_rate_real': 1.6583584173605164e-05, 'epoch': 4.12}

 41%|████      | 301/730 [26:39<36:53,  5.16s/it]
                                                 
{'debug/num_tok_total': 2865.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2865.0, 'debug/num_lat_loss': 1781.0, 'epoch': 4.12}

 41%|████      | 301/730 [26:41<36:53,  5.16s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.35189786553382874, 'epoch': 4.12}

 41%|████      | 301/730 [26:41<36:53,  5.16s/it]
                                                 
{'train/learning_rate_real': 1.6583584173605164e-05, 'epoch': 4.12}

 41%|████      | 301/730 [26:41<36:53,  5.16s/it]
                                                 
{'debug/num_tok_total': 2456.0, 'debug/num_tok_loss': 1800.0, 'debug/num_lat_total': 2456.0, 'debug/num_lat_loss': 1800.0, 'epoch': 4.12}

 41%|████      | 301/730 [26:42<36:53,  5.16s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.44780659675598145, 'epoch': 4.12}

 41%|████      | 301/730 [26:42<36:53,  5.16s/it]
                                                 
{'train/learning_rate_real': 1.6583584173605164e-05, 'epoch': 4.12}

 41%|████      | 301/730 [26:42<36:53,  5.16s/it]
 41%|████▏     | 302/730 [26:42<36:54,  5.17s/it]
                                                 
{'debug/num_tok_total': 2439.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2439.0, 'debug/num_lat_loss': 1802.0, 'epoch': 4.14}

 41%|████▏     | 302/730 [26:43<36:54,  5.17s/it]
                                                 
{'train/ce_loss': 2.046875, 'train/diffusion_loss': 0.4349138140678406, 'epoch': 4.14}

 41%|████▏     | 302/730 [26:43<36:54,  5.17s/it]
                                                 
{'train/learning_rate_real': 1.6531121441356362e-05, 'epoch': 4.14}

 41%|████▏     | 302/730 [26:43<36:54,  5.17s/it]
                                                 
{'debug/num_tok_total': 3072.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 3072.0, 'debug/num_lat_loss': 1780.0, 'epoch': 4.14}

 41%|████▏     | 302/730 [26:45<36:54,  5.17s/it]
                                                 
{'train/ce_loss': 2.203125, 'train/diffusion_loss': 0.28650379180908203, 'epoch': 4.14}

 41%|████▏     | 302/730 [26:45<36:54,  5.17s/it]
                                                 
{'train/learning_rate_real': 1.6531121441356362e-05, 'epoch': 4.14}

 41%|████▏     | 302/730 [26:45<36:54,  5.17s/it]
                                                 
{'debug/num_tok_total': 2813.0, 'debug/num_tok_loss': 1754.0, 'debug/num_lat_total': 2813.0, 'debug/num_lat_loss': 1754.0, 'epoch': 4.14}

 41%|████▏     | 302/730 [26:46<36:54,  5.17s/it]
                                                 
{'train/ce_loss': 1.9609375, 'train/diffusion_loss': 0.33171990513801575, 'epoch': 4.14}

 41%|████▏     | 302/730 [26:46<36:54,  5.17s/it]
                                                 
{'train/learning_rate_real': 1.6531121441356362e-05, 'epoch': 4.14}

 41%|████▏     | 302/730 [26:46<36:54,  5.17s/it]
                                                 
{'debug/num_tok_total': 3066.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 3066.0, 'debug/num_lat_loss': 1782.0, 'epoch': 4.14}

 41%|████▏     | 302/730 [26:47<36:54,  5.17s/it]
                                                 
{'train/ce_loss': 2.109375, 'train/diffusion_loss': 0.33921998739242554, 'epoch': 4.14}

 41%|████▏     | 302/730 [26:47<36:54,  5.17s/it]
                                                 
{'train/learning_rate_real': 1.6531121441356362e-05, 'epoch': 4.14}

 41%|████▏     | 302/730 [26:47<36:54,  5.17s/it]
 42%|████▏     | 303/730 [26:48<37:30,  5.27s/it]
                                                 
{'debug/num_tok_total': 2885.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2885.0, 'debug/num_lat_loss': 1796.0, 'epoch': 4.15}

 42%|████▏     | 303/730 [26:49<37:30,  5.27s/it]
                                                 
{'train/ce_loss': 2.078125, 'train/diffusion_loss': 0.3834764361381531, 'epoch': 4.15}

 42%|████▏     | 303/730 [26:49<37:30,  5.27s/it]
                                                 
{'train/learning_rate_real': 1.6478579338738577e-05, 'epoch': 4.15}

 42%|████▏     | 303/730 [26:49<37:30,  5.27s/it]
                                                 
{'debug/num_tok_total': 2409.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 2409.0, 'debug/num_lat_loss': 1765.0, 'epoch': 4.15}

 42%|████▏     | 303/730 [26:50<37:30,  5.27s/it]
                                                 
{'train/ce_loss': 1.90625, 'train/diffusion_loss': 0.44248247146606445, 'epoch': 4.15}

 42%|████▏     | 303/730 [26:50<37:30,  5.27s/it]
                                                 
{'train/learning_rate_real': 1.6478579338738577e-05, 'epoch': 4.15}

 42%|████▏     | 303/730 [26:50<37:30,  5.27s/it]
                                                 
{'debug/num_tok_total': 2679.0, 'debug/num_tok_loss': 1798.0, 'debug/num_lat_total': 2679.0, 'debug/num_lat_loss': 1798.0, 'epoch': 4.15}

 42%|████▏     | 303/730 [26:51<37:30,  5.27s/it]
                                                 
{'train/ce_loss': 2.109375, 'train/diffusion_loss': 0.42659714818000793, 'epoch': 4.15}

 42%|████▏     | 303/730 [26:51<37:30,  5.27s/it]
                                                 
{'train/learning_rate_real': 1.6478579338738577e-05, 'epoch': 4.15}

 42%|████▏     | 303/730 [26:51<37:30,  5.27s/it]
                                                 
{'debug/num_tok_total': 2673.0, 'debug/num_tok_loss': 1798.0, 'debug/num_lat_total': 2673.0, 'debug/num_lat_loss': 1798.0, 'epoch': 4.15}

 42%|████▏     | 303/730 [26:53<37:30,  5.27s/it]
                                                 
{'train/ce_loss': 1.9921875, 'train/diffusion_loss': 0.40202435851097107, 'epoch': 4.15}

 42%|████▏     | 303/730 [26:53<37:30,  5.27s/it]
                                                 
{'train/learning_rate_real': 1.6478579338738577e-05, 'epoch': 4.15}

 42%|████▏     | 303/730 [26:53<37:30,  5.27s/it]
 42%|████▏     | 304/730 [26:53<37:22,  5.26s/it]
                                                 
{'debug/num_tok_total': 2186.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2186.0, 'debug/num_lat_loss': 1763.0, 'epoch': 4.16}

 42%|████▏     | 304/730 [26:54<37:22,  5.26s/it]
                                                 
{'train/ce_loss': 1.9453125, 'train/diffusion_loss': 0.4802837073802948, 'epoch': 4.16}

 42%|████▏     | 304/730 [26:54<37:22,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.6425958900274362e-05, 'epoch': 4.16}

 42%|████▏     | 304/730 [26:54<37:22,  5.26s/it]
                                                 
{'debug/num_tok_total': 2456.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2456.0, 'debug/num_lat_loss': 1795.0, 'epoch': 4.16}

 42%|████▏     | 304/730 [26:55<37:22,  5.26s/it]
                                                 
{'train/ce_loss': 2.015625, 'train/diffusion_loss': 0.4443361461162567, 'epoch': 4.16}

 42%|████▏     | 304/730 [26:55<37:22,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.6425958900274362e-05, 'epoch': 4.16}

 42%|████▏     | 304/730 [26:55<37:22,  5.26s/it]
                                                 
{'debug/num_tok_total': 2061.0, 'debug/num_tok_loss': 1623.0, 'debug/num_lat_total': 2061.0, 'debug/num_lat_loss': 1623.0, 'epoch': 4.16}

 42%|████▏     | 304/730 [26:56<37:22,  5.26s/it]
                                                 
{'train/ce_loss': 1.96875, 'train/diffusion_loss': 0.46792516112327576, 'epoch': 4.16}

 42%|████▏     | 304/730 [26:56<37:22,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.6425958900274362e-05, 'epoch': 4.16}

 42%|████▏     | 304/730 [26:56<37:22,  5.26s/it]
                                                 
{'debug/num_tok_total': 2282.0, 'debug/num_tok_loss': 1628.0, 'debug/num_lat_total': 2282.0, 'debug/num_lat_loss': 1628.0, 'epoch': 4.16}

 42%|████▏     | 304/730 [26:58<37:22,  5.26s/it]
                                                 
{'train/ce_loss': 1.890625, 'train/diffusion_loss': 0.4190717339515686, 'epoch': 4.16}

 42%|████▏     | 304/730 [26:58<37:22,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.6425958900274362e-05, 'epoch': 4.16}

 42%|████▏     | 304/730 [26:58<37:22,  5.26s/it]
 42%|████▏     | 305/730 [26:58<36:40,  5.18s/it]
                                                 
{'debug/num_tok_total': 2866.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2866.0, 'debug/num_lat_loss': 1794.0, 'epoch': 4.18}

 42%|████▏     | 305/730 [26:59<36:40,  5.18s/it]
                                                 
{'train/ce_loss': 2.09375, 'train/diffusion_loss': 0.3558429181575775, 'epoch': 4.18}

 42%|████▏     | 305/730 [26:59<36:40,  5.18s/it]
                                                 
{'train/learning_rate_real': 1.6373261162028666e-05, 'epoch': 4.18}

 42%|████▏     | 305/730 [26:59<36:40,  5.18s/it]
                                                 
{'debug/num_tok_total': 2636.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2636.0, 'debug/num_lat_loss': 1781.0, 'epoch': 4.18}

 42%|████▏     | 305/730 [27:01<36:40,  5.18s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.4292551577091217, 'epoch': 4.18}

 42%|████▏     | 305/730 [27:01<36:40,  5.18s/it]
                                                 
{'train/learning_rate_real': 1.6373261162028666e-05, 'epoch': 4.18}

 42%|████▏     | 305/730 [27:01<36:40,  5.18s/it]
                                                 
{'debug/num_tok_total': 2209.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2209.0, 'debug/num_lat_loss': 1784.0, 'epoch': 4.18}

 42%|████▏     | 305/730 [27:02<36:40,  5.18s/it]
                                                 
{'train/ce_loss': 2.1875, 'train/diffusion_loss': 0.49365416169166565, 'epoch': 4.18}

 42%|████▏     | 305/730 [27:02<36:40,  5.18s/it]
                                                 
{'train/learning_rate_real': 1.6373261162028666e-05, 'epoch': 4.18}

 42%|████▏     | 305/730 [27:02<36:40,  5.18s/it]
                                                 
{'debug/num_tok_total': 2661.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2661.0, 'debug/num_lat_loss': 1792.0, 'epoch': 4.18}

 42%|████▏     | 305/730 [27:03<36:40,  5.18s/it]
                                                 
{'train/ce_loss': 2.09375, 'train/diffusion_loss': 0.3843974769115448, 'epoch': 4.18}

 42%|████▏     | 305/730 [27:03<36:40,  5.18s/it]
                                                 
{'train/learning_rate_real': 1.6373261162028666e-05, 'epoch': 4.18}

 42%|████▏     | 305/730 [27:03<36:40,  5.18s/it]
 42%|████▏     | 306/730 [27:03<37:06,  5.25s/it]
                                                 
{'debug/num_tok_total': 2644.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2644.0, 'debug/num_lat_loss': 1779.0, 'epoch': 4.19}

 42%|████▏     | 306/730 [27:05<37:06,  5.25s/it]
                                                 
{'train/ce_loss': 2.15625, 'train/diffusion_loss': 0.4051734507083893, 'epoch': 4.19}

 42%|████▏     | 306/730 [27:05<37:06,  5.25s/it]
                                                 
{'train/learning_rate_real': 1.6320487161588423e-05, 'epoch': 4.19}

 42%|████▏     | 306/730 [27:05<37:06,  5.25s/it]
                                                 
{'debug/num_tok_total': 2429.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2429.0, 'debug/num_lat_loss': 1786.0, 'epoch': 4.19}

 42%|████▏     | 306/730 [27:06<37:06,  5.25s/it]
                                                 
{'train/ce_loss': 1.953125, 'train/diffusion_loss': 0.4380723536014557, 'epoch': 4.19}

 42%|████▏     | 306/730 [27:06<37:06,  5.25s/it]
                                                 
{'train/learning_rate_real': 1.6320487161588423e-05, 'epoch': 4.19}

 42%|████▏     | 306/730 [27:06<37:06,  5.25s/it]
                                                 
{'debug/num_tok_total': 2228.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2228.0, 'debug/num_lat_loss': 1802.0, 'epoch': 4.19}

 42%|████▏     | 306/730 [27:07<37:06,  5.25s/it]
                                                 
{'train/ce_loss': 2.09375, 'train/diffusion_loss': 0.48873797059059143, 'epoch': 4.19}

 42%|████▏     | 306/730 [27:07<37:06,  5.25s/it]
                                                 
{'train/learning_rate_real': 1.6320487161588423e-05, 'epoch': 4.19}

 42%|████▏     | 306/730 [27:07<37:06,  5.25s/it]
                                                 
{'debug/num_tok_total': 2467.0, 'debug/num_tok_loss': 1809.0, 'debug/num_lat_total': 2467.0, 'debug/num_lat_loss': 1809.0, 'epoch': 4.19}

 42%|████▏     | 306/730 [27:08<37:06,  5.25s/it]
                                                 
{'train/ce_loss': 2.21875, 'train/diffusion_loss': 0.45243874192237854, 'epoch': 4.19}

 42%|████▏     | 306/730 [27:08<37:06,  5.25s/it]
                                                 
{'train/learning_rate_real': 1.6320487161588423e-05, 'epoch': 4.19}

 42%|████▏     | 306/730 [27:08<37:06,  5.25s/it]
 42%|████▏     | 307/730 [27:09<36:37,  5.19s/it]
                                                 
{'debug/num_tok_total': 2608.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 2608.0, 'debug/num_lat_loss': 1767.0, 'epoch': 4.21}

 42%|████▏     | 307/730 [27:10<36:37,  5.19s/it]
                                                 
{'train/ce_loss': 1.953125, 'train/diffusion_loss': 0.4109704792499542, 'epoch': 4.21}

 42%|████▏     | 307/730 [27:10<36:37,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.6267637938042126e-05, 'epoch': 4.21}

 42%|████▏     | 307/730 [27:10<36:37,  5.19s/it]
                                                 
{'debug/num_tok_total': 2144.0, 'debug/num_tok_loss': 1709.0, 'debug/num_lat_total': 2144.0, 'debug/num_lat_loss': 1709.0, 'epoch': 4.21}

 42%|████▏     | 307/730 [27:11<36:37,  5.19s/it]
                                                 
{'train/ce_loss': 2.09375, 'train/diffusion_loss': 0.47451481223106384, 'epoch': 4.21}

 42%|████▏     | 307/730 [27:11<36:37,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.6267637938042126e-05, 'epoch': 4.21}

 42%|████▏     | 307/730 [27:11<36:37,  5.19s/it]
                                                 
{'debug/num_tok_total': 2622.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2622.0, 'debug/num_lat_loss': 1774.0, 'epoch': 4.21}

 42%|████▏     | 307/730 [27:12<36:37,  5.19s/it]
                                                 
{'train/ce_loss': 1.9375, 'train/diffusion_loss': 0.4249735176563263, 'epoch': 4.21}

 42%|████▏     | 307/730 [27:12<36:37,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.6267637938042126e-05, 'epoch': 4.21}

 42%|████▏     | 307/730 [27:12<36:37,  5.19s/it]
                                                 
{'debug/num_tok_total': 3106.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 3106.0, 'debug/num_lat_loss': 1802.0, 'epoch': 4.21}

 42%|████▏     | 307/730 [27:13<36:37,  5.19s/it]
                                                 
{'train/ce_loss': 1.9453125, 'train/diffusion_loss': 0.34032824635505676, 'epoch': 4.21}

 42%|████▏     | 307/730 [27:13<36:37,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.6267637938042126e-05, 'epoch': 4.21}

 42%|████▏     | 307/730 [27:13<36:37,  5.19s/it]
 42%|████▏     | 308/730 [27:14<36:35,  5.20s/it]
                                                 
{'debug/num_tok_total': 2863.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2863.0, 'debug/num_lat_loss': 1788.0, 'epoch': 4.22}

 42%|████▏     | 308/730 [27:15<36:35,  5.20s/it]
                                                 
{'train/ce_loss': 2.078125, 'train/diffusion_loss': 0.35815247893333435, 'epoch': 4.22}

 42%|████▏     | 308/730 [27:15<36:35,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.6214714531959348e-05, 'epoch': 4.22}

 42%|████▏     | 308/730 [27:15<36:35,  5.20s/it]
                                                 
{'debug/num_tok_total': 2070.0, 'debug/num_tok_loss': 1714.0, 'debug/num_lat_total': 2070.0, 'debug/num_lat_loss': 1714.0, 'epoch': 4.22}

 42%|████▏     | 308/730 [27:16<36:35,  5.20s/it]
                                                 
{'train/ce_loss': 1.9453125, 'train/diffusion_loss': 0.463321715593338, 'epoch': 4.22}

 42%|████▏     | 308/730 [27:16<36:35,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.6214714531959348e-05, 'epoch': 4.22}

 42%|████▏     | 308/730 [27:16<36:35,  5.20s/it]
                                                 
{'debug/num_tok_total': 2644.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2644.0, 'debug/num_lat_loss': 1774.0, 'epoch': 4.22}

 42%|████▏     | 308/730 [27:17<36:35,  5.20s/it]
                                                 
{'train/ce_loss': 1.96875, 'train/diffusion_loss': 0.4105816185474396, 'epoch': 4.22}

 42%|████▏     | 308/730 [27:17<36:35,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.6214714531959348e-05, 'epoch': 4.22}

 42%|████▏     | 308/730 [27:17<36:35,  5.20s/it]
                                                 
{'debug/num_tok_total': 2862.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2862.0, 'debug/num_lat_loss': 1777.0, 'epoch': 4.22}

 42%|████▏     | 308/730 [27:19<36:35,  5.20s/it]
                                                 
{'train/ce_loss': 2.25, 'train/diffusion_loss': 0.3394912779331207, 'epoch': 4.22}

 42%|████▏     | 308/730 [27:19<36:35,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.6214714531959348e-05, 'epoch': 4.22}

 42%|████▏     | 308/730 [27:19<36:35,  5.20s/it]
 42%|████▏     | 309/730 [27:19<36:44,  5.24s/it]
                                                 
{'debug/num_tok_total': 2594.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2594.0, 'debug/num_lat_loss': 1763.0, 'epoch': 4.23}

 42%|████▏     | 309/730 [27:20<36:44,  5.24s/it]
                                                 
{'train/ce_loss': 1.8828125, 'train/diffusion_loss': 0.392853707075119, 'epoch': 4.23}

 42%|████▏     | 309/730 [27:20<36:44,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.6161717985370302e-05, 'epoch': 4.23}

 42%|████▏     | 309/730 [27:20<36:44,  5.24s/it]
                                                 
{'debug/num_tok_total': 2643.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2643.0, 'debug/num_lat_loss': 1790.0, 'epoch': 4.23}

 42%|████▏     | 309/730 [27:21<36:44,  5.24s/it]
                                                 
{'train/ce_loss': 2.078125, 'train/diffusion_loss': 0.4154187738895416, 'epoch': 4.23}

 42%|████▏     | 309/730 [27:21<36:44,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.6161717985370302e-05, 'epoch': 4.23}

 42%|████▏     | 309/730 [27:21<36:44,  5.24s/it]
                                                 
{'debug/num_tok_total': 3125.0, 'debug/num_tok_loss': 1815.0, 'debug/num_lat_total': 3125.0, 'debug/num_lat_loss': 1815.0, 'epoch': 4.23}

 42%|████▏     | 309/730 [27:23<36:44,  5.24s/it]
                                                 
{'train/ce_loss': 1.9296875, 'train/diffusion_loss': 0.33852076530456543, 'epoch': 4.23}

 42%|████▏     | 309/730 [27:23<36:44,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.6161717985370302e-05, 'epoch': 4.23}

 42%|████▏     | 309/730 [27:23<36:44,  5.24s/it]
                                                 
{'debug/num_tok_total': 2405.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2405.0, 'debug/num_lat_loss': 1771.0, 'epoch': 4.23}

 42%|████▏     | 309/730 [27:24<36:44,  5.24s/it]
                                                 
{'train/ce_loss': 2.046875, 'train/diffusion_loss': 0.4304135739803314, 'epoch': 4.23}

 42%|████▏     | 309/730 [27:24<36:44,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.6161717985370302e-05, 'epoch': 4.23}

 42%|████▏     | 309/730 [27:24<36:44,  5.24s/it]03/16/2026 07:14:56 - INFO - __main__ - LoRA debug step 310: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 42%|████▏     | 310/730 [27:25<37:01,  5.29s/it]
                                                 
{'loss': 2.6415, 'grad_norm': 1.5299005508422852, 'learning_rate': 1.6161717985370302e-05, 'epoch': 4.25}

 42%|████▏     | 310/730 [27:25<37:01,  5.29s/it]
                                                 
{'debug/num_tok_total': 2648.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2648.0, 'debug/num_lat_loss': 1785.0, 'epoch': 4.25}

 42%|████▏     | 310/730 [27:26<37:01,  5.29s/it]
                                                 
{'train/ce_loss': 2.015625, 'train/diffusion_loss': 0.417819619178772, 'epoch': 4.25}

 42%|████▏     | 310/730 [27:26<37:01,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.6108649341745262e-05, 'epoch': 4.25}

 42%|████▏     | 310/730 [27:26<37:01,  5.29s/it]
                                                 
{'debug/num_tok_total': 2682.0, 'debug/num_tok_loss': 1807.0, 'debug/num_lat_total': 2682.0, 'debug/num_lat_loss': 1807.0, 'epoch': 4.25}

 42%|████▏     | 310/730 [27:27<37:01,  5.29s/it]
                                                 
{'train/ce_loss': 2.265625, 'train/diffusion_loss': 0.39688125252723694, 'epoch': 4.25}

 42%|████▏     | 310/730 [27:27<37:01,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.6108649341745262e-05, 'epoch': 4.25}

 42%|████▏     | 310/730 [27:27<37:01,  5.29s/it]
                                                 
{'debug/num_tok_total': 3049.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 3049.0, 'debug/num_lat_loss': 1775.0, 'epoch': 4.25}

 42%|████▏     | 310/730 [27:28<37:01,  5.29s/it]
                                                 
{'train/ce_loss': 1.9921875, 'train/diffusion_loss': 0.30879974365234375, 'epoch': 4.25}

 42%|████▏     | 310/730 [27:28<37:01,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.6108649341745262e-05, 'epoch': 4.25}

 42%|████▏     | 310/730 [27:28<37:01,  5.29s/it]
                                                 
{'debug/num_tok_total': 2435.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2435.0, 'debug/num_lat_loss': 1780.0, 'epoch': 4.25}

 42%|████▏     | 310/730 [27:29<37:01,  5.29s/it]
                                                 
{'train/ce_loss': 2.015625, 'train/diffusion_loss': 0.4344693720340729, 'epoch': 4.25}

 42%|████▏     | 310/730 [27:29<37:01,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.6108649341745262e-05, 'epoch': 4.25}

 42%|████▏     | 310/730 [27:29<37:01,  5.29s/it]
 43%|████▎     | 311/730 [27:30<36:49,  5.27s/it]
                                                 
{'debug/num_tok_total': 2609.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2609.0, 'debug/num_lat_loss': 1766.0, 'epoch': 4.26}

 43%|████▎     | 311/730 [27:31<36:49,  5.27s/it]
                                                 
{'train/ce_loss': 2.078125, 'train/diffusion_loss': 0.3929198086261749, 'epoch': 4.26}

 43%|████▎     | 311/730 [27:31<36:49,  5.27s/it]
                                                 
{'train/learning_rate_real': 1.6055509645974082e-05, 'epoch': 4.26}

 43%|████▎     | 311/730 [27:31<36:49,  5.27s/it]
                                                 
{'debug/num_tok_total': 2700.0, 'debug/num_tok_loss': 1815.0, 'debug/num_lat_total': 2700.0, 'debug/num_lat_loss': 1815.0, 'epoch': 4.26}

 43%|████▎     | 311/730 [27:32<36:49,  5.27s/it]
                                                 
{'train/ce_loss': 1.8984375, 'train/diffusion_loss': 0.4018617272377014, 'epoch': 4.26}

 43%|████▎     | 311/730 [27:32<36:49,  5.27s/it]
                                                 
{'train/learning_rate_real': 1.6055509645974082e-05, 'epoch': 4.26}

 43%|████▎     | 311/730 [27:32<36:49,  5.27s/it]
                                                 
{'debug/num_tok_total': 2883.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2883.0, 'debug/num_lat_loss': 1795.0, 'epoch': 4.26}

 43%|████▎     | 311/730 [27:33<36:49,  5.27s/it]
                                                 
{'train/ce_loss': 2.203125, 'train/diffusion_loss': 0.3610696494579315, 'epoch': 4.26}

 43%|████▎     | 311/730 [27:33<36:49,  5.27s/it]
                                                 
{'train/learning_rate_real': 1.6055509645974082e-05, 'epoch': 4.26}

 43%|████▎     | 311/730 [27:33<36:49,  5.27s/it]
                                                 
{'debug/num_tok_total': 2217.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2217.0, 'debug/num_lat_loss': 1789.0, 'epoch': 4.26}

 43%|████▎     | 311/730 [27:35<36:49,  5.27s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.45695215463638306, 'epoch': 4.26}

 43%|████▎     | 311/730 [27:35<36:49,  5.27s/it]
                                                 
{'train/learning_rate_real': 1.6055509645974082e-05, 'epoch': 4.26}

 43%|████▎     | 311/730 [27:35<36:49,  5.27s/it]
 43%|████▎     | 312/730 [27:35<36:32,  5.25s/it]
                                                 
{'debug/num_tok_total': 2225.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2225.0, 'debug/num_lat_loss': 1783.0, 'epoch': 4.27}

 43%|████▎     | 312/730 [27:36<36:32,  5.25s/it]
                                                 
{'train/ce_loss': 2.640625, 'train/diffusion_loss': 0.49278274178504944, 'epoch': 4.27}

 43%|████▎     | 312/730 [27:36<36:32,  5.25s/it]
                                                 
{'train/learning_rate_real': 1.6002299944345555e-05, 'epoch': 4.27}

 43%|████▎     | 312/730 [27:36<36:32,  5.25s/it]
                                                 
{'debug/num_tok_total': 2859.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2859.0, 'debug/num_lat_loss': 1784.0, 'epoch': 4.27}

 43%|████▎     | 312/730 [27:37<36:32,  5.25s/it]
                                                 
{'train/ce_loss': 1.96875, 'train/diffusion_loss': 0.3408409357070923, 'epoch': 4.27}

 43%|████▎     | 312/730 [27:37<36:32,  5.25s/it]
                                                 
{'train/learning_rate_real': 1.6002299944345555e-05, 'epoch': 4.27}

 43%|████▎     | 312/730 [27:37<36:32,  5.25s/it]
                                                 
{'debug/num_tok_total': 2893.0, 'debug/num_tok_loss': 1810.0, 'debug/num_lat_total': 2893.0, 'debug/num_lat_loss': 1810.0, 'epoch': 4.27}

 43%|████▎     | 312/730 [27:39<36:32,  5.25s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.3809254467487335, 'epoch': 4.27}

 43%|████▎     | 312/730 [27:39<36:32,  5.25s/it]
                                                 
{'train/learning_rate_real': 1.6002299944345555e-05, 'epoch': 4.27}

 43%|████▎     | 312/730 [27:39<36:32,  5.25s/it]
                                                 
{'debug/num_tok_total': 2274.0, 'debug/num_tok_loss': 1825.0, 'debug/num_lat_total': 2274.0, 'debug/num_lat_loss': 1825.0, 'epoch': 4.27}

 43%|████▎     | 312/730 [27:40<36:32,  5.25s/it]
                                                 
{'train/ce_loss': 1.890625, 'train/diffusion_loss': 0.49229633808135986, 'epoch': 4.27}

 43%|████▎     | 312/730 [27:40<36:32,  5.25s/it]
                                                 
{'train/learning_rate_real': 1.6002299944345555e-05, 'epoch': 4.27}

 43%|████▎     | 312/730 [27:40<36:32,  5.25s/it]
 43%|████▎     | 313/730 [27:40<36:21,  5.23s/it]
                                                 
{'debug/num_tok_total': 2881.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2881.0, 'debug/num_lat_loss': 1796.0, 'epoch': 4.29}

 43%|████▎     | 313/730 [27:41<36:21,  5.23s/it]
                                                 
{'train/ce_loss': 2.125, 'train/diffusion_loss': 0.3535464107990265, 'epoch': 4.29}

 43%|████▎     | 313/730 [27:41<36:21,  5.23s/it]
                                                 
{'train/learning_rate_real': 1.594902128452688e-05, 'epoch': 4.29}

 43%|████▎     | 313/730 [27:41<36:21,  5.23s/it]
                                                 
{'debug/num_tok_total': 3162.0, 'debug/num_tok_loss': 1718.0, 'debug/num_lat_total': 3162.0, 'debug/num_lat_loss': 1718.0, 'epoch': 4.29}

 43%|████▎     | 313/730 [27:43<36:21,  5.23s/it]
                                                 
{'train/ce_loss': 2.171875, 'train/diffusion_loss': 0.29265493154525757, 'epoch': 4.29}

 43%|████▎     | 313/730 [27:43<36:21,  5.23s/it]
                                                 
{'train/learning_rate_real': 1.594902128452688e-05, 'epoch': 4.29}

 43%|████▎     | 313/730 [27:43<36:21,  5.23s/it]
                                                 
{'debug/num_tok_total': 2659.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2659.0, 'debug/num_lat_loss': 1796.0, 'epoch': 4.29}

 43%|████▎     | 313/730 [27:44<36:21,  5.23s/it]
                                                 
{'train/ce_loss': 2.078125, 'train/diffusion_loss': 0.43232548236846924, 'epoch': 4.29}

 43%|████▎     | 313/730 [27:44<36:21,  5.23s/it]
                                                 
{'train/learning_rate_real': 1.594902128452688e-05, 'epoch': 4.29}

 43%|████▎     | 313/730 [27:44<36:21,  5.23s/it]
                                                 
{'debug/num_tok_total': 2651.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2651.0, 'debug/num_lat_loss': 1777.0, 'epoch': 4.29}

 43%|████▎     | 313/730 [27:45<36:21,  5.23s/it]
                                                 
{'train/ce_loss': 2.203125, 'train/diffusion_loss': 0.38878774642944336, 'epoch': 4.29}

 43%|████▎     | 313/730 [27:45<36:21,  5.23s/it]
                                                 
{'train/learning_rate_real': 1.594902128452688e-05, 'epoch': 4.29}

 43%|████▎     | 313/730 [27:45<36:21,  5.23s/it]
 43%|████▎     | 314/730 [27:46<37:10,  5.36s/it]
                                                 
{'debug/num_tok_total': 2966.0, 'debug/num_tok_loss': 1617.0, 'debug/num_lat_total': 2966.0, 'debug/num_lat_loss': 1617.0, 'epoch': 4.3}

 43%|████▎     | 314/730 [27:47<37:10,  5.36s/it]
                                                 
{'train/ce_loss': 1.984375, 'train/diffusion_loss': 0.28403106331825256, 'epoch': 4.3}

 43%|████▎     | 314/730 [27:47<37:10,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.5895674715542985e-05, 'epoch': 4.3}

 43%|████▎     | 314/730 [27:47<37:10,  5.36s/it]
                                                 
{'debug/num_tok_total': 2427.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2427.0, 'debug/num_lat_loss': 1785.0, 'epoch': 4.3}

 43%|████▎     | 314/730 [27:48<37:10,  5.36s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.4388461410999298, 'epoch': 4.3}

 43%|████▎     | 314/730 [27:48<37:10,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.5895674715542985e-05, 'epoch': 4.3}

 43%|████▎     | 314/730 [27:48<37:10,  5.36s/it]
                                                 
{'debug/num_tok_total': 2100.0, 'debug/num_tok_loss': 1614.0, 'debug/num_lat_total': 2100.0, 'debug/num_lat_loss': 1614.0, 'epoch': 4.3}

 43%|████▎     | 314/730 [27:49<37:10,  5.36s/it]
                                                 
{'train/ce_loss': 1.9375, 'train/diffusion_loss': 0.4645499289035797, 'epoch': 4.3}

 43%|████▎     | 314/730 [27:49<37:10,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.5895674715542985e-05, 'epoch': 4.3}

 43%|████▎     | 314/730 [27:49<37:10,  5.36s/it]
                                                 
{'debug/num_tok_total': 2851.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2851.0, 'debug/num_lat_loss': 1773.0, 'epoch': 4.3}

 43%|████▎     | 314/730 [27:51<37:10,  5.36s/it]
                                                 
{'train/ce_loss': 1.9453125, 'train/diffusion_loss': 0.34211722016334534, 'epoch': 4.3}

 43%|████▎     | 314/730 [27:51<37:10,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.5895674715542985e-05, 'epoch': 4.3}

 43%|████▎     | 314/730 [27:51<37:10,  5.36s/it]
 43%|████▎     | 315/730 [27:51<36:46,  5.32s/it]
                                                 
{'debug/num_tok_total': 2198.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 2198.0, 'debug/num_lat_loss': 1769.0, 'epoch': 4.32}

 43%|████▎     | 315/730 [27:52<36:46,  5.32s/it]
                                                 
{'train/ce_loss': 2.09375, 'train/diffusion_loss': 0.48268836736679077, 'epoch': 4.32}

 43%|████▎     | 315/730 [27:52<36:46,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.584226128775589e-05, 'epoch': 4.32}

 43%|████▎     | 315/730 [27:52<36:46,  5.32s/it]
                                                 
{'debug/num_tok_total': 2348.0, 'debug/num_tok_loss': 1704.0, 'debug/num_lat_total': 2348.0, 'debug/num_lat_loss': 1704.0, 'epoch': 4.32}

 43%|████▎     | 315/730 [27:53<36:46,  5.32s/it]
                                                 
{'train/ce_loss': 1.9921875, 'train/diffusion_loss': 0.4525335729122162, 'epoch': 4.32}

 43%|████▎     | 315/730 [27:53<36:46,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.584226128775589e-05, 'epoch': 4.32}

 43%|████▎     | 315/730 [27:53<36:46,  5.32s/it]
                                                 
{'debug/num_tok_total': 2857.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2857.0, 'debug/num_lat_loss': 1779.0, 'epoch': 4.32}

 43%|████▎     | 315/730 [27:54<36:46,  5.32s/it]
                                                 
{'train/ce_loss': 1.984375, 'train/diffusion_loss': 0.33578816056251526, 'epoch': 4.32}

 43%|████▎     | 315/730 [27:54<36:46,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.584226128775589e-05, 'epoch': 4.32}

 43%|████▎     | 315/730 [27:54<36:46,  5.32s/it]
                                                 
{'debug/num_tok_total': 2849.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2849.0, 'debug/num_lat_loss': 1775.0, 'epoch': 4.32}

 43%|████▎     | 315/730 [27:56<36:46,  5.32s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.35660043358802795, 'epoch': 4.32}

 43%|████▎     | 315/730 [27:56<36:46,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.584226128775589e-05, 'epoch': 4.32}

 43%|████▎     | 315/730 [27:56<36:46,  5.32s/it]
 43%|████▎     | 316/730 [27:56<36:18,  5.26s/it]
                                                 
{'debug/num_tok_total': 2440.0, 'debug/num_tok_loss': 1798.0, 'debug/num_lat_total': 2440.0, 'debug/num_lat_loss': 1798.0, 'epoch': 4.33}

 43%|████▎     | 316/730 [27:57<36:18,  5.26s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.4368599057197571, 'epoch': 4.33}

 43%|████▎     | 316/730 [27:57<36:18,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.5788782052844015e-05, 'epoch': 4.33}

 43%|████▎     | 316/730 [27:57<36:18,  5.26s/it]
                                                 
{'debug/num_tok_total': 2404.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2404.0, 'debug/num_lat_loss': 1780.0, 'epoch': 4.33}

 43%|████▎     | 316/730 [27:58<36:18,  5.26s/it]
                                                 
{'train/ce_loss': 1.9609375, 'train/diffusion_loss': 0.4441825747489929, 'epoch': 4.33}

 43%|████▎     | 316/730 [27:58<36:18,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.5788782052844015e-05, 'epoch': 4.33}

 43%|████▎     | 316/730 [27:58<36:18,  5.26s/it]
                                                 
{'debug/num_tok_total': 2459.0, 'debug/num_tok_loss': 1806.0, 'debug/num_lat_total': 2459.0, 'debug/num_lat_loss': 1806.0, 'epoch': 4.33}

 43%|████▎     | 316/730 [28:00<36:18,  5.26s/it]
                                                 
{'train/ce_loss': 2.140625, 'train/diffusion_loss': 0.4697098135948181, 'epoch': 4.33}

 43%|████▎     | 316/730 [28:00<36:18,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.5788782052844015e-05, 'epoch': 4.33}

 43%|████▎     | 316/730 [28:00<36:18,  5.26s/it]
                                                 
{'debug/num_tok_total': 1995.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 1995.0, 'debug/num_lat_loss': 1783.0, 'epoch': 4.33}

 43%|████▎     | 316/730 [28:01<36:18,  5.26s/it]
                                                 
{'train/ce_loss': 1.9921875, 'train/diffusion_loss': 0.5420702695846558, 'epoch': 4.33}

 43%|████▎     | 316/730 [28:01<36:18,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.5788782052844015e-05, 'epoch': 4.33}

 43%|████▎     | 316/730 [28:01<36:18,  5.26s/it]
 43%|████▎     | 317/730 [28:01<35:19,  5.13s/it]
                                                 
{'debug/num_tok_total': 3307.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 3307.0, 'debug/num_lat_loss': 1793.0, 'epoch': 4.34}

 43%|████▎     | 317/730 [28:02<35:19,  5.13s/it]
                                                 
{'train/ce_loss': 2.03125, 'train/diffusion_loss': 0.28982651233673096, 'epoch': 4.34}

 43%|████▎     | 317/730 [28:02<35:19,  5.13s/it]
                                                 
{'train/learning_rate_real': 1.573523806378151e-05, 'epoch': 4.34}

 43%|████▎     | 317/730 [28:02<35:19,  5.13s/it]
                                                 
{'debug/num_tok_total': 2404.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 2404.0, 'debug/num_lat_loss': 1765.0, 'epoch': 4.34}

 43%|████▎     | 317/730 [28:03<35:19,  5.13s/it]
                                                 
{'train/ce_loss': 1.9453125, 'train/diffusion_loss': 0.43960776925086975, 'epoch': 4.34}

 43%|████▎     | 317/730 [28:03<35:19,  5.13s/it]
                                                 
{'train/learning_rate_real': 1.573523806378151e-05, 'epoch': 4.34}

 43%|████▎     | 317/730 [28:03<35:19,  5.13s/it]
                                                 
{'debug/num_tok_total': 3084.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 3084.0, 'debug/num_lat_loss': 1788.0, 'epoch': 4.34}

 43%|████▎     | 317/730 [28:05<35:19,  5.13s/it]
                                                 
{'train/ce_loss': 2.09375, 'train/diffusion_loss': 0.3340285122394562, 'epoch': 4.34}

 43%|████▎     | 317/730 [28:05<35:19,  5.13s/it]
                                                 
{'train/learning_rate_real': 1.573523806378151e-05, 'epoch': 4.34}

 43%|████▎     | 317/730 [28:05<35:19,  5.13s/it]
                                                 
{'debug/num_tok_total': 2621.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2621.0, 'debug/num_lat_loss': 1771.0, 'epoch': 4.34}

 43%|████▎     | 317/730 [28:06<35:19,  5.13s/it]
                                                 
{'train/ce_loss': 2.15625, 'train/diffusion_loss': 0.3833756148815155, 'epoch': 4.34}

 43%|████▎     | 317/730 [28:06<35:19,  5.13s/it]
                                                 
{'train/learning_rate_real': 1.573523806378151e-05, 'epoch': 4.34}

 43%|████▎     | 317/730 [28:06<35:19,  5.13s/it]
 44%|████▎     | 318/730 [28:06<35:57,  5.24s/it]
                                                 
{'debug/num_tok_total': 2106.0, 'debug/num_tok_loss': 1609.0, 'debug/num_lat_total': 2106.0, 'debug/num_lat_loss': 1609.0, 'epoch': 4.36}

 44%|████▎     | 318/730 [28:08<35:57,  5.24s/it]
                                                 
{'train/ce_loss': 2.40625, 'train/diffusion_loss': 0.47296836972236633, 'epoch': 4.36}

 44%|████▎     | 318/730 [28:08<35:57,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.568163037481747e-05, 'epoch': 4.36}

 44%|████▎     | 318/730 [28:08<35:57,  5.24s/it]
                                                 
{'debug/num_tok_total': 2826.0, 'debug/num_tok_loss': 1758.0, 'debug/num_lat_total': 2826.0, 'debug/num_lat_loss': 1758.0, 'epoch': 4.36}

 44%|████▎     | 318/730 [28:09<35:57,  5.24s/it]
                                                 
{'train/ce_loss': 1.90625, 'train/diffusion_loss': 0.34956642985343933, 'epoch': 4.36}

 44%|████▎     | 318/730 [28:09<35:57,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.568163037481747e-05, 'epoch': 4.36}

 44%|████▎     | 318/730 [28:09<35:57,  5.24s/it]
                                                 
{'debug/num_tok_total': 2050.0, 'debug/num_tok_loss': 1709.0, 'debug/num_lat_total': 2050.0, 'debug/num_lat_loss': 1709.0, 'epoch': 4.36}

 44%|████▎     | 318/730 [28:10<35:57,  5.24s/it]
                                                 
{'train/ce_loss': 2.03125, 'train/diffusion_loss': 0.5021321177482605, 'epoch': 4.36}

 44%|████▎     | 318/730 [28:10<35:57,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.568163037481747e-05, 'epoch': 4.36}

 44%|████▎     | 318/730 [28:10<35:57,  5.24s/it]
                                                 
{'debug/num_tok_total': 2369.0, 'debug/num_tok_loss': 1737.0, 'debug/num_lat_total': 2369.0, 'debug/num_lat_loss': 1737.0, 'epoch': 4.36}

 44%|████▎     | 318/730 [28:11<35:57,  5.24s/it]
                                                 
{'train/ce_loss': 2.234375, 'train/diffusion_loss': 0.44898754358291626, 'epoch': 4.36}

 44%|████▎     | 318/730 [28:11<35:57,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.568163037481747e-05, 'epoch': 4.36}

 44%|████▎     | 318/730 [28:11<35:57,  5.24s/it]
 44%|████▎     | 319/730 [28:12<35:31,  5.19s/it]
                                                 
{'debug/num_tok_total': 2398.0, 'debug/num_tok_loss': 1761.0, 'debug/num_lat_total': 2398.0, 'debug/num_lat_loss': 1761.0, 'epoch': 4.37}

 44%|████▎     | 319/730 [28:13<35:31,  5.19s/it]
                                                 
{'train/ce_loss': 1.96875, 'train/diffusion_loss': 0.41769304871559143, 'epoch': 4.37}

 44%|████▎     | 319/730 [28:13<35:31,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.562796004145522e-05, 'epoch': 4.37}

 44%|████▎     | 319/730 [28:13<35:31,  5.19s/it]
                                                 
{'debug/num_tok_total': 2821.0, 'debug/num_tok_loss': 1760.0, 'debug/num_lat_total': 2821.0, 'debug/num_lat_loss': 1760.0, 'epoch': 4.37}

 44%|████▎     | 319/730 [28:14<35:31,  5.19s/it]
                                                 
{'train/ce_loss': 1.9609375, 'train/diffusion_loss': 0.3780413866043091, 'epoch': 4.37}

 44%|████▎     | 319/730 [28:14<35:31,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.562796004145522e-05, 'epoch': 4.37}

 44%|████▎     | 319/730 [28:14<35:31,  5.19s/it]
                                                 
{'debug/num_tok_total': 2626.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2626.0, 'debug/num_lat_loss': 1776.0, 'epoch': 4.37}

 44%|████▎     | 319/730 [28:15<35:31,  5.19s/it]
                                                 
{'train/ce_loss': 1.9921875, 'train/diffusion_loss': 0.42228835821151733, 'epoch': 4.37}

 44%|████▎     | 319/730 [28:15<35:31,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.562796004145522e-05, 'epoch': 4.37}

 44%|████▎     | 319/730 [28:15<35:31,  5.19s/it]
                                                 
{'debug/num_tok_total': 2233.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 2233.0, 'debug/num_lat_loss': 1797.0, 'epoch': 4.37}

 44%|████▎     | 319/730 [28:16<35:31,  5.19s/it]
                                                 
{'train/ce_loss': 1.9765625, 'train/diffusion_loss': 0.47614622116088867, 'epoch': 4.37}

 44%|████▎     | 319/730 [28:16<35:31,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.562796004145522e-05, 'epoch': 4.37}

 44%|████▎     | 319/730 [28:16<35:31,  5.19s/it]03/16/2026 07:15:49 - INFO - __main__ - LoRA debug step 320: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 44%|████▍     | 320/730 [28:17<35:30,  5.20s/it]
                                                 
{'loss': 2.6111, 'grad_norm': 1.3093547821044922, 'learning_rate': 1.562796004145522e-05, 'epoch': 4.38}

 44%|████▍     | 320/730 [28:17<35:30,  5.20s/it]
                                                 
{'debug/num_tok_total': 2599.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2599.0, 'debug/num_lat_loss': 1766.0, 'epoch': 4.38}

 44%|████▍     | 320/730 [28:18<35:30,  5.20s/it]
                                                 
{'train/ce_loss': 1.9453125, 'train/diffusion_loss': 0.440812349319458, 'epoch': 4.38}

 44%|████▍     | 320/730 [28:18<35:30,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.5574228120431497e-05, 'epoch': 4.38}

 44%|████▍     | 320/730 [28:18<35:30,  5.20s/it]
                                                 
{'debug/num_tok_total': 2640.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2640.0, 'debug/num_lat_loss': 1780.0, 'epoch': 4.38}

 44%|████▍     | 320/730 [28:19<35:30,  5.20s/it]
                                                 
{'train/ce_loss': 1.9453125, 'train/diffusion_loss': 0.4079349935054779, 'epoch': 4.38}

 44%|████▍     | 320/730 [28:19<35:30,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.5574228120431497e-05, 'epoch': 4.38}

 44%|████▍     | 320/730 [28:19<35:30,  5.20s/it]
                                                 
{'debug/num_tok_total': 2788.0, 'debug/num_tok_loss': 1641.0, 'debug/num_lat_total': 2788.0, 'debug/num_lat_loss': 1641.0, 'epoch': 4.38}

 44%|████▍     | 320/730 [28:20<35:30,  5.20s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.33988478779792786, 'epoch': 4.38}

 44%|████▍     | 320/730 [28:20<35:30,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.5574228120431497e-05, 'epoch': 4.38}

 44%|████▍     | 320/730 [28:20<35:30,  5.20s/it]
                                                 
{'debug/num_tok_total': 2591.0, 'debug/num_tok_loss': 1754.0, 'debug/num_lat_total': 2591.0, 'debug/num_lat_loss': 1754.0, 'epoch': 4.38}

 44%|████▍     | 320/730 [28:22<35:30,  5.20s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.40686947107315063, 'epoch': 4.38}

 44%|████▍     | 320/730 [28:22<35:30,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.5574228120431497e-05, 'epoch': 4.38}

 44%|████▍     | 320/730 [28:22<35:30,  5.20s/it]
 44%|████▍     | 321/730 [28:22<35:44,  5.24s/it]
                                                 
{'debug/num_tok_total': 3035.0, 'debug/num_tok_loss': 1762.0, 'debug/num_lat_total': 3035.0, 'debug/num_lat_loss': 1762.0, 'epoch': 4.4}

 44%|████▍     | 321/730 [28:23<35:44,  5.24s/it]
                                                 
{'train/ce_loss': 1.9140625, 'train/diffusion_loss': 0.32785743474960327, 'epoch': 4.4}

 44%|████▍     | 321/730 [28:23<35:44,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.552043566969568e-05, 'epoch': 4.4}

 44%|████▍     | 321/730 [28:23<35:44,  5.24s/it]
                                                 
{'debug/num_tok_total': 2171.0, 'debug/num_tok_loss': 1754.0, 'debug/num_lat_total': 2171.0, 'debug/num_lat_loss': 1754.0, 'epoch': 4.4}

 44%|████▍     | 321/730 [28:24<35:44,  5.24s/it]
                                                 
{'train/ce_loss': 1.9765625, 'train/diffusion_loss': 0.47331783175468445, 'epoch': 4.4}

 44%|████▍     | 321/730 [28:24<35:44,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.552043566969568e-05, 'epoch': 4.4}

 44%|████▍     | 321/730 [28:24<35:44,  5.24s/it]
                                                 
{'debug/num_tok_total': 2424.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2424.0, 'debug/num_lat_loss': 1777.0, 'epoch': 4.4}

 44%|████▍     | 321/730 [28:26<35:44,  5.24s/it]
                                                 
{'train/ce_loss': 1.9609375, 'train/diffusion_loss': 0.47246018052101135, 'epoch': 4.4}

 44%|████▍     | 321/730 [28:26<35:44,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.552043566969568e-05, 'epoch': 4.4}

 44%|████▍     | 321/730 [28:26<35:44,  5.24s/it]
                                                 
{'debug/num_tok_total': 2851.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2851.0, 'debug/num_lat_loss': 1782.0, 'epoch': 4.4}

 44%|████▍     | 321/730 [28:27<35:44,  5.24s/it]
                                                 
{'train/ce_loss': 1.9375, 'train/diffusion_loss': 0.3304142653942108, 'epoch': 4.4}

 44%|████▍     | 321/730 [28:27<35:44,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.552043566969568e-05, 'epoch': 4.4}

 44%|████▍     | 321/730 [28:27<35:44,  5.24s/it]
 44%|████▍     | 322/730 [28:27<35:44,  5.26s/it]
                                                 
{'debug/num_tok_total': 2673.0, 'debug/num_tok_loss': 1608.0, 'debug/num_lat_total': 2673.0, 'debug/num_lat_loss': 1608.0, 'epoch': 4.41}

 44%|████▍     | 322/730 [28:29<35:44,  5.26s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.31228986382484436, 'epoch': 4.41}

 44%|████▍     | 322/730 [28:29<35:44,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.546658374838894e-05, 'epoch': 4.41}

 44%|████▍     | 322/730 [28:29<35:44,  5.26s/it]
                                                 
{'debug/num_tok_total': 2195.0, 'debug/num_tok_loss': 1760.0, 'debug/num_lat_total': 2195.0, 'debug/num_lat_loss': 1760.0, 'epoch': 4.41}

 44%|████▍     | 322/730 [28:30<35:44,  5.26s/it]
                                                 
{'train/ce_loss': 2.15625, 'train/diffusion_loss': 0.49159204959869385, 'epoch': 4.41}

 44%|████▍     | 322/730 [28:30<35:44,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.546658374838894e-05, 'epoch': 4.41}

 44%|████▍     | 322/730 [28:30<35:44,  5.26s/it]
                                                 
{'debug/num_tok_total': 2843.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2843.0, 'debug/num_lat_loss': 1773.0, 'epoch': 4.41}

 44%|████▍     | 322/730 [28:31<35:44,  5.26s/it]
                                                 
{'train/ce_loss': 1.9296875, 'train/diffusion_loss': 0.3542118966579437, 'epoch': 4.41}

 44%|████▍     | 322/730 [28:31<35:44,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.546658374838894e-05, 'epoch': 4.41}

 44%|████▍     | 322/730 [28:31<35:44,  5.26s/it]
                                                 
{'debug/num_tok_total': 2661.0, 'debug/num_tok_loss': 1804.0, 'debug/num_lat_total': 2661.0, 'debug/num_lat_loss': 1804.0, 'epoch': 4.41}

 44%|████▍     | 322/730 [28:32<35:44,  5.26s/it]
                                                 
{'train/ce_loss': 1.9765625, 'train/diffusion_loss': 0.3923184275627136, 'epoch': 4.41}

 44%|████▍     | 322/730 [28:32<35:44,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.546658374838894e-05, 'epoch': 4.41}

 44%|████▍     | 322/730 [28:32<35:44,  5.26s/it]
 44%|████▍     | 323/730 [28:33<36:00,  5.31s/it]
                                                 
{'debug/num_tok_total': 2645.0, 'debug/num_tok_loss': 1592.0, 'debug/num_lat_total': 2645.0, 'debug/num_lat_loss': 1592.0, 'epoch': 4.42}

 44%|████▍     | 323/730 [28:34<36:00,  5.31s/it]
                                                 
{'train/ce_loss': 2.03125, 'train/diffusion_loss': 0.34810689091682434, 'epoch': 4.42}

 44%|████▍     | 323/730 [28:34<36:00,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.5412673416823367e-05, 'epoch': 4.42}

 44%|████▍     | 323/730 [28:34<36:00,  5.31s/it]
                                                 
{'debug/num_tok_total': 3060.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 3060.0, 'debug/num_lat_loss': 1778.0, 'epoch': 4.42}

 44%|████▍     | 323/730 [28:35<36:00,  5.31s/it]
                                                 
{'train/ce_loss': 1.9921875, 'train/diffusion_loss': 0.3222844898700714, 'epoch': 4.42}

 44%|████▍     | 323/730 [28:35<36:00,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.5412673416823367e-05, 'epoch': 4.42}

 44%|████▍     | 323/730 [28:35<36:00,  5.31s/it]
                                                 
{'debug/num_tok_total': 2012.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2012.0, 'debug/num_lat_loss': 1787.0, 'epoch': 4.42}

 44%|████▍     | 323/730 [28:36<36:00,  5.31s/it]
                                                 
{'train/ce_loss': 1.953125, 'train/diffusion_loss': 0.5204927921295166, 'epoch': 4.42}

 44%|████▍     | 323/730 [28:36<36:00,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.5412673416823367e-05, 'epoch': 4.42}

 44%|████▍     | 323/730 [28:36<36:00,  5.31s/it]
                                                 
{'debug/num_tok_total': 3037.0, 'debug/num_tok_loss': 1762.0, 'debug/num_lat_total': 3037.0, 'debug/num_lat_loss': 1762.0, 'epoch': 4.42}

 44%|████▍     | 323/730 [28:38<36:00,  5.31s/it]
                                                 
{'train/ce_loss': 1.9375, 'train/diffusion_loss': 0.29368212819099426, 'epoch': 4.42}

 44%|████▍     | 323/730 [28:38<36:00,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.5412673416823367e-05, 'epoch': 4.42}

 44%|████▍     | 323/730 [28:38<36:00,  5.31s/it]
 44%|████▍     | 324/730 [28:38<35:46,  5.29s/it]
                                                 
{'debug/num_tok_total': 2469.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2469.0, 'debug/num_lat_loss': 1802.0, 'epoch': 4.44}

 44%|████▍     | 324/730 [28:39<35:46,  5.29s/it]
                                                 
{'train/ce_loss': 2.109375, 'train/diffusion_loss': 0.45224830508232117, 'epoch': 4.44}

 44%|████▍     | 324/730 [28:39<35:46,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.5358705736461147e-05, 'epoch': 4.44}

 44%|████▍     | 324/730 [28:39<35:46,  5.29s/it]
                                                 
{'debug/num_tok_total': 2617.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2617.0, 'debug/num_lat_loss': 1766.0, 'epoch': 4.44}

 44%|████▍     | 324/730 [28:40<35:46,  5.29s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.4042223393917084, 'epoch': 4.44}

 44%|████▍     | 324/730 [28:40<35:46,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.5358705736461147e-05, 'epoch': 4.44}

 44%|████▍     | 324/730 [28:40<35:46,  5.29s/it]
                                                 
{'debug/num_tok_total': 3032.0, 'debug/num_tok_loss': 1761.0, 'debug/num_lat_total': 3032.0, 'debug/num_lat_loss': 1761.0, 'epoch': 4.44}

 44%|████▍     | 324/730 [28:42<35:46,  5.29s/it]
                                                 
{'train/ce_loss': 2.03125, 'train/diffusion_loss': 0.36794137954711914, 'epoch': 4.44}

 44%|████▍     | 324/730 [28:42<35:46,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.5358705736461147e-05, 'epoch': 4.44}

 44%|████▍     | 324/730 [28:42<35:46,  5.29s/it]
                                                 
{'debug/num_tok_total': 2084.0, 'debug/num_tok_loss': 1660.0, 'debug/num_lat_total': 2084.0, 'debug/num_lat_loss': 1660.0, 'epoch': 4.44}

 44%|████▍     | 324/730 [28:43<35:46,  5.29s/it]
                                                 
{'train/ce_loss': 2.078125, 'train/diffusion_loss': 0.464321494102478, 'epoch': 4.44}

 44%|████▍     | 324/730 [28:43<35:46,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.5358705736461147e-05, 'epoch': 4.44}

 44%|████▍     | 324/730 [28:43<35:46,  5.29s/it]
 45%|████▍     | 325/730 [28:43<35:55,  5.32s/it]
                                                 
{'debug/num_tok_total': 2769.0, 'debug/num_tok_loss': 1631.0, 'debug/num_lat_total': 2769.0, 'debug/num_lat_loss': 1631.0, 'epoch': 4.45}

 45%|████▍     | 325/730 [28:45<35:55,  5.32s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.2974879741668701, 'epoch': 4.45}

 45%|████▍     | 325/730 [28:45<35:55,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.5304681769893603e-05, 'epoch': 4.45}

 45%|████▍     | 325/730 [28:45<35:55,  5.32s/it]
                                                 
{'debug/num_tok_total': 2836.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2836.0, 'debug/num_lat_loss': 1771.0, 'epoch': 4.45}

 45%|████▍     | 325/730 [28:46<35:55,  5.32s/it]
                                                 
{'train/ce_loss': 1.875, 'train/diffusion_loss': 0.3495127558708191, 'epoch': 4.45}

 45%|████▍     | 325/730 [28:46<35:55,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.5304681769893603e-05, 'epoch': 4.45}

 45%|████▍     | 325/730 [28:46<35:55,  5.32s/it]
                                                 
{'debug/num_tok_total': 2821.0, 'debug/num_tok_loss': 1759.0, 'debug/num_lat_total': 2821.0, 'debug/num_lat_loss': 1759.0, 'epoch': 4.45}

 45%|████▍     | 325/730 [28:47<35:55,  5.32s/it]
                                                 
{'train/ce_loss': 1.921875, 'train/diffusion_loss': 0.33067697286605835, 'epoch': 4.45}

 45%|████▍     | 325/730 [28:47<35:55,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.5304681769893603e-05, 'epoch': 4.45}

 45%|████▍     | 325/730 [28:47<35:55,  5.32s/it]
                                                 
{'debug/num_tok_total': 2887.0, 'debug/num_tok_loss': 1806.0, 'debug/num_lat_total': 2887.0, 'debug/num_lat_loss': 1806.0, 'epoch': 4.45}

 45%|████▍     | 325/730 [28:48<35:55,  5.32s/it]
                                                 
{'train/ce_loss': 2.171875, 'train/diffusion_loss': 0.36880069971084595, 'epoch': 4.45}

 45%|████▍     | 325/730 [28:49<35:55,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.5304681769893603e-05, 'epoch': 4.45}

 45%|████▍     | 325/730 [28:49<35:55,  5.32s/it]
 45%|████▍     | 326/730 [28:49<36:11,  5.37s/it]
                                                 
{'debug/num_tok_total': 2113.0, 'debug/num_tok_loss': 1634.0, 'debug/num_lat_total': 2113.0, 'debug/num_lat_loss': 1634.0, 'epoch': 4.47}

 45%|████▍     | 326/730 [28:50<36:11,  5.37s/it]
                                                 
{'train/ce_loss': 2.140625, 'train/diffusion_loss': 0.4598163068294525, 'epoch': 4.47}

 45%|████▍     | 326/730 [28:50<36:11,  5.37s/it]
                                                 
{'train/learning_rate_real': 1.525060258082031e-05, 'epoch': 4.47}

 45%|████▍     | 326/730 [28:50<36:11,  5.37s/it]
                                                 
{'debug/num_tok_total': 3054.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 3054.0, 'debug/num_lat_loss': 1776.0, 'epoch': 4.47}

 45%|████▍     | 326/730 [28:51<36:11,  5.37s/it]
                                                 
{'train/ce_loss': 2.15625, 'train/diffusion_loss': 0.32549595832824707, 'epoch': 4.47}

 45%|████▍     | 326/730 [28:51<36:11,  5.37s/it]
                                                 
{'train/learning_rate_real': 1.525060258082031e-05, 'epoch': 4.47}

 45%|████▍     | 326/730 [28:51<36:11,  5.37s/it]
                                                 
{'debug/num_tok_total': 2874.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2874.0, 'debug/num_lat_loss': 1789.0, 'epoch': 4.47}

 45%|████▍     | 326/730 [28:53<36:11,  5.37s/it]
                                                 
{'train/ce_loss': 1.9609375, 'train/diffusion_loss': 0.3679076135158539, 'epoch': 4.47}

 45%|████▍     | 326/730 [28:53<36:11,  5.37s/it]
                                                 
{'train/learning_rate_real': 1.525060258082031e-05, 'epoch': 4.47}

 45%|████▍     | 326/730 [28:53<36:11,  5.37s/it]
                                                 
{'debug/num_tok_total': 2335.0, 'debug/num_tok_loss': 1636.0, 'debug/num_lat_total': 2335.0, 'debug/num_lat_loss': 1636.0, 'epoch': 4.47}

 45%|████▍     | 326/730 [28:54<36:11,  5.37s/it]
                                                 
{'train/ce_loss': 1.8828125, 'train/diffusion_loss': 0.4183434247970581, 'epoch': 4.47}

 45%|████▍     | 326/730 [28:54<36:11,  5.37s/it]
                                                 
{'train/learning_rate_real': 1.525060258082031e-05, 'epoch': 4.47}

 45%|████▍     | 326/730 [28:54<36:11,  5.37s/it]
 45%|████▍     | 327/730 [28:54<36:16,  5.40s/it]
                                                 
{'debug/num_tok_total': 2450.0, 'debug/num_tok_loss': 1696.0, 'debug/num_lat_total': 2450.0, 'debug/num_lat_loss': 1696.0, 'epoch': 4.48}

 45%|████▍     | 327/730 [28:56<36:16,  5.40s/it]
                                                 
{'train/ce_loss': 2.09375, 'train/diffusion_loss': 0.4099813997745514, 'epoch': 4.48}

 45%|████▍     | 327/730 [28:56<36:16,  5.40s/it]
                                                 
{'train/learning_rate_real': 1.5196469234028144e-05, 'epoch': 4.48}

 45%|████▍     | 327/730 [28:56<36:16,  5.40s/it]
                                                 
{'debug/num_tok_total': 2427.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2427.0, 'debug/num_lat_loss': 1786.0, 'epoch': 4.48}

 45%|████▍     | 327/730 [28:57<36:16,  5.40s/it]
                                                 
{'train/ce_loss': 2.09375, 'train/diffusion_loss': 0.43626102805137634, 'epoch': 4.48}

 45%|████▍     | 327/730 [28:57<36:16,  5.40s/it]
                                                 
{'train/learning_rate_real': 1.5196469234028144e-05, 'epoch': 4.48}

 45%|████▍     | 327/730 [28:57<36:16,  5.40s/it]
                                                 
{'debug/num_tok_total': 2230.0, 'debug/num_tok_loss': 1805.0, 'debug/num_lat_total': 2230.0, 'debug/num_lat_loss': 1805.0, 'epoch': 4.48}

 45%|████▍     | 327/730 [28:58<36:16,  5.40s/it]
                                                 
{'train/ce_loss': 1.9921875, 'train/diffusion_loss': 0.49635446071624756, 'epoch': 4.48}

 45%|████▍     | 327/730 [28:58<36:16,  5.40s/it]
                                                 
{'train/learning_rate_real': 1.5196469234028144e-05, 'epoch': 4.48}

 45%|████▍     | 327/730 [28:58<36:16,  5.40s/it]
                                                 
{'debug/num_tok_total': 2236.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2236.0, 'debug/num_lat_loss': 1799.0, 'epoch': 4.48}

 45%|████▍     | 327/730 [28:59<36:16,  5.40s/it]
                                                 
{'train/ce_loss': 1.9375, 'train/diffusion_loss': 0.49344027042388916, 'epoch': 4.48}

 45%|████▍     | 327/730 [28:59<36:16,  5.40s/it]
                                                 
{'train/learning_rate_real': 1.5196469234028144e-05, 'epoch': 4.48}

 45%|████▍     | 327/730 [28:59<36:16,  5.40s/it]
 45%|████▍     | 328/730 [29:00<35:38,  5.32s/it]
                                                 
{'debug/num_tok_total': 2401.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2401.0, 'debug/num_lat_loss': 1763.0, 'epoch': 4.49}

 45%|████▍     | 328/730 [29:01<35:38,  5.32s/it]
                                                 
{'train/ce_loss': 2.015625, 'train/diffusion_loss': 0.4326101243495941, 'epoch': 4.49}

 45%|████▍     | 328/730 [29:01<35:38,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.5142282795370305e-05, 'epoch': 4.49}

 45%|████▍     | 328/730 [29:01<35:38,  5.32s/it]
                                                 
{'debug/num_tok_total': 2400.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 2400.0, 'debug/num_lat_loss': 1767.0, 'epoch': 4.49}

 45%|████▍     | 328/730 [29:02<35:38,  5.32s/it]
                                                 
{'train/ce_loss': 1.9140625, 'train/diffusion_loss': 0.409049391746521, 'epoch': 4.49}

 45%|████▍     | 328/730 [29:02<35:38,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.5142282795370305e-05, 'epoch': 4.49}

 45%|████▍     | 328/730 [29:02<35:38,  5.32s/it]
                                                 
{'debug/num_tok_total': 2125.0, 'debug/num_tok_loss': 1418.0, 'debug/num_lat_total': 2125.0, 'debug/num_lat_loss': 1418.0, 'epoch': 4.49}

 45%|████▍     | 328/730 [29:03<35:38,  5.32s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.36707603931427, 'epoch': 4.49}

 45%|████▍     | 328/730 [29:03<35:38,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.5142282795370305e-05, 'epoch': 4.49}

 45%|████▍     | 328/730 [29:03<35:38,  5.32s/it]
                                                 
{'debug/num_tok_total': 2403.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2403.0, 'debug/num_lat_loss': 1766.0, 'epoch': 4.49}

 45%|████▍     | 328/730 [29:04<35:38,  5.32s/it]
                                                 
{'train/ce_loss': 1.90625, 'train/diffusion_loss': 0.43733519315719604, 'epoch': 4.49}

 45%|████▍     | 328/730 [29:04<35:38,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.5142282795370305e-05, 'epoch': 4.49}

 45%|████▍     | 328/730 [29:04<35:38,  5.32s/it]
 45%|████▌     | 329/730 [29:05<34:52,  5.22s/it]
                                                 
{'debug/num_tok_total': 2350.0, 'debug/num_tok_loss': 1629.0, 'debug/num_lat_total': 2350.0, 'debug/num_lat_loss': 1629.0, 'epoch': 4.51}

 45%|████▌     | 329/730 [29:06<34:52,  5.22s/it]
                                                 
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.41867128014564514, 'epoch': 4.51}

 45%|████▌     | 329/730 [29:06<34:52,  5.22s/it]
                                                 
{'train/learning_rate_real': 1.5088044331745352e-05, 'epoch': 4.51}

 45%|████▌     | 329/730 [29:06<34:52,  5.22s/it]
                                                 
{'debug/num_tok_total': 3094.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 3094.0, 'debug/num_lat_loss': 1789.0, 'epoch': 4.51}

 45%|████▌     | 329/730 [29:07<34:52,  5.22s/it]
                                                 
{'train/ce_loss': 2.046875, 'train/diffusion_loss': 0.3149670660495758, 'epoch': 4.51}

 45%|████▌     | 329/730 [29:07<34:52,  5.22s/it]
                                                 
{'train/learning_rate_real': 1.5088044331745352e-05, 'epoch': 4.51}

 45%|████▌     | 329/730 [29:07<34:52,  5.22s/it]
                                                 
{'debug/num_tok_total': 2768.0, 'debug/num_tok_loss': 1673.0, 'debug/num_lat_total': 2768.0, 'debug/num_lat_loss': 1673.0, 'epoch': 4.51}

 45%|████▌     | 329/730 [29:08<34:52,  5.22s/it]
                                                 
{'train/ce_loss': 1.96875, 'train/diffusion_loss': 0.31741902232170105, 'epoch': 4.51}

 45%|████▌     | 329/730 [29:08<34:52,  5.22s/it]
                                                 
{'train/learning_rate_real': 1.5088044331745352e-05, 'epoch': 4.51}

 45%|████▌     | 329/730 [29:08<34:52,  5.22s/it]
                                                 
{'debug/num_tok_total': 2689.0, 'debug/num_tok_loss': 1810.0, 'debug/num_lat_total': 2689.0, 'debug/num_lat_loss': 1810.0, 'epoch': 4.51}

 45%|████▌     | 329/730 [29:10<34:52,  5.22s/it]
                                                 
{'train/ce_loss': 2.015625, 'train/diffusion_loss': 0.3984423577785492, 'epoch': 4.51}

 45%|████▌     | 329/730 [29:10<34:52,  5.22s/it]
                                                 
{'train/learning_rate_real': 1.5088044331745352e-05, 'epoch': 4.51}

 45%|████▌     | 329/730 [29:10<34:52,  5.22s/it]03/16/2026 07:16:42 - INFO - __main__ - LoRA debug step 330: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 45%|████▌     | 330/730 [29:10<35:26,  5.32s/it]
                                                 
{'loss': 2.5261, 'grad_norm': 1.4394961595535278, 'learning_rate': 1.5088044331745352e-05, 'epoch': 4.52}

 45%|████▌     | 330/730 [29:10<35:26,  5.32s/it]
                                                 
{'debug/num_tok_total': 1997.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 1997.0, 'debug/num_lat_loss': 1776.0, 'epoch': 4.52}

 45%|████▌     | 330/730 [29:11<35:26,  5.32s/it]
                                                 
{'train/ce_loss': 2.203125, 'train/diffusion_loss': 0.5411708950996399, 'epoch': 4.52}

 45%|████▌     | 330/730 [29:11<35:26,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.503375491107617e-05, 'epoch': 4.52}

 45%|████▌     | 330/730 [29:11<35:26,  5.32s/it]
                                                 
{'debug/num_tok_total': 2628.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2628.0, 'debug/num_lat_loss': 1780.0, 'epoch': 4.52}

 45%|████▌     | 330/730 [29:12<35:26,  5.32s/it]
                                                 
{'train/ce_loss': 1.921875, 'train/diffusion_loss': 0.42081862688064575, 'epoch': 4.52}

 45%|████▌     | 330/730 [29:12<35:26,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.503375491107617e-05, 'epoch': 4.52}

 45%|████▌     | 330/730 [29:12<35:26,  5.32s/it]
                                                 
{'debug/num_tok_total': 2432.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2432.0, 'debug/num_lat_loss': 1791.0, 'epoch': 4.52}

 45%|████▌     | 330/730 [29:13<35:26,  5.32s/it]
                                                 
{'train/ce_loss': 1.9609375, 'train/diffusion_loss': 0.43405744433403015, 'epoch': 4.52}

 45%|████▌     | 330/730 [29:13<35:26,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.503375491107617e-05, 'epoch': 4.52}

 45%|████▌     | 330/730 [29:13<35:26,  5.32s/it]
                                                 
{'debug/num_tok_total': 2633.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2633.0, 'debug/num_lat_loss': 1773.0, 'epoch': 4.52}

 45%|████▌     | 330/730 [29:15<35:26,  5.32s/it]
                                                 
{'train/ce_loss': 1.96875, 'train/diffusion_loss': 0.38711363077163696, 'epoch': 4.52}

 45%|████▌     | 330/730 [29:15<35:26,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.503375491107617e-05, 'epoch': 4.52}

 45%|████▌     | 330/730 [29:15<35:26,  5.32s/it]
 45%|████▌     | 331/730 [29:15<34:52,  5.24s/it]
                                                 
{'debug/num_tok_total': 2209.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2209.0, 'debug/num_lat_loss': 1776.0, 'epoch': 4.53}

 45%|████▌     | 331/730 [29:16<34:52,  5.24s/it]
                                                 
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.4632979929447174, 'epoch': 4.53}

 45%|████▌     | 331/730 [29:16<34:52,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.4979415602288974e-05, 'epoch': 4.53}

 45%|████▌     | 331/730 [29:16<34:52,  5.24s/it]
                                                 
{'debug/num_tok_total': 2621.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2621.0, 'debug/num_lat_loss': 1774.0, 'epoch': 4.53}

 45%|████▌     | 331/730 [29:17<34:52,  5.24s/it]
                                                 
{'train/ce_loss': 1.8828125, 'train/diffusion_loss': 0.41623392701148987, 'epoch': 4.53}

 45%|████▌     | 331/730 [29:17<34:52,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.4979415602288974e-05, 'epoch': 4.53}

 45%|████▌     | 331/730 [29:17<34:52,  5.24s/it]
                                                 
{'debug/num_tok_total': 2037.0, 'debug/num_tok_loss': 1601.0, 'debug/num_lat_total': 2037.0, 'debug/num_lat_loss': 1601.0, 'epoch': 4.53}

 45%|████▌     | 331/730 [29:19<34:52,  5.24s/it]
                                                 
{'train/ce_loss': 1.9453125, 'train/diffusion_loss': 0.4638037085533142, 'epoch': 4.53}

 45%|████▌     | 331/730 [29:19<34:52,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.4979415602288974e-05, 'epoch': 4.53}

 45%|████▌     | 331/730 [29:19<34:52,  5.24s/it]
                                                 
{'debug/num_tok_total': 2670.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2670.0, 'debug/num_lat_loss': 1795.0, 'epoch': 4.53}

 45%|████▌     | 331/730 [29:20<34:52,  5.24s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.3933698236942291, 'epoch': 4.53}

 45%|████▌     | 331/730 [29:20<34:52,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.4979415602288974e-05, 'epoch': 4.53}

 45%|████▌     | 331/730 [29:20<34:52,  5.24s/it]
 45%|████▌     | 332/730 [29:20<34:27,  5.19s/it]
                                                 
{'debug/num_tok_total': 2444.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2444.0, 'debug/num_lat_loss': 1785.0, 'epoch': 4.55}

 45%|████▌     | 332/730 [29:21<34:27,  5.19s/it]
                                                 
{'train/ce_loss': 2.109375, 'train/diffusion_loss': 0.43886104226112366, 'epoch': 4.55}

 45%|████▌     | 332/730 [29:21<34:27,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.492502747529223e-05, 'epoch': 4.55}

 45%|████▌     | 332/730 [29:21<34:27,  5.19s/it]
                                                 
{'debug/num_tok_total': 2247.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2247.0, 'debug/num_lat_loss': 1802.0, 'epoch': 4.55}

 45%|████▌     | 332/730 [29:22<34:27,  5.19s/it]
                                                 
{'train/ce_loss': 2.375, 'train/diffusion_loss': 0.48674336075782776, 'epoch': 4.55}

 45%|████▌     | 332/730 [29:22<34:27,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.492502747529223e-05, 'epoch': 4.55}

 45%|████▌     | 332/730 [29:22<34:27,  5.19s/it]
                                                 
{'debug/num_tok_total': 2866.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2866.0, 'debug/num_lat_loss': 1796.0, 'epoch': 4.55}

 45%|████▌     | 332/730 [29:24<34:27,  5.19s/it]
                                                 
{'train/ce_loss': 2.015625, 'train/diffusion_loss': 0.37624648213386536, 'epoch': 4.55}

 45%|████▌     | 332/730 [29:24<34:27,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.492502747529223e-05, 'epoch': 4.55}

 45%|████▌     | 332/730 [29:24<34:27,  5.19s/it]
                                                 
{'debug/num_tok_total': 2828.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2828.0, 'debug/num_lat_loss': 1775.0, 'epoch': 4.55}

 45%|████▌     | 332/730 [29:25<34:27,  5.19s/it]
                                                 
{'train/ce_loss': 2.046875, 'train/diffusion_loss': 0.37082526087760925, 'epoch': 4.55}

 45%|████▌     | 332/730 [29:25<34:27,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.492502747529223e-05, 'epoch': 4.55}

 45%|████▌     | 332/730 [29:25<34:27,  5.19s/it]
 46%|████▌     | 333/730 [29:25<34:27,  5.21s/it]
                                                 
{'debug/num_tok_total': 2624.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2624.0, 'debug/num_lat_loss': 1770.0, 'epoch': 4.56}

 46%|████▌     | 333/730 [29:27<34:27,  5.21s/it]
                                                 
{'train/ce_loss': 2.125, 'train/diffusion_loss': 0.3982437252998352, 'epoch': 4.56}

 46%|████▌     | 333/730 [29:27<34:27,  5.21s/it]
                                                 
{'train/learning_rate_real': 1.4870591600955622e-05, 'epoch': 4.56}

 46%|████▌     | 333/730 [29:27<34:27,  5.21s/it]
                                                 
{'debug/num_tok_total': 2298.0, 'debug/num_tok_loss': 1648.0, 'debug/num_lat_total': 2298.0, 'debug/num_lat_loss': 1648.0, 'epoch': 4.56}

 46%|████▌     | 333/730 [29:28<34:27,  5.21s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.44113418459892273, 'epoch': 4.56}

 46%|████▌     | 333/730 [29:28<34:27,  5.21s/it]
                                                 
{'train/learning_rate_real': 1.4870591600955622e-05, 'epoch': 4.56}

 46%|████▌     | 333/730 [29:28<34:27,  5.21s/it]
                                                 
{'debug/num_tok_total': 2641.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2641.0, 'debug/num_lat_loss': 1781.0, 'epoch': 4.56}

 46%|████▌     | 333/730 [29:29<34:27,  5.21s/it]
                                                 
{'train/ce_loss': 2.046875, 'train/diffusion_loss': 0.38748985528945923, 'epoch': 4.56}

 46%|████▌     | 333/730 [29:29<34:27,  5.21s/it]
                                                 
{'train/learning_rate_real': 1.4870591600955622e-05, 'epoch': 4.56}

 46%|████▌     | 333/730 [29:29<34:27,  5.21s/it]
                                                 
{'debug/num_tok_total': 2854.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2854.0, 'debug/num_lat_loss': 1783.0, 'epoch': 4.56}

 46%|████▌     | 333/730 [29:30<34:27,  5.21s/it]
                                                 
{'train/ce_loss': 2.1875, 'train/diffusion_loss': 0.39017578959465027, 'epoch': 4.56}

 46%|████▌     | 333/730 [29:30<34:27,  5.21s/it]
                                                 
{'train/learning_rate_real': 1.4870591600955622e-05, 'epoch': 4.56}

 46%|████▌     | 333/730 [29:30<34:27,  5.21s/it]
 46%|████▌     | 334/730 [29:31<34:24,  5.21s/it]
                                                 
{'debug/num_tok_total': 3040.0, 'debug/num_tok_loss': 1756.0, 'debug/num_lat_total': 3040.0, 'debug/num_lat_loss': 1756.0, 'epoch': 4.58}

 46%|████▌     | 334/730 [29:32<34:24,  5.21s/it]
                                                 
{'train/ce_loss': 1.9609375, 'train/diffusion_loss': 0.30143100023269653, 'epoch': 4.58}

 46%|████▌     | 334/730 [29:32<34:24,  5.21s/it]
                                                 
{'train/learning_rate_real': 1.4816109051088931e-05, 'epoch': 4.58}

 46%|████▌     | 334/730 [29:32<34:24,  5.21s/it]
                                                 
{'debug/num_tok_total': 2178.0, 'debug/num_tok_loss': 1735.0, 'debug/num_lat_total': 2178.0, 'debug/num_lat_loss': 1735.0, 'epoch': 4.58}

 46%|████▌     | 334/730 [29:33<34:24,  5.21s/it]
                                                 
{'train/ce_loss': 2.03125, 'train/diffusion_loss': 0.48089736700057983, 'epoch': 4.58}

 46%|████▌     | 334/730 [29:33<34:24,  5.21s/it]
                                                 
{'train/learning_rate_real': 1.4816109051088931e-05, 'epoch': 4.58}

 46%|████▌     | 334/730 [29:33<34:24,  5.21s/it]
                                                 
{'debug/num_tok_total': 2183.0, 'debug/num_tok_loss': 1761.0, 'debug/num_lat_total': 2183.0, 'debug/num_lat_loss': 1761.0, 'epoch': 4.58}

 46%|████▌     | 334/730 [29:34<34:24,  5.21s/it]
                                                 
{'train/ce_loss': 2.015625, 'train/diffusion_loss': 0.49683213233947754, 'epoch': 4.58}

 46%|████▌     | 334/730 [29:34<34:24,  5.21s/it]
                                                 
{'train/learning_rate_real': 1.4816109051088931e-05, 'epoch': 4.58}

 46%|████▌     | 334/730 [29:34<34:24,  5.21s/it]
                                                 
{'debug/num_tok_total': 3103.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 3103.0, 'debug/num_lat_loss': 1796.0, 'epoch': 4.58}

 46%|████▌     | 334/730 [29:36<34:24,  5.21s/it]
                                                 
{'train/ce_loss': 2.078125, 'train/diffusion_loss': 0.3006209433078766, 'epoch': 4.58}

 46%|████▌     | 334/730 [29:36<34:24,  5.21s/it]
                                                 
{'train/learning_rate_real': 1.4816109051088931e-05, 'epoch': 4.58}

 46%|████▌     | 334/730 [29:36<34:24,  5.21s/it]
 46%|████▌     | 335/730 [29:36<34:45,  5.28s/it]
                                                 
{'debug/num_tok_total': 2408.0, 'debug/num_tok_loss': 1743.0, 'debug/num_lat_total': 2408.0, 'debug/num_lat_loss': 1743.0, 'epoch': 4.59}

 46%|████▌     | 335/730 [29:37<34:45,  5.28s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.4070529639720917, 'epoch': 4.59}

 46%|████▌     | 335/730 [29:37<34:45,  5.28s/it]
                                                 
{'train/learning_rate_real': 1.4761580898420969e-05, 'epoch': 4.59}

 46%|████▌     | 335/730 [29:37<34:45,  5.28s/it]
                                                 
{'debug/num_tok_total': 2884.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2884.0, 'debug/num_lat_loss': 1794.0, 'epoch': 4.59}

 46%|████▌     | 335/730 [29:38<34:45,  5.28s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.34525975584983826, 'epoch': 4.59}

 46%|████▌     | 335/730 [29:38<34:45,  5.28s/it]
                                                 
{'train/learning_rate_real': 1.4761580898420969e-05, 'epoch': 4.59}

 46%|████▌     | 335/730 [29:38<34:45,  5.28s/it]
                                                 
{'debug/num_tok_total': 2878.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2878.0, 'debug/num_lat_loss': 1792.0, 'epoch': 4.59}

 46%|████▌     | 335/730 [29:40<34:45,  5.28s/it]
                                                 
{'train/ce_loss': 1.8828125, 'train/diffusion_loss': 0.3471052348613739, 'epoch': 4.59}

 46%|████▌     | 335/730 [29:40<34:45,  5.28s/it]
                                                 
{'train/learning_rate_real': 1.4761580898420969e-05, 'epoch': 4.59}

 46%|████▌     | 335/730 [29:40<34:45,  5.28s/it]
                                                 
{'debug/num_tok_total': 2425.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2425.0, 'debug/num_lat_loss': 1781.0, 'epoch': 4.59}

 46%|████▌     | 335/730 [29:41<34:45,  5.28s/it]
                                                 
{'train/ce_loss': 2.03125, 'train/diffusion_loss': 0.42148303985595703, 'epoch': 4.59}

 46%|████▌     | 335/730 [29:41<34:45,  5.28s/it]
                                                 
{'train/learning_rate_real': 1.4761580898420969e-05, 'epoch': 4.59}

 46%|████▌     | 335/730 [29:41<34:45,  5.28s/it]
 46%|████▌     | 336/730 [29:41<34:28,  5.25s/it]
                                                 
{'debug/num_tok_total': 2670.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2670.0, 'debug/num_lat_loss': 1793.0, 'epoch': 4.6}

 46%|████▌     | 336/730 [29:43<34:28,  5.25s/it]
                                                 
{'train/ce_loss': 1.921875, 'train/diffusion_loss': 0.39624324440956116, 'epoch': 4.6}

 46%|████▌     | 336/730 [29:43<34:28,  5.25s/it]
                                                 
{'train/learning_rate_real': 1.470700821657843e-05, 'epoch': 4.6}

 46%|████▌     | 336/730 [29:43<34:28,  5.25s/it]
                                                 
{'debug/num_tok_total': 2776.0, 'debug/num_tok_loss': 1713.0, 'debug/num_lat_total': 2776.0, 'debug/num_lat_loss': 1713.0, 'epoch': 4.6}

 46%|████▌     | 336/730 [29:44<34:28,  5.25s/it]
                                                 
{'train/ce_loss': 1.96875, 'train/diffusion_loss': 0.3550858199596405, 'epoch': 4.6}

 46%|████▌     | 336/730 [29:44<34:28,  5.25s/it]
                                                 
{'train/learning_rate_real': 1.470700821657843e-05, 'epoch': 4.6}

 46%|████▌     | 336/730 [29:44<34:28,  5.25s/it]
                                                 
{'debug/num_tok_total': 3346.0, 'debug/num_tok_loss': 1813.0, 'debug/num_lat_total': 3346.0, 'debug/num_lat_loss': 1813.0, 'epoch': 4.6}

 46%|████▌     | 336/730 [29:45<34:28,  5.25s/it]
                                                 
{'train/ce_loss': 1.9140625, 'train/diffusion_loss': 0.2674638628959656, 'epoch': 4.6}

 46%|████▌     | 336/730 [29:45<34:28,  5.25s/it]
                                                 
{'train/learning_rate_real': 1.470700821657843e-05, 'epoch': 4.6}

 46%|████▌     | 336/730 [29:45<34:28,  5.25s/it]
                                                 
{'debug/num_tok_total': 2416.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2416.0, 'debug/num_lat_loss': 1774.0, 'epoch': 4.6}

 46%|████▌     | 336/730 [29:46<34:28,  5.25s/it]
                                                 
{'train/ce_loss': 2.015625, 'train/diffusion_loss': 0.4429453909397125, 'epoch': 4.6}

 46%|████▌     | 336/730 [29:46<34:28,  5.25s/it]
                                                 
{'train/learning_rate_real': 1.470700821657843e-05, 'epoch': 4.6}

 46%|████▌     | 336/730 [29:46<34:28,  5.25s/it]
 46%|████▌     | 337/730 [29:47<35:01,  5.35s/it]
                                                 
{'debug/num_tok_total': 2649.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2649.0, 'debug/num_lat_loss': 1787.0, 'epoch': 4.62}

 46%|████▌     | 337/730 [29:48<35:01,  5.35s/it]
                                                 
{'train/ce_loss': 2.046875, 'train/diffusion_loss': 0.4100815951824188, 'epoch': 4.62}

 46%|████▌     | 337/730 [29:48<35:01,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.4652392080064766e-05, 'epoch': 4.62}

 46%|████▌     | 337/730 [29:48<35:01,  5.35s/it]
                                                 
{'debug/num_tok_total': 2258.0, 'debug/num_tok_loss': 1812.0, 'debug/num_lat_total': 2258.0, 'debug/num_lat_loss': 1812.0, 'epoch': 4.62}

 46%|████▌     | 337/730 [29:49<35:01,  5.35s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.48984119296073914, 'epoch': 4.62}

 46%|████▌     | 337/730 [29:49<35:01,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.4652392080064766e-05, 'epoch': 4.62}

 46%|████▌     | 337/730 [29:49<35:01,  5.35s/it]
                                                 
{'debug/num_tok_total': 2333.0, 'debug/num_tok_loss': 1697.0, 'debug/num_lat_total': 2333.0, 'debug/num_lat_loss': 1697.0, 'epoch': 4.62}

 46%|████▌     | 337/730 [29:50<35:01,  5.35s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.471801221370697, 'epoch': 4.62}

 46%|████▌     | 337/730 [29:50<35:01,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.4652392080064766e-05, 'epoch': 4.62}

 46%|████▌     | 337/730 [29:50<35:01,  5.35s/it]
                                                 
{'debug/num_tok_total': 2943.0, 'debug/num_tok_loss': 1653.0, 'debug/num_lat_total': 2943.0, 'debug/num_lat_loss': 1653.0, 'epoch': 4.62}

 46%|████▌     | 337/730 [29:52<35:01,  5.35s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.34953343868255615, 'epoch': 4.62}

 46%|████▌     | 337/730 [29:52<35:01,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.4652392080064766e-05, 'epoch': 4.62}

 46%|████▌     | 337/730 [29:52<35:01,  5.35s/it]
 46%|████▋     | 338/730 [29:52<34:32,  5.29s/it]
                                                 
{'debug/num_tok_total': 2744.0, 'debug/num_tok_loss': 1636.0, 'debug/num_lat_total': 2744.0, 'debug/num_lat_loss': 1636.0, 'epoch': 4.63}

 46%|████▋     | 338/730 [29:53<34:32,  5.29s/it]
                                                 
{'train/ce_loss': 1.8828125, 'train/diffusion_loss': 0.33348938822746277, 'epoch': 4.63}

 46%|████▋     | 338/730 [29:53<34:32,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.4597733564239022e-05, 'epoch': 4.63}

 46%|████▋     | 338/730 [29:53<34:32,  5.29s/it]
                                                 
{'debug/num_tok_total': 2443.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2443.0, 'debug/num_lat_loss': 1787.0, 'epoch': 4.63}

 46%|████▋     | 338/730 [29:54<34:32,  5.29s/it]
                                                 
{'train/ce_loss': 1.9609375, 'train/diffusion_loss': 0.4484327733516693, 'epoch': 4.63}

 46%|████▋     | 338/730 [29:54<34:32,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.4597733564239022e-05, 'epoch': 4.63}

 46%|████▋     | 338/730 [29:54<34:32,  5.29s/it]
                                                 
{'debug/num_tok_total': 2840.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2840.0, 'debug/num_lat_loss': 1779.0, 'epoch': 4.63}

 46%|████▋     | 338/730 [29:56<34:32,  5.29s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.37863439321517944, 'epoch': 4.63}

 46%|████▋     | 338/730 [29:56<34:32,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.4597733564239022e-05, 'epoch': 4.63}

 46%|████▋     | 338/730 [29:56<34:32,  5.29s/it]
                                                 
{'debug/num_tok_total': 3330.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 3330.0, 'debug/num_lat_loss': 1801.0, 'epoch': 4.63}

 46%|████▋     | 338/730 [29:57<34:32,  5.29s/it]
                                                 
{'train/ce_loss': 2.125, 'train/diffusion_loss': 0.26973050832748413, 'epoch': 4.63}

 46%|████▋     | 338/730 [29:57<34:32,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.4597733564239022e-05, 'epoch': 4.63}

 46%|████▋     | 338/730 [29:57<34:32,  5.29s/it]
 46%|████▋     | 339/730 [29:58<34:55,  5.36s/it]
                                                 
{'debug/num_tok_total': 2622.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 2622.0, 'debug/num_lat_loss': 1769.0, 'epoch': 4.64}

 46%|████▋     | 339/730 [29:59<34:55,  5.36s/it]
                                                 
{'train/ce_loss': 1.9609375, 'train/diffusion_loss': 0.41400426626205444, 'epoch': 4.64}

 46%|████▋     | 339/730 [29:59<34:55,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.4543033745294664e-05, 'epoch': 4.64}

 46%|████▋     | 339/730 [29:59<34:55,  5.36s/it]
                                                 
{'debug/num_tok_total': 2432.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2432.0, 'debug/num_lat_loss': 1779.0, 'epoch': 4.64}

 46%|████▋     | 339/730 [30:00<34:55,  5.36s/it]
                                                 
{'train/ce_loss': 2.03125, 'train/diffusion_loss': 0.43190518021583557, 'epoch': 4.64}

 46%|████▋     | 339/730 [30:00<34:55,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.4543033745294664e-05, 'epoch': 4.64}

 46%|████▋     | 339/730 [30:00<34:55,  5.36s/it]
                                                 
{'debug/num_tok_total': 2633.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2633.0, 'debug/num_lat_loss': 1778.0, 'epoch': 4.64}

 46%|████▋     | 339/730 [30:01<34:55,  5.36s/it]
                                                 
{'train/ce_loss': 1.984375, 'train/diffusion_loss': 0.4079636335372925, 'epoch': 4.64}

 46%|████▋     | 339/730 [30:01<34:55,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.4543033745294664e-05, 'epoch': 4.64}

 46%|████▋     | 339/730 [30:01<34:55,  5.36s/it]
                                                 
{'debug/num_tok_total': 2636.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2636.0, 'debug/num_lat_loss': 1776.0, 'epoch': 4.64}

 46%|████▋     | 339/730 [30:02<34:55,  5.36s/it]
                                                 
{'train/ce_loss': 1.9140625, 'train/diffusion_loss': 0.3774188756942749, 'epoch': 4.64}

 46%|████▋     | 339/730 [30:02<34:55,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.4543033745294664e-05, 'epoch': 4.64}

 46%|████▋     | 339/730 [30:02<34:55,  5.36s/it]03/16/2026 07:17:35 - INFO - __main__ - LoRA debug step 340: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 47%|████▋     | 340/730 [30:03<34:49,  5.36s/it]
                                                 
{'loss': 2.5798, 'grad_norm': 1.4297795295715332, 'learning_rate': 1.4543033745294664e-05, 'epoch': 4.66}

 47%|████▋     | 340/730 [30:03<34:49,  5.36s/it]
                                                 
{'debug/num_tok_total': 2879.0, 'debug/num_tok_loss': 1798.0, 'debug/num_lat_total': 2879.0, 'debug/num_lat_loss': 1798.0, 'epoch': 4.66}

 47%|████▋     | 340/730 [30:04<34:49,  5.36s/it]
                                                 
{'train/ce_loss': 1.921875, 'train/diffusion_loss': 0.35169917345046997, 'epoch': 4.66}

 47%|████▋     | 340/730 [30:04<34:49,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.4488293700238401e-05, 'epoch': 4.66}

 47%|████▋     | 340/730 [30:04<34:49,  5.36s/it]
                                                 
{'debug/num_tok_total': 2222.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2222.0, 'debug/num_lat_loss': 1787.0, 'epoch': 4.66}

 47%|████▋     | 340/730 [30:05<34:49,  5.36s/it]
                                                 
{'train/ce_loss': 1.8515625, 'train/diffusion_loss': 0.5089139938354492, 'epoch': 4.66}

 47%|████▋     | 340/730 [30:05<34:49,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.4488293700238401e-05, 'epoch': 4.66}

 47%|████▋     | 340/730 [30:05<34:49,  5.36s/it]
                                                 
{'debug/num_tok_total': 2455.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2455.0, 'debug/num_lat_loss': 1794.0, 'epoch': 4.66}

 47%|████▋     | 340/730 [30:06<34:49,  5.36s/it]
                                                 
{'train/ce_loss': 2.046875, 'train/diffusion_loss': 0.47563648223876953, 'epoch': 4.66}

 47%|████▋     | 340/730 [30:06<34:49,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.4488293700238401e-05, 'epoch': 4.66}

 47%|████▋     | 340/730 [30:06<34:49,  5.36s/it]
                                                 
{'debug/num_tok_total': 2266.0, 'debug/num_tok_loss': 1598.0, 'debug/num_lat_total': 2266.0, 'debug/num_lat_loss': 1598.0, 'epoch': 4.66}

 47%|████▋     | 340/730 [30:08<34:49,  5.36s/it]
                                                 
{'train/ce_loss': 2.03125, 'train/diffusion_loss': 0.4347985088825226, 'epoch': 4.66}

 47%|████▋     | 340/730 [30:08<34:49,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.4488293700238401e-05, 'epoch': 4.66}

 47%|████▋     | 340/730 [30:08<34:49,  5.36s/it]
 47%|████▋     | 341/730 [30:08<34:39,  5.35s/it]
                                                 
{'debug/num_tok_total': 2809.0, 'debug/num_tok_loss': 1654.0, 'debug/num_lat_total': 2809.0, 'debug/num_lat_loss': 1654.0, 'epoch': 4.67}

 47%|████▋     | 341/730 [30:09<34:39,  5.35s/it]
                                                 
{'train/ce_loss': 1.9375, 'train/diffusion_loss': 0.32905861735343933, 'epoch': 4.67}

 47%|████▋     | 341/730 [30:09<34:39,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.443351450686896e-05, 'epoch': 4.67}

 47%|████▋     | 341/730 [30:09<34:39,  5.35s/it]
                                                 
{'debug/num_tok_total': 2345.0, 'debug/num_tok_loss': 1739.0, 'debug/num_lat_total': 2345.0, 'debug/num_lat_loss': 1739.0, 'epoch': 4.67}

 47%|████▋     | 341/730 [30:11<34:39,  5.35s/it]
                                                 
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.4307333827018738, 'epoch': 4.67}

 47%|████▋     | 341/730 [30:11<34:39,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.443351450686896e-05, 'epoch': 4.67}

 47%|████▋     | 341/730 [30:11<34:39,  5.35s/it]
                                                 
{'debug/num_tok_total': 2417.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2417.0, 'debug/num_lat_loss': 1771.0, 'epoch': 4.67}

 47%|████▋     | 341/730 [30:12<34:39,  5.35s/it]
                                                 
{'train/ce_loss': 2.109375, 'train/diffusion_loss': 0.44704023003578186, 'epoch': 4.67}

 47%|████▋     | 341/730 [30:12<34:39,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.443351450686896e-05, 'epoch': 4.67}

 47%|████▋     | 341/730 [30:12<34:39,  5.35s/it]
                                                 
{'debug/num_tok_total': 2646.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2646.0, 'debug/num_lat_loss': 1789.0, 'epoch': 4.67}

 47%|████▋     | 341/730 [30:13<34:39,  5.35s/it]
                                                 
{'train/ce_loss': 2.203125, 'train/diffusion_loss': 0.4116402566432953, 'epoch': 4.67}

 47%|████▋     | 341/730 [30:13<34:39,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.443351450686896e-05, 'epoch': 4.67}

 47%|████▋     | 341/730 [30:13<34:39,  5.35s/it]
 47%|████▋     | 342/730 [30:13<34:24,  5.32s/it]
                                                 
{'debug/num_tok_total': 2217.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2217.0, 'debug/num_lat_loss': 1792.0, 'epoch': 4.68}

 47%|████▋     | 342/730 [30:15<34:24,  5.32s/it]
                                                 
{'train/ce_loss': 1.921875, 'train/diffusion_loss': 0.47604432702064514, 'epoch': 4.68}

 47%|████▋     | 342/730 [30:15<34:24,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.437869724375588e-05, 'epoch': 4.68}

 47%|████▋     | 342/730 [30:15<34:24,  5.32s/it]
                                                 
{'debug/num_tok_total': 2373.0, 'debug/num_tok_loss': 1720.0, 'debug/num_lat_total': 2373.0, 'debug/num_lat_loss': 1720.0, 'epoch': 4.68}

 47%|████▋     | 342/730 [30:16<34:24,  5.32s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.46728429198265076, 'epoch': 4.68}

 47%|████▋     | 342/730 [30:16<34:24,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.437869724375588e-05, 'epoch': 4.68}

 47%|████▋     | 342/730 [30:16<34:24,  5.32s/it]
                                                 
{'debug/num_tok_total': 3043.0, 'debug/num_tok_loss': 1759.0, 'debug/num_lat_total': 3043.0, 'debug/num_lat_loss': 1759.0, 'epoch': 4.68}

 47%|████▋     | 342/730 [30:17<34:24,  5.32s/it]
                                                 
{'train/ce_loss': 1.875, 'train/diffusion_loss': 0.31619134545326233, 'epoch': 4.68}

 47%|████▋     | 342/730 [30:17<34:24,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.437869724375588e-05, 'epoch': 4.68}

 47%|████▋     | 342/730 [30:17<34:24,  5.32s/it]
                                                 
{'debug/num_tok_total': 3012.0, 'debug/num_tok_loss': 1749.0, 'debug/num_lat_total': 3012.0, 'debug/num_lat_loss': 1749.0, 'epoch': 4.68}

 47%|████▋     | 342/730 [30:18<34:24,  5.32s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.31996843218803406, 'epoch': 4.68}

 47%|████▋     | 342/730 [30:18<34:24,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.437869724375588e-05, 'epoch': 4.68}

 47%|████▋     | 342/730 [30:18<34:24,  5.32s/it]
 47%|████▋     | 343/730 [30:19<34:31,  5.35s/it]
                                                 
{'debug/num_tok_total': 2671.0, 'debug/num_tok_loss': 1602.0, 'debug/num_lat_total': 2671.0, 'debug/num_lat_loss': 1602.0, 'epoch': 4.7}

 47%|████▋     | 343/730 [30:20<34:31,  5.35s/it]
                                                 
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.3488011360168457, 'epoch': 4.7}

 47%|████▋     | 343/730 [30:20<34:31,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.4323842990218273e-05, 'epoch': 4.7}

 47%|████▋     | 343/730 [30:20<34:31,  5.35s/it]
                                                 
{'debug/num_tok_total': 2028.0, 'debug/num_tok_loss': 1805.0, 'debug/num_lat_total': 2028.0, 'debug/num_lat_loss': 1805.0, 'epoch': 4.7}

 47%|████▋     | 343/730 [30:21<34:31,  5.35s/it]
                                                 
{'train/ce_loss': 1.9609375, 'train/diffusion_loss': 0.5258029699325562, 'epoch': 4.7}

 47%|████▋     | 343/730 [30:21<34:31,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.4323842990218273e-05, 'epoch': 4.7}

 47%|████▋     | 343/730 [30:21<34:31,  5.35s/it]
                                                 
{'debug/num_tok_total': 2204.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2204.0, 'debug/num_lat_loss': 1772.0, 'epoch': 4.7}

 47%|████▋     | 343/730 [30:22<34:31,  5.35s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.48519283533096313, 'epoch': 4.7}

 47%|████▋     | 343/730 [30:22<34:31,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.4323842990218273e-05, 'epoch': 4.7}

 47%|████▋     | 343/730 [30:22<34:31,  5.35s/it]
                                                 
{'debug/num_tok_total': 3328.0, 'debug/num_tok_loss': 1805.0, 'debug/num_lat_total': 3328.0, 'debug/num_lat_loss': 1805.0, 'epoch': 4.7}

 47%|████▋     | 343/730 [30:24<34:31,  5.35s/it]
                                                 
{'train/ce_loss': 1.953125, 'train/diffusion_loss': 0.2600420117378235, 'epoch': 4.7}

 47%|████▋     | 343/730 [30:24<34:31,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.4323842990218273e-05, 'epoch': 4.7}

 47%|████▋     | 343/730 [30:24<34:31,  5.35s/it]
 47%|████▋     | 344/730 [30:24<33:57,  5.28s/it]
                                                 
{'debug/num_tok_total': 2705.0, 'debug/num_tok_loss': 1620.0, 'debug/num_lat_total': 2705.0, 'debug/num_lat_loss': 1620.0, 'epoch': 4.71}

 47%|████▋     | 344/730 [30:25<33:57,  5.28s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.3364776372909546, 'epoch': 4.71}

 47%|████▋     | 344/730 [30:25<33:57,  5.28s/it]
                                                 
{'train/learning_rate_real': 1.4268952826303561e-05, 'epoch': 4.71}

 47%|████▋     | 344/730 [30:25<33:57,  5.28s/it]
                                                 
{'debug/num_tok_total': 2331.0, 'debug/num_tok_loss': 1693.0, 'debug/num_lat_total': 2331.0, 'debug/num_lat_loss': 1693.0, 'epoch': 4.71}

 47%|████▋     | 344/730 [30:26<33:57,  5.28s/it]
                                                 
{'train/ce_loss': 1.8984375, 'train/diffusion_loss': 0.4540991187095642, 'epoch': 4.71}

 47%|████▋     | 344/730 [30:26<33:57,  5.28s/it]
                                                 
{'train/learning_rate_real': 1.4268952826303561e-05, 'epoch': 4.71}

 47%|████▋     | 344/730 [30:26<33:57,  5.28s/it]
                                                 
{'debug/num_tok_total': 3066.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 3066.0, 'debug/num_lat_loss': 1778.0, 'epoch': 4.71}

 47%|████▋     | 344/730 [30:28<33:57,  5.28s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.31021925806999207, 'epoch': 4.71}

 47%|████▋     | 344/730 [30:28<33:57,  5.28s/it]
                                                 
{'train/learning_rate_real': 1.4268952826303561e-05, 'epoch': 4.71}

 47%|████▋     | 344/730 [30:28<33:57,  5.28s/it]
                                                 
{'debug/num_tok_total': 2033.0, 'debug/num_tok_loss': 1809.0, 'debug/num_lat_total': 2033.0, 'debug/num_lat_loss': 1809.0, 'epoch': 4.71}

 47%|████▋     | 344/730 [30:29<33:57,  5.28s/it]
                                                 
{'train/ce_loss': 1.9921875, 'train/diffusion_loss': 0.5523440837860107, 'epoch': 4.71}

 47%|████▋     | 344/730 [30:29<33:57,  5.28s/it]
                                                 
{'train/learning_rate_real': 1.4268952826303561e-05, 'epoch': 4.71}

 47%|████▋     | 344/730 [30:29<33:57,  5.28s/it]
 47%|████▋     | 345/730 [30:29<34:06,  5.31s/it]
                                                 
{'debug/num_tok_total': 2638.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2638.0, 'debug/num_lat_loss': 1791.0, 'epoch': 4.73}

 47%|████▋     | 345/730 [30:31<34:06,  5.31s/it]
                                                 
{'train/ce_loss': 1.875, 'train/diffusion_loss': 0.41924095153808594, 'epoch': 4.73}

 47%|████▋     | 345/730 [30:31<34:06,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.4214027832766228e-05, 'epoch': 4.73}

 47%|████▋     | 345/730 [30:31<34:06,  5.31s/it]
                                                 
{'debug/num_tok_total': 2437.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2437.0, 'debug/num_lat_loss': 1789.0, 'epoch': 4.73}

 47%|████▋     | 345/730 [30:32<34:06,  5.31s/it]
                                                 
{'train/ce_loss': 2.03125, 'train/diffusion_loss': 0.4393394887447357, 'epoch': 4.73}

 47%|████▋     | 345/730 [30:32<34:06,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.4214027832766228e-05, 'epoch': 4.73}

 47%|████▋     | 345/730 [30:32<34:06,  5.31s/it]
                                                 
{'debug/num_tok_total': 2427.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2427.0, 'debug/num_lat_loss': 1773.0, 'epoch': 4.73}

 47%|████▋     | 345/730 [30:33<34:06,  5.31s/it]
                                                 
{'train/ce_loss': 2.109375, 'train/diffusion_loss': 0.42627352476119995, 'epoch': 4.73}

 47%|████▋     | 345/730 [30:33<34:06,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.4214027832766228e-05, 'epoch': 4.73}

 47%|████▋     | 345/730 [30:33<34:06,  5.31s/it]
                                                 
{'debug/num_tok_total': 2412.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 2412.0, 'debug/num_lat_loss': 1768.0, 'epoch': 4.73}

 47%|████▋     | 345/730 [30:34<34:06,  5.31s/it]
                                                 
{'train/ce_loss': 1.875, 'train/diffusion_loss': 0.43422767519950867, 'epoch': 4.73}

 47%|████▋     | 345/730 [30:34<34:06,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.4214027832766228e-05, 'epoch': 4.73}

 47%|████▋     | 345/730 [30:34<34:06,  5.31s/it]
 47%|████▋     | 346/730 [30:34<33:31,  5.24s/it]
                                                 
{'debug/num_tok_total': 2828.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 2828.0, 'debug/num_lat_loss': 1765.0, 'epoch': 4.74}

 47%|████▋     | 346/730 [30:36<33:31,  5.24s/it]
                                                 
{'train/ce_loss': 1.96875, 'train/diffusion_loss': 0.3489011526107788, 'epoch': 4.74}

 47%|████▋     | 346/730 [30:36<33:31,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.4159069091046526e-05, 'epoch': 4.74}

 47%|████▋     | 346/730 [30:36<33:31,  5.24s/it]
                                                 
{'debug/num_tok_total': 2679.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2679.0, 'debug/num_lat_loss': 1802.0, 'epoch': 4.74}

 47%|████▋     | 346/730 [30:37<33:31,  5.24s/it]
                                                 
{'train/ce_loss': 2.03125, 'train/diffusion_loss': 0.4036855101585388, 'epoch': 4.74}

 47%|████▋     | 346/730 [30:37<33:31,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.4159069091046526e-05, 'epoch': 4.74}

 47%|████▋     | 346/730 [30:37<33:31,  5.24s/it]
                                                 
{'debug/num_tok_total': 2108.0, 'debug/num_tok_loss': 1677.0, 'debug/num_lat_total': 2108.0, 'debug/num_lat_loss': 1677.0, 'epoch': 4.74}

 47%|████▋     | 346/730 [30:38<33:31,  5.24s/it]
                                                 
{'train/ce_loss': 2.046875, 'train/diffusion_loss': 0.4629563093185425, 'epoch': 4.74}

 47%|████▋     | 346/730 [30:38<33:31,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.4159069091046526e-05, 'epoch': 4.74}

 47%|████▋     | 346/730 [30:38<33:31,  5.24s/it]
                                                 
{'debug/num_tok_total': 2429.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2429.0, 'debug/num_lat_loss': 1783.0, 'epoch': 4.74}

 47%|████▋     | 346/730 [30:39<33:31,  5.24s/it]
                                                 
{'train/ce_loss': 1.984375, 'train/diffusion_loss': 0.45152392983436584, 'epoch': 4.74}

 47%|████▋     | 346/730 [30:39<33:31,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.4159069091046526e-05, 'epoch': 4.74}

 47%|████▋     | 346/730 [30:39<33:31,  5.24s/it]
 48%|████▊     | 347/730 [30:40<33:09,  5.20s/it]
                                                 
{'debug/num_tok_total': 2395.0, 'debug/num_tok_loss': 1762.0, 'debug/num_lat_total': 2395.0, 'debug/num_lat_loss': 1762.0, 'epoch': 4.75}

 48%|████▊     | 347/730 [30:41<33:09,  5.20s/it]
                                                 
{'train/ce_loss': 2.015625, 'train/diffusion_loss': 0.46235424280166626, 'epoch': 4.75}

 48%|████▊     | 347/730 [30:41<33:09,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.4104077683249201e-05, 'epoch': 4.75}

 48%|████▊     | 347/730 [30:41<33:09,  5.20s/it]
                                                 
{'debug/num_tok_total': 2855.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2855.0, 'debug/num_lat_loss': 1779.0, 'epoch': 4.75}

 48%|████▊     | 347/730 [30:42<33:09,  5.20s/it]
                                                 
{'train/ce_loss': 1.9609375, 'train/diffusion_loss': 0.3713798224925995, 'epoch': 4.75}

 48%|████▊     | 347/730 [30:42<33:09,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.4104077683249201e-05, 'epoch': 4.75}

 48%|████▊     | 347/730 [30:42<33:09,  5.20s/it]
                                                 
{'debug/num_tok_total': 2630.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2630.0, 'debug/num_lat_loss': 1773.0, 'epoch': 4.75}

 48%|████▊     | 347/730 [30:43<33:09,  5.20s/it]
                                                 
{'train/ce_loss': 2.078125, 'train/diffusion_loss': 0.38970276713371277, 'epoch': 4.75}

 48%|████▊     | 347/730 [30:43<33:09,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.4104077683249201e-05, 'epoch': 4.75}

 48%|████▊     | 347/730 [30:43<33:09,  5.20s/it]
                                                 
{'debug/num_tok_total': 2645.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2645.0, 'debug/num_lat_loss': 1778.0, 'epoch': 4.75}

 48%|████▊     | 347/730 [30:44<33:09,  5.20s/it]
                                                 
{'train/ce_loss': 2.015625, 'train/diffusion_loss': 0.4143233001232147, 'epoch': 4.75}

 48%|████▊     | 347/730 [30:44<33:09,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.4104077683249201e-05, 'epoch': 4.75}

 48%|████▊     | 347/730 [30:44<33:09,  5.20s/it]
 48%|████▊     | 348/730 [30:45<33:53,  5.32s/it]
                                                 
{'debug/num_tok_total': 2550.0, 'debug/num_tok_loss': 1698.0, 'debug/num_lat_total': 2550.0, 'debug/num_lat_loss': 1698.0, 'epoch': 4.77}

 48%|████▊     | 348/730 [30:46<33:53,  5.32s/it]
                                                 
{'train/ce_loss': 2.171875, 'train/diffusion_loss': 0.41071975231170654, 'epoch': 4.77}

 48%|████▊     | 348/730 [30:46<33:53,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.404905469212216e-05, 'epoch': 4.77}

 48%|████▊     | 348/730 [30:46<33:53,  5.32s/it]
                                                 
{'debug/num_tok_total': 2453.0, 'debug/num_tok_loss': 1807.0, 'debug/num_lat_total': 2453.0, 'debug/num_lat_loss': 1807.0, 'epoch': 4.77}

 48%|████▊     | 348/730 [30:48<33:53,  5.32s/it]
                                                 
{'train/ce_loss': 2.109375, 'train/diffusion_loss': 0.45104700326919556, 'epoch': 4.77}

 48%|████▊     | 348/730 [30:48<33:53,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.404905469212216e-05, 'epoch': 4.77}

 48%|████▊     | 348/730 [30:48<33:53,  5.32s/it]
                                                 
{'debug/num_tok_total': 3005.0, 'debug/num_tok_loss': 1746.0, 'debug/num_lat_total': 3005.0, 'debug/num_lat_loss': 1746.0, 'epoch': 4.77}

 48%|████▊     | 348/730 [30:49<33:53,  5.32s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.290730357170105, 'epoch': 4.77}

 48%|████▊     | 348/730 [30:49<33:53,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.404905469212216e-05, 'epoch': 4.77}

 48%|████▊     | 348/730 [30:49<33:53,  5.32s/it]
                                                 
{'debug/num_tok_total': 2211.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2211.0, 'debug/num_lat_loss': 1785.0, 'epoch': 4.77}

 48%|████▊     | 348/730 [30:50<33:53,  5.32s/it]
                                                 
{'train/ce_loss': 1.90625, 'train/diffusion_loss': 0.47499173879623413, 'epoch': 4.77}

 48%|████▊     | 348/730 [30:50<33:53,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.404905469212216e-05, 'epoch': 4.77}

 48%|████▊     | 348/730 [30:50<33:53,  5.32s/it]
 48%|████▊     | 349/730 [30:50<33:35,  5.29s/it]
                                                 
{'debug/num_tok_total': 2209.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2209.0, 'debug/num_lat_loss': 1773.0, 'epoch': 4.78}

 48%|████▊     | 349/730 [30:51<33:35,  5.29s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.48611530661582947, 'epoch': 4.78}

 48%|████▊     | 349/730 [30:51<33:35,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.3994001201035174e-05, 'epoch': 4.78}

 48%|████▊     | 349/730 [30:51<33:35,  5.29s/it]
                                                 
{'debug/num_tok_total': 2619.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 2619.0, 'debug/num_lat_loss': 1768.0, 'epoch': 4.78}

 48%|████▊     | 349/730 [30:53<33:35,  5.29s/it]
                                                 
{'train/ce_loss': 2.171875, 'train/diffusion_loss': 0.4095076024532318, 'epoch': 4.78}

 48%|████▊     | 349/730 [30:53<33:35,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.3994001201035174e-05, 'epoch': 4.78}

 48%|████▊     | 349/730 [30:53<33:35,  5.29s/it]
                                                 
{'debug/num_tok_total': 2396.0, 'debug/num_tok_loss': 1762.0, 'debug/num_lat_total': 2396.0, 'debug/num_lat_loss': 1762.0, 'epoch': 4.78}

 48%|████▊     | 349/730 [30:54<33:35,  5.29s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.44954341650009155, 'epoch': 4.78}

 48%|████▊     | 349/730 [30:54<33:35,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.3994001201035174e-05, 'epoch': 4.78}

 48%|████▊     | 349/730 [30:54<33:35,  5.29s/it]
                                                 
{'debug/num_tok_total': 2835.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2835.0, 'debug/num_lat_loss': 1770.0, 'epoch': 4.78}

 48%|████▊     | 349/730 [30:55<33:35,  5.29s/it]
                                                 
{'train/ce_loss': 1.9140625, 'train/diffusion_loss': 0.3251146972179413, 'epoch': 4.78}

 48%|████▊     | 349/730 [30:55<33:35,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.3994001201035174e-05, 'epoch': 4.78}

 48%|████▊     | 349/730 [30:55<33:35,  5.29s/it]03/16/2026 07:18:28 - INFO - __main__ - LoRA debug step 350: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 48%|████▊     | 350/730 [30:56<33:18,  5.26s/it]
                                                 
{'loss': 2.6358, 'grad_norm': 1.2736142873764038, 'learning_rate': 1.3994001201035174e-05, 'epoch': 4.79}

 48%|████▊     | 350/730 [30:56<33:18,  5.26s/it]
                                                 
{'debug/num_tok_total': 2237.0, 'debug/num_tok_loss': 1803.0, 'debug/num_lat_total': 2237.0, 'debug/num_lat_loss': 1803.0, 'epoch': 4.79}

 48%|████▊     | 350/730 [30:57<33:18,  5.26s/it]
                                                 
{'train/ce_loss': 1.9375, 'train/diffusion_loss': 0.4950745105743408, 'epoch': 4.79}

 48%|████▊     | 350/730 [30:57<33:18,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.3938918293958536e-05, 'epoch': 4.79}

 48%|████▊     | 350/730 [30:57<33:18,  5.26s/it]
                                                 
{'debug/num_tok_total': 2622.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 2622.0, 'debug/num_lat_loss': 1767.0, 'epoch': 4.79}

 48%|████▊     | 350/730 [30:58<33:18,  5.26s/it]
                                                 
{'train/ce_loss': 1.9609375, 'train/diffusion_loss': 0.3680788576602936, 'epoch': 4.79}

 48%|████▊     | 350/730 [30:58<33:18,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.3938918293958536e-05, 'epoch': 4.79}

 48%|████▊     | 350/730 [30:58<33:18,  5.26s/it]
                                                 
{'debug/num_tok_total': 2629.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2629.0, 'debug/num_lat_loss': 1779.0, 'epoch': 4.79}

 48%|████▊     | 350/730 [30:59<33:18,  5.26s/it]
                                                 
{'train/ce_loss': 2.15625, 'train/diffusion_loss': 0.43818479776382446, 'epoch': 4.79}

 48%|████▊     | 350/730 [30:59<33:18,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.3938918293958536e-05, 'epoch': 4.79}

 48%|████▊     | 350/730 [30:59<33:18,  5.26s/it]
                                                 
{'debug/num_tok_total': 2493.0, 'debug/num_tok_loss': 1657.0, 'debug/num_lat_total': 2493.0, 'debug/num_lat_loss': 1657.0, 'epoch': 4.79}

 48%|████▊     | 350/730 [31:00<33:18,  5.26s/it]
                                                 
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.38977447152137756, 'epoch': 4.79}

 48%|████▊     | 350/730 [31:00<33:18,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.3938918293958536e-05, 'epoch': 4.79}

 48%|████▊     | 350/730 [31:00<33:18,  5.26s/it]
 48%|████▊     | 351/730 [31:01<33:13,  5.26s/it]
                                                 
{'debug/num_tok_total': 2640.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2640.0, 'debug/num_lat_loss': 1786.0, 'epoch': 4.81}

 48%|████▊     | 351/730 [31:02<33:13,  5.26s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.386251300573349, 'epoch': 4.81}

 48%|████▊     | 351/730 [31:02<33:13,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.3883807055441733e-05, 'epoch': 4.81}

 48%|████▊     | 351/730 [31:02<33:13,  5.26s/it]
                                                 
{'debug/num_tok_total': 2422.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2422.0, 'debug/num_lat_loss': 1778.0, 'epoch': 4.81}

 48%|████▊     | 351/730 [31:03<33:13,  5.26s/it]
                                                 
{'train/ce_loss': 1.9140625, 'train/diffusion_loss': 0.4440755546092987, 'epoch': 4.81}

 48%|████▊     | 351/730 [31:03<33:13,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.3883807055441733e-05, 'epoch': 4.81}

 48%|████▊     | 351/730 [31:03<33:13,  5.26s/it]
                                                 
{'debug/num_tok_total': 2392.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2392.0, 'debug/num_lat_loss': 1764.0, 'epoch': 4.81}

 48%|████▊     | 351/730 [31:04<33:13,  5.26s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.452121376991272, 'epoch': 4.81}

 48%|████▊     | 351/730 [31:04<33:13,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.3883807055441733e-05, 'epoch': 4.81}

 48%|████▊     | 351/730 [31:04<33:13,  5.26s/it]
                                                 
{'debug/num_tok_total': 3081.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 3081.0, 'debug/num_lat_loss': 1789.0, 'epoch': 4.81}

 48%|████▊     | 351/730 [31:06<33:13,  5.26s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.3008842468261719, 'epoch': 4.81}

 48%|████▊     | 351/730 [31:06<33:13,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.3883807055441733e-05, 'epoch': 4.81}

 48%|████▊     | 351/730 [31:06<33:13,  5.26s/it]
 48%|████▊     | 352/730 [31:06<33:14,  5.28s/it]
                                                 
{'debug/num_tok_total': 2432.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2432.0, 'debug/num_lat_loss': 1784.0, 'epoch': 4.82}

 48%|████▊     | 352/730 [31:07<33:14,  5.28s/it]
                                                 
{'train/ce_loss': 1.9453125, 'train/diffusion_loss': 0.46000051498413086, 'epoch': 4.82}

 48%|████▊     | 352/730 [31:07<33:14,  5.28s/it]
                                                 
{'train/learning_rate_real': 1.3828668570592069e-05, 'epoch': 4.82}

 48%|████▊     | 352/730 [31:07<33:14,  5.28s/it]
                                                 
{'debug/num_tok_total': 3041.0, 'debug/num_tok_loss': 1762.0, 'debug/num_lat_total': 3041.0, 'debug/num_lat_loss': 1762.0, 'epoch': 4.82}

 48%|████▊     | 352/730 [31:09<33:14,  5.28s/it]
                                                 
{'train/ce_loss': 1.8515625, 'train/diffusion_loss': 0.2769721448421478, 'epoch': 4.82}

 48%|████▊     | 352/730 [31:09<33:14,  5.28s/it]
                                                 
{'train/learning_rate_real': 1.3828668570592069e-05, 'epoch': 4.82}

 48%|████▊     | 352/730 [31:09<33:14,  5.28s/it]
                                                 
{'debug/num_tok_total': 2514.0, 'debug/num_tok_loss': 1653.0, 'debug/num_lat_total': 2514.0, 'debug/num_lat_loss': 1653.0, 'epoch': 4.82}

 48%|████▊     | 352/730 [31:10<33:14,  5.28s/it]
                                                 
{'train/ce_loss': 2.203125, 'train/diffusion_loss': 0.3928878903388977, 'epoch': 4.82}

 48%|████▊     | 352/730 [31:10<33:14,  5.28s/it]
                                                 
{'train/learning_rate_real': 1.3828668570592069e-05, 'epoch': 4.82}

 48%|████▊     | 352/730 [31:10<33:14,  5.28s/it]
                                                 
{'debug/num_tok_total': 2826.0, 'debug/num_tok_loss': 1762.0, 'debug/num_lat_total': 2826.0, 'debug/num_lat_loss': 1762.0, 'epoch': 4.82}

 48%|████▊     | 352/730 [31:11<33:14,  5.28s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.34956538677215576, 'epoch': 4.82}

 48%|████▊     | 352/730 [31:11<33:14,  5.28s/it]
                                                 
{'train/learning_rate_real': 1.3828668570592069e-05, 'epoch': 4.82}

 48%|████▊     | 352/730 [31:11<33:14,  5.28s/it]
 48%|████▊     | 353/730 [31:12<33:30,  5.33s/it]
                                                 
{'debug/num_tok_total': 2455.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 2455.0, 'debug/num_lat_loss': 1797.0, 'epoch': 4.84}

 48%|████▊     | 353/730 [31:13<33:30,  5.33s/it]
                                                 
{'train/ce_loss': 1.9609375, 'train/diffusion_loss': 0.41297292709350586, 'epoch': 4.84}

 48%|████▊     | 353/730 [31:13<33:30,  5.33s/it]
                                                 
{'train/learning_rate_real': 1.3773503925053314e-05, 'epoch': 4.84}

 48%|████▊     | 353/730 [31:13<33:30,  5.33s/it]
                                                 
{'debug/num_tok_total': 2881.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2881.0, 'debug/num_lat_loss': 1795.0, 'epoch': 4.84}

 48%|████▊     | 353/730 [31:14<33:30,  5.33s/it]
                                                 
{'train/ce_loss': 1.9375, 'train/diffusion_loss': 0.3616207540035248, 'epoch': 4.84}

 48%|████▊     | 353/730 [31:14<33:30,  5.33s/it]
                                                 
{'train/learning_rate_real': 1.3773503925053314e-05, 'epoch': 4.84}

 48%|████▊     | 353/730 [31:14<33:30,  5.33s/it]
                                                 
{'debug/num_tok_total': 2675.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2675.0, 'debug/num_lat_loss': 1799.0, 'epoch': 4.84}

 48%|████▊     | 353/730 [31:15<33:30,  5.33s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.41012150049209595, 'epoch': 4.84}

 48%|████▊     | 353/730 [31:15<33:30,  5.33s/it]
                                                 
{'train/learning_rate_real': 1.3773503925053314e-05, 'epoch': 4.84}

 48%|████▊     | 353/730 [31:15<33:30,  5.33s/it]
                                                 
{'debug/num_tok_total': 2827.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2827.0, 'debug/num_lat_loss': 1771.0, 'epoch': 4.84}

 48%|████▊     | 353/730 [31:16<33:30,  5.33s/it]
                                                 
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.34873032569885254, 'epoch': 4.84}

 48%|████▊     | 353/730 [31:16<33:30,  5.33s/it]
                                                 
{'train/learning_rate_real': 1.3773503925053314e-05, 'epoch': 4.84}

 48%|████▊     | 353/730 [31:16<33:30,  5.33s/it]
 48%|████▊     | 354/730 [31:17<33:23,  5.33s/it]
                                                 
{'debug/num_tok_total': 2114.0, 'debug/num_tok_loss': 1688.0, 'debug/num_lat_total': 2114.0, 'debug/num_lat_loss': 1688.0, 'epoch': 4.85}

 48%|████▊     | 354/730 [31:18<33:23,  5.33s/it]
                                                 
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.5083751678466797, 'epoch': 4.85}

 48%|████▊     | 354/730 [31:18<33:23,  5.33s/it]
                                                 
{'train/learning_rate_real': 1.3718314204984332e-05, 'epoch': 4.85}

 48%|████▊     | 354/730 [31:18<33:23,  5.33s/it]
                                                 
{'debug/num_tok_total': 2605.0, 'debug/num_tok_loss': 1752.0, 'debug/num_lat_total': 2605.0, 'debug/num_lat_loss': 1752.0, 'epoch': 4.85}

 48%|████▊     | 354/730 [31:19<33:23,  5.33s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.38362061977386475, 'epoch': 4.85}

 48%|████▊     | 354/730 [31:19<33:23,  5.33s/it]
                                                 
{'train/learning_rate_real': 1.3718314204984332e-05, 'epoch': 4.85}

 48%|████▊     | 354/730 [31:19<33:23,  5.33s/it]
                                                 
{'debug/num_tok_total': 2418.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2418.0, 'debug/num_lat_loss': 1777.0, 'epoch': 4.85}

 48%|████▊     | 354/730 [31:20<33:23,  5.33s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.433803528547287, 'epoch': 4.85}

 48%|████▊     | 354/730 [31:20<33:23,  5.33s/it]
                                                 
{'train/learning_rate_real': 1.3718314204984332e-05, 'epoch': 4.85}

 48%|████▊     | 354/730 [31:20<33:23,  5.33s/it]
                                                 
{'debug/num_tok_total': 2193.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2193.0, 'debug/num_lat_loss': 1776.0, 'epoch': 4.85}

 48%|████▊     | 354/730 [31:21<33:23,  5.33s/it]
                                                 
{'train/ce_loss': 1.9140625, 'train/diffusion_loss': 0.4754595458507538, 'epoch': 4.85}

 48%|████▊     | 354/730 [31:21<33:23,  5.33s/it]
                                                 
{'train/learning_rate_real': 1.3718314204984332e-05, 'epoch': 4.85}

 48%|████▊     | 354/730 [31:21<33:23,  5.33s/it]
 49%|████▊     | 355/730 [31:22<32:33,  5.21s/it]
                                                 
{'debug/num_tok_total': 2642.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2642.0, 'debug/num_lat_loss': 1781.0, 'epoch': 4.86}

 49%|████▊     | 355/730 [31:23<32:33,  5.21s/it]
                                                 
{'train/ce_loss': 2.078125, 'train/diffusion_loss': 0.3864595293998718, 'epoch': 4.86}

 49%|████▊     | 355/730 [31:23<32:33,  5.21s/it]
                                                 
{'train/learning_rate_real': 1.3663100497037681e-05, 'epoch': 4.86}

 49%|████▊     | 355/730 [31:23<32:33,  5.21s/it]
                                                 
{'debug/num_tok_total': 2229.0, 'debug/num_tok_loss': 1803.0, 'debug/num_lat_total': 2229.0, 'debug/num_lat_loss': 1803.0, 'epoch': 4.86}

 49%|████▊     | 355/730 [31:24<32:33,  5.21s/it]
                                                 
{'train/ce_loss': 2.390625, 'train/diffusion_loss': 0.5030604004859924, 'epoch': 4.86}

 49%|████▊     | 355/730 [31:24<32:33,  5.21s/it]
                                                 
{'train/learning_rate_real': 1.3663100497037681e-05, 'epoch': 4.86}

 49%|████▊     | 355/730 [31:24<32:33,  5.21s/it]
                                                 
{'debug/num_tok_total': 2221.0, 'debug/num_tok_loss': 1479.0, 'debug/num_lat_total': 2221.0, 'debug/num_lat_loss': 1479.0, 'epoch': 4.86}

 49%|████▊     | 355/730 [31:25<32:33,  5.21s/it]
                                                 
{'train/ce_loss': 1.9296875, 'train/diffusion_loss': 0.3796306848526001, 'epoch': 4.86}

 49%|████▊     | 355/730 [31:25<32:33,  5.21s/it]
                                                 
{'train/learning_rate_real': 1.3663100497037681e-05, 'epoch': 4.86}

 49%|████▊     | 355/730 [31:25<32:33,  5.21s/it]
                                                 
{'debug/num_tok_total': 2426.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2426.0, 'debug/num_lat_loss': 1792.0, 'epoch': 4.86}

 49%|████▊     | 355/730 [31:27<32:33,  5.21s/it]
                                                 
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.4208463728427887, 'epoch': 4.86}

 49%|████▊     | 355/730 [31:27<32:33,  5.21s/it]
                                                 
{'train/learning_rate_real': 1.3663100497037681e-05, 'epoch': 4.86}

 49%|████▊     | 355/730 [31:27<32:33,  5.21s/it]
 49%|████▉     | 356/730 [31:27<32:14,  5.17s/it]
                                                 
{'debug/num_tok_total': 2163.0, 'debug/num_tok_loss': 1742.0, 'debug/num_lat_total': 2163.0, 'debug/num_lat_loss': 1742.0, 'epoch': 4.88}

 49%|████▉     | 356/730 [31:28<32:14,  5.17s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.4776431918144226, 'epoch': 4.88}

 49%|████▉     | 356/730 [31:28<32:14,  5.17s/it]
                                                 
{'train/learning_rate_real': 1.360786388833824e-05, 'epoch': 4.88}

 49%|████▉     | 356/730 [31:28<32:14,  5.17s/it]
                                                 
{'debug/num_tok_total': 3109.0, 'debug/num_tok_loss': 1803.0, 'debug/num_lat_total': 3109.0, 'debug/num_lat_loss': 1803.0, 'epoch': 4.88}

 49%|████▉     | 356/730 [31:29<32:14,  5.17s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.3375520706176758, 'epoch': 4.88}

 49%|████▉     | 356/730 [31:29<32:14,  5.17s/it]
                                                 
{'train/learning_rate_real': 1.360786388833824e-05, 'epoch': 4.88}

 49%|████▉     | 356/730 [31:29<32:14,  5.17s/it]
                                                 
{'debug/num_tok_total': 2639.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2639.0, 'debug/num_lat_loss': 1786.0, 'epoch': 4.88}

 49%|████▉     | 356/730 [31:30<32:14,  5.17s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.39082589745521545, 'epoch': 4.88}

 49%|████▉     | 356/730 [31:31<32:14,  5.17s/it]
                                                 
{'train/learning_rate_real': 1.360786388833824e-05, 'epoch': 4.88}

 49%|████▉     | 356/730 [31:31<32:14,  5.17s/it]
                                                 
{'debug/num_tok_total': 2877.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2877.0, 'debug/num_lat_loss': 1790.0, 'epoch': 4.88}

 49%|████▉     | 356/730 [31:32<32:14,  5.17s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.3445090353488922, 'epoch': 4.88}

 49%|████▉     | 356/730 [31:32<32:14,  5.17s/it]
                                                 
{'train/learning_rate_real': 1.360786388833824e-05, 'epoch': 4.88}

 49%|████▉     | 356/730 [31:32<32:14,  5.17s/it]
 49%|████▉     | 357/730 [31:32<32:16,  5.19s/it]
                                                 
{'debug/num_tok_total': 2858.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2858.0, 'debug/num_lat_loss': 1783.0, 'epoch': 4.89}

 49%|████▉     | 357/730 [31:33<32:16,  5.19s/it]
                                                 
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.34473174810409546, 'epoch': 4.89}

 49%|████▉     | 357/730 [31:33<32:16,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.355260546646177e-05, 'epoch': 4.89}

 49%|████▉     | 357/730 [31:33<32:16,  5.19s/it]
                                                 
{'debug/num_tok_total': 2296.0, 'debug/num_tok_loss': 1650.0, 'debug/num_lat_total': 2296.0, 'debug/num_lat_loss': 1650.0, 'epoch': 4.89}

 49%|████▉     | 357/730 [31:35<32:16,  5.19s/it]
                                                 
{'train/ce_loss': 1.875, 'train/diffusion_loss': 0.43163198232650757, 'epoch': 4.89}

 49%|████▉     | 357/730 [31:35<32:16,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.355260546646177e-05, 'epoch': 4.89}

 49%|████▉     | 357/730 [31:35<32:16,  5.19s/it]
                                                 
{'debug/num_tok_total': 2846.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2846.0, 'debug/num_lat_loss': 1776.0, 'epoch': 4.89}

 49%|████▉     | 357/730 [31:36<32:16,  5.19s/it]
                                                 
{'train/ce_loss': 2.046875, 'train/diffusion_loss': 0.34775587916374207, 'epoch': 4.89}

 49%|████▉     | 357/730 [31:36<32:16,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.355260546646177e-05, 'epoch': 4.89}

 49%|████▉     | 357/730 [31:36<32:16,  5.19s/it]
                                                 
{'debug/num_tok_total': 2443.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2443.0, 'debug/num_lat_loss': 1783.0, 'epoch': 4.89}

 49%|████▉     | 357/730 [31:37<32:16,  5.19s/it]
                                                 
{'train/ce_loss': 1.953125, 'train/diffusion_loss': 0.4328345060348511, 'epoch': 4.89}

 49%|████▉     | 357/730 [31:37<32:16,  5.19s/it]
                                                 
{'train/learning_rate_real': 1.355260546646177e-05, 'epoch': 4.89}

 49%|████▉     | 357/730 [31:37<32:16,  5.19s/it]
 49%|████▉     | 358/730 [31:38<32:28,  5.24s/it]
                                                 
{'debug/num_tok_total': 2802.0, 'debug/num_tok_loss': 1742.0, 'debug/num_lat_total': 2802.0, 'debug/num_lat_loss': 1742.0, 'epoch': 4.9}

 49%|████▉     | 358/730 [31:39<32:28,  5.24s/it]
                                                 
{'train/ce_loss': 1.921875, 'train/diffusion_loss': 0.3326120972633362, 'epoch': 4.9}

 49%|████▉     | 358/730 [31:39<32:28,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.3497326319413539e-05, 'epoch': 4.9}

 49%|████▉     | 358/730 [31:39<32:28,  5.24s/it]
                                                 
{'debug/num_tok_total': 2687.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2687.0, 'debug/num_lat_loss': 1795.0, 'epoch': 4.9}

 49%|████▉     | 358/730 [31:40<32:28,  5.24s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.38338905572891235, 'epoch': 4.9}

 49%|████▉     | 358/730 [31:40<32:28,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.3497326319413539e-05, 'epoch': 4.9}

 49%|████▉     | 358/730 [31:40<32:28,  5.24s/it]
                                                 
{'debug/num_tok_total': 2367.0, 'debug/num_tok_loss': 1646.0, 'debug/num_lat_total': 2367.0, 'debug/num_lat_loss': 1646.0, 'epoch': 4.9}

 49%|████▉     | 358/730 [31:41<32:28,  5.24s/it]
                                                 
{'train/ce_loss': 2.046875, 'train/diffusion_loss': 0.4092291593551636, 'epoch': 4.9}

 49%|████▉     | 358/730 [31:41<32:28,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.3497326319413539e-05, 'epoch': 4.9}

 49%|████▉     | 358/730 [31:41<32:28,  5.24s/it]
                                                 
{'debug/num_tok_total': 2325.0, 'debug/num_tok_loss': 1616.0, 'debug/num_lat_total': 2325.0, 'debug/num_lat_loss': 1616.0, 'epoch': 4.9}

 49%|████▉     | 358/730 [31:43<32:28,  5.24s/it]
                                                 
{'train/ce_loss': 1.9140625, 'train/diffusion_loss': 0.4156040847301483, 'epoch': 4.9}

 49%|████▉     | 358/730 [31:43<32:28,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.3497326319413539e-05, 'epoch': 4.9}

 49%|████▉     | 358/730 [31:43<32:28,  5.24s/it]
 49%|████▉     | 359/730 [31:43<32:46,  5.30s/it]
                                                 
{'debug/num_tok_total': 2175.0, 'debug/num_tok_loss': 1758.0, 'debug/num_lat_total': 2175.0, 'debug/num_lat_loss': 1758.0, 'epoch': 4.92}

 49%|████▉     | 359/730 [31:44<32:46,  5.30s/it]
                                                 
{'train/ce_loss': 1.8828125, 'train/diffusion_loss': 0.4687121510505676, 'epoch': 4.92}

 49%|████▉     | 359/730 [31:44<32:46,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.3442027535606871e-05, 'epoch': 4.92}

 49%|████▉     | 359/730 [31:44<32:46,  5.30s/it]
                                                 
{'debug/num_tok_total': 2853.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2853.0, 'debug/num_lat_loss': 1770.0, 'epoch': 4.92}

 49%|████▉     | 359/730 [31:45<32:46,  5.30s/it]
                                                 
{'train/ce_loss': 1.8984375, 'train/diffusion_loss': 0.372743159532547, 'epoch': 4.92}

 49%|████▉     | 359/730 [31:45<32:46,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.3442027535606871e-05, 'epoch': 4.92}

 49%|████▉     | 359/730 [31:45<32:46,  5.30s/it]
                                                 
{'debug/num_tok_total': 2617.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2617.0, 'debug/num_lat_loss': 1774.0, 'epoch': 4.92}

 49%|████▉     | 359/730 [31:47<32:46,  5.30s/it]
                                                 
{'train/ce_loss': 1.984375, 'train/diffusion_loss': 0.41357186436653137, 'epoch': 4.92}

 49%|████▉     | 359/730 [31:47<32:46,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.3442027535606871e-05, 'epoch': 4.92}

 49%|████▉     | 359/730 [31:47<32:46,  5.30s/it]
                                                 
{'debug/num_tok_total': 3291.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 3291.0, 'debug/num_lat_loss': 1780.0, 'epoch': 4.92}

 49%|████▉     | 359/730 [31:48<32:46,  5.30s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.2430008500814438, 'epoch': 4.92}

 49%|████▉     | 359/730 [31:48<32:46,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.3442027535606871e-05, 'epoch': 4.92}

 49%|████▉     | 359/730 [31:48<32:46,  5.30s/it]03/16/2026 07:19:20 - INFO - __main__ - LoRA debug step 360: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 49%|████▉     | 360/730 [31:48<33:02,  5.36s/it]
                                                 
{'loss': 2.5371, 'grad_norm': 1.407039999961853, 'learning_rate': 1.3442027535606871e-05, 'epoch': 4.93}

 49%|████▉     | 360/730 [31:48<33:02,  5.36s/it]
                                                 
{'debug/num_tok_total': 2190.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 2190.0, 'debug/num_lat_loss': 1765.0, 'epoch': 4.93}

 49%|████▉     | 360/730 [31:49<33:02,  5.36s/it]
                                                 
{'train/ce_loss': 2.265625, 'train/diffusion_loss': 0.4956938624382019, 'epoch': 4.93}

 49%|████▉     | 360/730 [31:49<33:02,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.3386710203841732e-05, 'epoch': 4.93}

 49%|████▉     | 360/730 [31:49<33:02,  5.36s/it]
                                                 
{'debug/num_tok_total': 2416.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2416.0, 'debug/num_lat_loss': 1778.0, 'epoch': 4.93}

 49%|████▉     | 360/730 [31:51<33:02,  5.36s/it]
                                                 
{'train/ce_loss': 1.8828125, 'train/diffusion_loss': 0.43857550621032715, 'epoch': 4.93}

 49%|████▉     | 360/730 [31:51<33:02,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.3386710203841732e-05, 'epoch': 4.93}

 49%|████▉     | 360/730 [31:51<33:02,  5.36s/it]
                                                 
{'debug/num_tok_total': 2801.0, 'debug/num_tok_loss': 1751.0, 'debug/num_lat_total': 2801.0, 'debug/num_lat_loss': 1751.0, 'epoch': 4.93}

 49%|████▉     | 360/730 [31:52<33:02,  5.36s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.3335510492324829, 'epoch': 4.93}

 49%|████▉     | 360/730 [31:52<33:02,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.3386710203841732e-05, 'epoch': 4.93}

 49%|████▉     | 360/730 [31:52<33:02,  5.36s/it]
                                                 
{'debug/num_tok_total': 2210.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2210.0, 'debug/num_lat_loss': 1783.0, 'epoch': 4.93}

 49%|████▉     | 360/730 [31:53<33:02,  5.36s/it]
                                                 
{'train/ce_loss': 2.09375, 'train/diffusion_loss': 0.487785667181015, 'epoch': 4.93}

 49%|████▉     | 360/730 [31:53<33:02,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.3386710203841732e-05, 'epoch': 4.93}

 49%|████▉     | 360/730 [31:53<33:02,  5.36s/it]
 49%|████▉     | 361/730 [31:53<32:12,  5.24s/it]
                                                 
{'debug/num_tok_total': 2109.0, 'debug/num_tok_loss': 1626.0, 'debug/num_lat_total': 2109.0, 'debug/num_lat_loss': 1626.0, 'epoch': 4.95}

 49%|████▉     | 361/730 [31:55<32:12,  5.24s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.4661395251750946, 'epoch': 4.95}

 49%|████▉     | 361/730 [31:55<32:12,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.3331375413283281e-05, 'epoch': 4.95}

 49%|████▉     | 361/730 [31:55<32:12,  5.24s/it]
                                                 
{'debug/num_tok_total': 2848.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2848.0, 'debug/num_lat_loss': 1773.0, 'epoch': 4.95}

 49%|████▉     | 361/730 [31:56<32:12,  5.24s/it]
                                                 
{'train/ce_loss': 1.8984375, 'train/diffusion_loss': 0.33667802810668945, 'epoch': 4.95}

 49%|████▉     | 361/730 [31:56<32:12,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.3331375413283281e-05, 'epoch': 4.95}

 49%|████▉     | 361/730 [31:56<32:12,  5.24s/it]
                                                 
{'debug/num_tok_total': 1888.0, 'debug/num_tok_loss': 1622.0, 'debug/num_lat_total': 1888.0, 'debug/num_lat_loss': 1622.0, 'epoch': 4.95}

 49%|████▉     | 361/730 [31:57<32:12,  5.24s/it]
                                                 
{'train/ce_loss': 1.8515625, 'train/diffusion_loss': 0.47830551862716675, 'epoch': 4.95}

 49%|████▉     | 361/730 [31:57<32:12,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.3331375413283281e-05, 'epoch': 4.95}

 49%|████▉     | 361/730 [31:57<32:12,  5.24s/it]
                                                 
{'debug/num_tok_total': 2236.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2236.0, 'debug/num_lat_loss': 1795.0, 'epoch': 4.95}

 49%|████▉     | 361/730 [31:58<32:12,  5.24s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.48921123147010803, 'epoch': 4.95}

 49%|████▉     | 361/730 [31:58<32:12,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.3331375413283281e-05, 'epoch': 4.95}

 49%|████▉     | 361/730 [31:58<32:12,  5.24s/it]
 50%|████▉     | 362/730 [31:59<32:10,  5.25s/it]
                                                 
{'debug/num_tok_total': 2447.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2447.0, 'debug/num_lat_loss': 1788.0, 'epoch': 4.96}

 50%|████▉     | 362/730 [32:00<32:10,  5.25s/it]
                                                 
{'train/ce_loss': 1.96875, 'train/diffusion_loss': 0.43193677067756653, 'epoch': 4.96}

 50%|████▉     | 362/730 [32:00<32:10,  5.25s/it]
                                                 
{'train/learning_rate_real': 1.3276024253440425e-05, 'epoch': 4.96}

 50%|████▉     | 362/730 [32:00<32:10,  5.25s/it]
                                                 
{'debug/num_tok_total': 2867.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2867.0, 'debug/num_lat_loss': 1784.0, 'epoch': 4.96}

 50%|████▉     | 362/730 [32:01<32:10,  5.25s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.3341422379016876, 'epoch': 4.96}

 50%|████▉     | 362/730 [32:01<32:10,  5.25s/it]
                                                 
{'train/learning_rate_real': 1.3276024253440425e-05, 'epoch': 4.96}

 50%|████▉     | 362/730 [32:01<32:10,  5.25s/it]
                                                 
{'debug/num_tok_total': 2427.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2427.0, 'debug/num_lat_loss': 1787.0, 'epoch': 4.96}

 50%|████▉     | 362/730 [32:02<32:10,  5.25s/it]
                                                 
{'train/ce_loss': 1.953125, 'train/diffusion_loss': 0.4473453462123871, 'epoch': 4.96}

 50%|████▉     | 362/730 [32:02<32:10,  5.25s/it]
                                                 
{'train/learning_rate_real': 1.3276024253440425e-05, 'epoch': 4.96}

 50%|████▉     | 362/730 [32:02<32:10,  5.25s/it]
                                                 
{'debug/num_tok_total': 2219.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2219.0, 'debug/num_lat_loss': 1780.0, 'epoch': 4.96}

 50%|████▉     | 362/730 [32:03<32:10,  5.25s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.4929288327693939, 'epoch': 4.96}

 50%|████▉     | 362/730 [32:03<32:10,  5.25s/it]
                                                 
{'train/learning_rate_real': 1.3276024253440425e-05, 'epoch': 4.96}

 50%|████▉     | 362/730 [32:03<32:10,  5.25s/it]
 50%|████▉     | 363/730 [32:04<31:55,  5.22s/it]
                                                 
{'debug/num_tok_total': 2866.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2866.0, 'debug/num_lat_loss': 1782.0, 'epoch': 4.97}

 50%|████▉     | 363/730 [32:05<31:55,  5.22s/it]
                                                 
{'train/ce_loss': 2.015625, 'train/diffusion_loss': 0.38495710492134094, 'epoch': 4.97}

 50%|████▉     | 363/730 [32:05<31:55,  5.22s/it]
                                                 
{'train/learning_rate_real': 1.3220657814144394e-05, 'epoch': 4.97}

 50%|████▉     | 363/730 [32:05<31:55,  5.22s/it]
                                                 
{'debug/num_tok_total': 3115.0, 'debug/num_tok_loss': 1604.0, 'debug/num_lat_total': 3115.0, 'debug/num_lat_loss': 1604.0, 'epoch': 4.97}

 50%|████▉     | 363/730 [32:06<31:55,  5.22s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.2391890585422516, 'epoch': 4.97}

 50%|████▉     | 363/730 [32:06<31:55,  5.22s/it]
                                                 
{'train/learning_rate_real': 1.3220657814144394e-05, 'epoch': 4.97}

 50%|████▉     | 363/730 [32:06<31:55,  5.22s/it]
                                                 
{'debug/num_tok_total': 3247.0, 'debug/num_tok_loss': 1759.0, 'debug/num_lat_total': 3247.0, 'debug/num_lat_loss': 1759.0, 'epoch': 4.97}

 50%|████▉     | 363/730 [32:08<31:55,  5.22s/it]
                                                 
{'train/ce_loss': 1.953125, 'train/diffusion_loss': 0.2867518663406372, 'epoch': 4.97}

 50%|████▉     | 363/730 [32:08<31:55,  5.22s/it]
                                                 
{'train/learning_rate_real': 1.3220657814144394e-05, 'epoch': 4.97}

 50%|████▉     | 363/730 [32:08<31:55,  5.22s/it]
                                                 
{'debug/num_tok_total': 2625.0, 'debug/num_tok_loss': 1759.0, 'debug/num_lat_total': 2625.0, 'debug/num_lat_loss': 1759.0, 'epoch': 4.97}

 50%|████▉     | 363/730 [32:09<31:55,  5.22s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.43952733278274536, 'epoch': 4.97}

 50%|████▉     | 363/730 [32:09<31:55,  5.22s/it]
                                                 
{'train/learning_rate_real': 1.3220657814144394e-05, 'epoch': 4.97}

 50%|████▉     | 363/730 [32:09<31:55,  5.22s/it]
 50%|████▉     | 364/730 [32:09<32:32,  5.33s/it]
                                                 
{'debug/num_tok_total': 2640.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2640.0, 'debug/num_lat_loss': 1782.0, 'epoch': 4.99}

 50%|████▉     | 364/730 [32:11<32:32,  5.33s/it]
                                                 
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.382199227809906, 'epoch': 4.99}

 50%|████▉     | 364/730 [32:11<32:32,  5.33s/it]
                                                 
{'train/learning_rate_real': 1.3165277185527234e-05, 'epoch': 4.99}

 50%|████▉     | 364/730 [32:11<32:32,  5.33s/it]
                                                 
{'debug/num_tok_total': 2251.0, 'debug/num_tok_loss': 1811.0, 'debug/num_lat_total': 2251.0, 'debug/num_lat_loss': 1811.0, 'epoch': 4.99}

 50%|████▉     | 364/730 [32:12<32:32,  5.33s/it]
                                                 
{'train/ce_loss': 1.875, 'train/diffusion_loss': 0.5001628994941711, 'epoch': 4.99}

 50%|████▉     | 364/730 [32:12<32:32,  5.33s/it]
                                                 
{'train/learning_rate_real': 1.3165277185527234e-05, 'epoch': 4.99}

 50%|████▉     | 364/730 [32:12<32:32,  5.33s/it]
                                                 
{'debug/num_tok_total': 2839.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2839.0, 'debug/num_lat_loss': 1771.0, 'epoch': 4.99}

 50%|████▉     | 364/730 [32:13<32:32,  5.33s/it]
                                                 
{'train/ce_loss': 2.140625, 'train/diffusion_loss': 0.3672467768192291, 'epoch': 4.99}

 50%|████▉     | 364/730 [32:13<32:32,  5.33s/it]
                                                 
{'train/learning_rate_real': 1.3165277185527234e-05, 'epoch': 4.99}

 50%|████▉     | 364/730 [32:13<32:32,  5.33s/it]
                                                 
{'debug/num_tok_total': 634.0, 'debug/num_tok_loss': 429.0, 'debug/num_lat_total': 634.0, 'debug/num_lat_loss': 429.0, 'epoch': 4.99}

 50%|████▉     | 364/730 [32:14<32:32,  5.33s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.3789863586425781, 'epoch': 4.99}

 50%|████▉     | 364/730 [32:14<32:32,  5.33s/it]
                                                 
{'train/learning_rate_real': 1.3165277185527234e-05, 'epoch': 4.99}

 50%|████▉     | 364/730 [32:14<32:32,  5.33s/it]
 50%|█████     | 365/730 [32:14<30:27,  5.01s/it]
                                                 
{'debug/num_tok_total': 2672.0, 'debug/num_tok_loss': 1687.0, 'debug/num_lat_total': 2672.0, 'debug/num_lat_loss': 1687.0, 'epoch': 5.0}

 50%|█████     | 365/730 [32:15<30:27,  5.01s/it]
                                                 
{'train/ce_loss': 2.03125, 'train/diffusion_loss': 0.34250688552856445, 'epoch': 5.0}

 50%|█████     | 365/730 [32:15<30:27,  5.01s/it]
                                                 
{'train/learning_rate_real': 1.3109883458000391e-05, 'epoch': 5.0}

 50%|█████     | 365/730 [32:15<30:27,  5.01s/it]
                                                 
{'debug/num_tok_total': 2026.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2026.0, 'debug/num_lat_loss': 1802.0, 'epoch': 5.0}

 50%|█████     | 365/730 [32:16<30:27,  5.01s/it]
                                                 
{'train/ce_loss': 1.9375, 'train/diffusion_loss': 0.5316159129142761, 'epoch': 5.0}

 50%|█████     | 365/730 [32:16<30:27,  5.01s/it]
                                                 
{'train/learning_rate_real': 1.3109883458000391e-05, 'epoch': 5.0}

 50%|█████     | 365/730 [32:16<30:27,  5.01s/it]
                                                 
{'debug/num_tok_total': 2221.0, 'debug/num_tok_loss': 1687.0, 'debug/num_lat_total': 2221.0, 'debug/num_lat_loss': 1687.0, 'epoch': 5.0}

 50%|█████     | 365/730 [32:17<30:27,  5.01s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.4441116750240326, 'epoch': 5.0}

 50%|█████     | 365/730 [32:17<30:27,  5.01s/it]
                                                 
{'train/learning_rate_real': 1.3109883458000391e-05, 'epoch': 5.0}

 50%|█████     | 365/730 [32:17<30:27,  5.01s/it]
                                                 
{'debug/num_tok_total': 2198.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 2198.0, 'debug/num_lat_loss': 1767.0, 'epoch': 5.0}

 50%|█████     | 365/730 [32:18<30:27,  5.01s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.5039470791816711, 'epoch': 5.0}

 50%|█████     | 365/730 [32:18<30:27,  5.01s/it]
                                                 
{'train/learning_rate_real': 1.3109883458000391e-05, 'epoch': 5.0}

 50%|█████     | 365/730 [32:18<30:27,  5.01s/it]
 50%|█████     | 366/730 [32:19<30:29,  5.03s/it]
                                                 
{'debug/num_tok_total': 2692.0, 'debug/num_tok_loss': 1817.0, 'debug/num_lat_total': 2692.0, 'debug/num_lat_loss': 1817.0, 'epoch': 5.01}

 50%|█████     | 366/730 [32:20<30:29,  5.03s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.3879682719707489, 'epoch': 5.01}

 50%|█████     | 366/730 [32:20<30:29,  5.03s/it]
                                                 
{'train/learning_rate_real': 1.3054477722233206e-05, 'epoch': 5.01}

 50%|█████     | 366/730 [32:20<30:29,  5.03s/it]
                                                 
{'debug/num_tok_total': 2440.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2440.0, 'debug/num_lat_loss': 1783.0, 'epoch': 5.01}

 50%|█████     | 366/730 [32:21<30:29,  5.03s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.42739516496658325, 'epoch': 5.01}

 50%|█████     | 366/730 [32:21<30:29,  5.03s/it]
                                                 
{'train/learning_rate_real': 1.3054477722233206e-05, 'epoch': 5.01}

 50%|█████     | 366/730 [32:21<30:29,  5.03s/it]
                                                 
{'debug/num_tok_total': 2605.0, 'debug/num_tok_loss': 1756.0, 'debug/num_lat_total': 2605.0, 'debug/num_lat_loss': 1756.0, 'epoch': 5.01}

 50%|█████     | 366/730 [32:22<30:29,  5.03s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.39000993967056274, 'epoch': 5.01}

 50%|█████     | 366/730 [32:22<30:29,  5.03s/it]
                                                 
{'train/learning_rate_real': 1.3054477722233206e-05, 'epoch': 5.01}

 50%|█████     | 366/730 [32:22<30:29,  5.03s/it]
                                                 
{'debug/num_tok_total': 2408.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 2408.0, 'debug/num_lat_loss': 1769.0, 'epoch': 5.01}

 50%|█████     | 366/730 [32:24<30:29,  5.03s/it]
                                                 
{'train/ce_loss': 1.9765625, 'train/diffusion_loss': 0.4404014050960541, 'epoch': 5.01}

 50%|█████     | 366/730 [32:24<30:29,  5.03s/it]
                                                 
{'train/learning_rate_real': 1.3054477722233206e-05, 'epoch': 5.01}

 50%|█████     | 366/730 [32:24<30:29,  5.03s/it]
 50%|█████     | 367/730 [32:24<30:51,  5.10s/it]
                                                 
{'debug/num_tok_total': 2276.0, 'debug/num_tok_loss': 1717.0, 'debug/num_lat_total': 2276.0, 'debug/num_lat_loss': 1717.0, 'epoch': 5.03}

 50%|█████     | 367/730 [32:25<30:51,  5.10s/it]
                                                 
{'train/ce_loss': 1.9921875, 'train/diffusion_loss': 0.4457962214946747, 'epoch': 5.03}

 50%|█████     | 367/730 [32:25<30:51,  5.10s/it]
                                                 
{'train/learning_rate_real': 1.299906106913147e-05, 'epoch': 5.03}

 50%|█████     | 367/730 [32:25<30:51,  5.10s/it]
                                                 
{'debug/num_tok_total': 2860.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2860.0, 'debug/num_lat_loss': 1774.0, 'epoch': 5.03}

 50%|█████     | 367/730 [32:26<30:51,  5.10s/it]
                                                 
{'train/ce_loss': 1.9296875, 'train/diffusion_loss': 0.34024664759635925, 'epoch': 5.03}

 50%|█████     | 367/730 [32:26<30:51,  5.10s/it]
                                                 
{'train/learning_rate_real': 1.299906106913147e-05, 'epoch': 5.03}

 50%|█████     | 367/730 [32:26<30:51,  5.10s/it]
                                                 
{'debug/num_tok_total': 2644.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2644.0, 'debug/num_lat_loss': 1781.0, 'epoch': 5.03}

 50%|█████     | 367/730 [32:28<30:51,  5.10s/it]
                                                 
{'train/ce_loss': 2.125, 'train/diffusion_loss': 0.408486008644104, 'epoch': 5.03}

 50%|█████     | 367/730 [32:28<30:51,  5.10s/it]
                                                 
{'train/learning_rate_real': 1.299906106913147e-05, 'epoch': 5.03}

 50%|█████     | 367/730 [32:28<30:51,  5.10s/it]
                                                 
{'debug/num_tok_total': 2405.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2405.0, 'debug/num_lat_loss': 1771.0, 'epoch': 5.03}

 50%|█████     | 367/730 [32:29<30:51,  5.10s/it]
                                                 
{'train/ce_loss': 2.078125, 'train/diffusion_loss': 0.44742128252983093, 'epoch': 5.03}

 50%|█████     | 367/730 [32:29<30:51,  5.10s/it]
                                                 
{'train/learning_rate_real': 1.299906106913147e-05, 'epoch': 5.03}

 50%|█████     | 367/730 [32:29<30:51,  5.10s/it]
 50%|█████     | 368/730 [32:29<30:55,  5.13s/it]
                                                 
{'debug/num_tok_total': 2645.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2645.0, 'debug/num_lat_loss': 1782.0, 'epoch': 5.04}

 50%|█████     | 368/730 [32:30<30:55,  5.13s/it]
                                                 
{'train/ce_loss': 1.984375, 'train/diffusion_loss': 0.3868914544582367, 'epoch': 5.04}

 50%|█████     | 368/730 [32:30<30:55,  5.13s/it]
                                                 
{'train/learning_rate_real': 1.2943634589815915e-05, 'epoch': 5.04}

 50%|█████     | 368/730 [32:30<30:55,  5.13s/it]
                                                 
{'debug/num_tok_total': 2662.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2662.0, 'debug/num_lat_loss': 1794.0, 'epoch': 5.04}

 50%|█████     | 368/730 [32:32<30:55,  5.13s/it]
                                                 
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.3968140780925751, 'epoch': 5.04}

 50%|█████     | 368/730 [32:32<30:55,  5.13s/it]
                                                 
{'train/learning_rate_real': 1.2943634589815915e-05, 'epoch': 5.04}

 50%|█████     | 368/730 [32:32<30:55,  5.13s/it]
                                                 
{'debug/num_tok_total': 2867.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2867.0, 'debug/num_lat_loss': 1788.0, 'epoch': 5.04}

 50%|█████     | 368/730 [32:33<30:55,  5.13s/it]
                                                 
{'train/ce_loss': 1.9296875, 'train/diffusion_loss': 0.3705860674381256, 'epoch': 5.04}

 50%|█████     | 368/730 [32:33<30:55,  5.13s/it]
                                                 
{'train/learning_rate_real': 1.2943634589815915e-05, 'epoch': 5.04}

 50%|█████     | 368/730 [32:33<30:55,  5.13s/it]
                                                 
{'debug/num_tok_total': 2604.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2604.0, 'debug/num_lat_loss': 1764.0, 'epoch': 5.04}

 50%|█████     | 368/730 [32:34<30:55,  5.13s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.40020421147346497, 'epoch': 5.04}

 50%|█████     | 368/730 [32:34<30:55,  5.13s/it]
                                                 
{'train/learning_rate_real': 1.2943634589815915e-05, 'epoch': 5.04}

 50%|█████     | 368/730 [32:34<30:55,  5.13s/it]
 51%|█████     | 369/730 [32:35<31:27,  5.23s/it]
                                                 
{'debug/num_tok_total': 2392.0, 'debug/num_tok_loss': 1762.0, 'debug/num_lat_total': 2392.0, 'debug/num_lat_loss': 1762.0, 'epoch': 5.05}

 51%|█████     | 369/730 [32:36<31:27,  5.23s/it]
                                                 
{'train/ce_loss': 1.984375, 'train/diffusion_loss': 0.4384942352771759, 'epoch': 5.05}

 51%|█████     | 369/730 [32:36<31:27,  5.23s/it]
                                                 
{'train/learning_rate_real': 1.2888199375600754e-05, 'epoch': 5.05}

 51%|█████     | 369/730 [32:36<31:27,  5.23s/it]
                                                 
{'debug/num_tok_total': 2166.0, 'debug/num_tok_loss': 1728.0, 'debug/num_lat_total': 2166.0, 'debug/num_lat_loss': 1728.0, 'epoch': 5.05}

 51%|█████     | 369/730 [32:37<31:27,  5.23s/it]
                                                 
{'train/ce_loss': 2.1875, 'train/diffusion_loss': 0.4792060852050781, 'epoch': 5.05}

 51%|█████     | 369/730 [32:37<31:27,  5.23s/it]
                                                 
{'train/learning_rate_real': 1.2888199375600754e-05, 'epoch': 5.05}

 51%|█████     | 369/730 [32:37<31:27,  5.23s/it]
                                                 
{'debug/num_tok_total': 3099.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 3099.0, 'debug/num_lat_loss': 1785.0, 'epoch': 5.05}

 51%|█████     | 369/730 [32:38<31:27,  5.23s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.32182613015174866, 'epoch': 5.05}

 51%|█████     | 369/730 [32:38<31:27,  5.23s/it]
                                                 
{'train/learning_rate_real': 1.2888199375600754e-05, 'epoch': 5.05}

 51%|█████     | 369/730 [32:38<31:27,  5.23s/it]
                                                 
{'debug/num_tok_total': 2642.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2642.0, 'debug/num_lat_loss': 1776.0, 'epoch': 5.05}

 51%|█████     | 369/730 [32:40<31:27,  5.23s/it]
                                                 
{'train/ce_loss': 1.8515625, 'train/diffusion_loss': 0.39480388164520264, 'epoch': 5.05}

 51%|█████     | 369/730 [32:40<31:27,  5.23s/it]
                                                 
{'train/learning_rate_real': 1.2888199375600754e-05, 'epoch': 5.05}

 51%|█████     | 369/730 [32:40<31:27,  5.23s/it]03/16/2026 07:20:12 - INFO - __main__ - LoRA debug step 370: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 51%|█████     | 370/730 [32:40<31:28,  5.24s/it]
                                                 
{'loss': 2.6214, 'grad_norm': 1.216352105140686, 'learning_rate': 1.2888199375600754e-05, 'epoch': 5.07}

 51%|█████     | 370/730 [32:40<31:28,  5.24s/it]
                                                 
{'debug/num_tok_total': 2209.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2209.0, 'debug/num_lat_loss': 1787.0, 'epoch': 5.07}

 51%|█████     | 370/730 [32:41<31:28,  5.24s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.4800213873386383, 'epoch': 5.07}

 51%|█████     | 370/730 [32:41<31:28,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.2832756517972185e-05, 'epoch': 5.07}

 51%|█████     | 370/730 [32:41<31:28,  5.24s/it]
                                                 
{'debug/num_tok_total': 2674.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2674.0, 'debug/num_lat_loss': 1788.0, 'epoch': 5.07}

 51%|█████     | 370/730 [32:42<31:28,  5.24s/it]
                                                 
{'train/ce_loss': 1.9453125, 'train/diffusion_loss': 0.3754417300224304, 'epoch': 5.07}

 51%|█████     | 370/730 [32:42<31:28,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.2832756517972185e-05, 'epoch': 5.07}

 51%|█████     | 370/730 [32:42<31:28,  5.24s/it]
                                                 
{'debug/num_tok_total': 2451.0, 'debug/num_tok_loss': 1680.0, 'debug/num_lat_total': 2451.0, 'debug/num_lat_loss': 1680.0, 'epoch': 5.07}

 51%|█████     | 370/730 [32:44<31:28,  5.24s/it]
                                                 
{'train/ce_loss': 1.9140625, 'train/diffusion_loss': 0.4207042455673218, 'epoch': 5.07}

 51%|█████     | 370/730 [32:44<31:28,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.2832756517972185e-05, 'epoch': 5.07}

 51%|█████     | 370/730 [32:44<31:28,  5.24s/it]
                                                 
{'debug/num_tok_total': 2827.0, 'debug/num_tok_loss': 1760.0, 'debug/num_lat_total': 2827.0, 'debug/num_lat_loss': 1760.0, 'epoch': 5.07}

 51%|█████     | 370/730 [32:45<31:28,  5.24s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.3967849016189575, 'epoch': 5.07}

 51%|█████     | 370/730 [32:45<31:28,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.2832756517972185e-05, 'epoch': 5.07}

 51%|█████     | 370/730 [32:45<31:28,  5.24s/it]
 51%|█████     | 371/730 [32:45<31:24,  5.25s/it]
                                                 
{'debug/num_tok_total': 2455.0, 'debug/num_tok_loss': 1817.0, 'debug/num_lat_total': 2455.0, 'debug/num_lat_loss': 1817.0, 'epoch': 5.08}

 51%|█████     | 371/730 [32:46<31:24,  5.25s/it]
                                                 
{'train/ce_loss': 1.90625, 'train/diffusion_loss': 0.42998573184013367, 'epoch': 5.08}

 51%|█████     | 371/730 [32:46<31:24,  5.25s/it]
                                                 
{'train/learning_rate_real': 1.2777307108566896e-05, 'epoch': 5.08}

 51%|█████     | 371/730 [32:46<31:24,  5.25s/it]
                                                 
{'debug/num_tok_total': 2584.0, 'debug/num_tok_loss': 1750.0, 'debug/num_lat_total': 2584.0, 'debug/num_lat_loss': 1750.0, 'epoch': 5.08}

 51%|█████     | 371/730 [32:48<31:24,  5.25s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.39576098322868347, 'epoch': 5.08}

 51%|█████     | 371/730 [32:48<31:24,  5.25s/it]
                                                 
{'train/learning_rate_real': 1.2777307108566896e-05, 'epoch': 5.08}

 51%|█████     | 371/730 [32:48<31:24,  5.25s/it]
                                                 
{'debug/num_tok_total': 2644.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2644.0, 'debug/num_lat_loss': 1781.0, 'epoch': 5.08}

 51%|█████     | 371/730 [32:49<31:24,  5.25s/it]
                                                 
{'train/ce_loss': 1.9765625, 'train/diffusion_loss': 0.393177330493927, 'epoch': 5.08}

 51%|█████     | 371/730 [32:49<31:24,  5.25s/it]
                                                 
{'train/learning_rate_real': 1.2777307108566896e-05, 'epoch': 5.08}

 51%|█████     | 371/730 [32:49<31:24,  5.25s/it]
                                                 
{'debug/num_tok_total': 2629.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 2629.0, 'debug/num_lat_loss': 1767.0, 'epoch': 5.08}

 51%|█████     | 371/730 [32:50<31:24,  5.25s/it]
                                                 
{'train/ce_loss': 1.9296875, 'train/diffusion_loss': 0.389772891998291, 'epoch': 5.08}

 51%|█████     | 371/730 [32:50<31:24,  5.25s/it]
                                                 
{'train/learning_rate_real': 1.2777307108566896e-05, 'epoch': 5.08}

 51%|█████     | 371/730 [32:50<31:24,  5.25s/it]
 51%|█████     | 372/730 [32:50<31:13,  5.23s/it]
                                                 
{'debug/num_tok_total': 2686.0, 'debug/num_tok_loss': 1614.0, 'debug/num_lat_total': 2686.0, 'debug/num_lat_loss': 1614.0, 'epoch': 5.1}

 51%|█████     | 372/730 [32:52<31:13,  5.23s/it]
                                                 
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.33671796321868896, 'epoch': 5.1}

 51%|█████     | 372/730 [32:52<31:13,  5.23s/it]
                                                 
{'train/learning_rate_real': 1.2721852239150577e-05, 'epoch': 5.1}

 51%|█████     | 372/730 [32:52<31:13,  5.23s/it]
                                                 
{'debug/num_tok_total': 3071.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 3071.0, 'debug/num_lat_loss': 1784.0, 'epoch': 5.1}

 51%|█████     | 372/730 [32:53<31:13,  5.23s/it]
                                                 
{'train/ce_loss': 1.9140625, 'train/diffusion_loss': 0.30351942777633667, 'epoch': 5.1}

 51%|█████     | 372/730 [32:53<31:13,  5.23s/it]
                                                 
{'train/learning_rate_real': 1.2721852239150577e-05, 'epoch': 5.1}

 51%|█████     | 372/730 [32:53<31:13,  5.23s/it]
                                                 
{'debug/num_tok_total': 2644.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2644.0, 'debug/num_lat_loss': 1787.0, 'epoch': 5.1}

 51%|█████     | 372/730 [32:54<31:13,  5.23s/it]
                                                 
{'train/ce_loss': 1.890625, 'train/diffusion_loss': 0.37298259139060974, 'epoch': 5.1}

 51%|█████     | 372/730 [32:54<31:13,  5.23s/it]
                                                 
{'train/learning_rate_real': 1.2721852239150577e-05, 'epoch': 5.1}

 51%|█████     | 372/730 [32:54<31:13,  5.23s/it]
                                                 
{'debug/num_tok_total': 2501.0, 'debug/num_tok_loss': 1663.0, 'debug/num_lat_total': 2501.0, 'debug/num_lat_loss': 1663.0, 'epoch': 5.1}

 51%|█████     | 372/730 [32:55<31:13,  5.23s/it]
                                                 
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.375630259513855, 'epoch': 5.1}

 51%|█████     | 372/730 [32:55<31:13,  5.23s/it]
                                                 
{'train/learning_rate_real': 1.2721852239150577e-05, 'epoch': 5.1}

 51%|█████     | 372/730 [32:55<31:13,  5.23s/it]
 51%|█████     | 373/730 [32:56<31:34,  5.31s/it]
                                                 
{'debug/num_tok_total': 2900.0, 'debug/num_tok_loss': 1618.0, 'debug/num_lat_total': 2900.0, 'debug/num_lat_loss': 1618.0, 'epoch': 5.11}

 51%|█████     | 373/730 [32:57<31:34,  5.31s/it]
                                                 
{'train/ce_loss': 1.9375, 'train/diffusion_loss': 0.2602494955062866, 'epoch': 5.11}

 51%|█████     | 373/730 [32:57<31:34,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.2666393001596424e-05, 'epoch': 5.11}

 51%|█████     | 373/730 [32:57<31:34,  5.31s/it]
                                                 
{'debug/num_tok_total': 2465.0, 'debug/num_tok_loss': 1808.0, 'debug/num_lat_total': 2465.0, 'debug/num_lat_loss': 1808.0, 'epoch': 5.11}

 51%|█████     | 373/730 [32:58<31:34,  5.31s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.4340699315071106, 'epoch': 5.11}

 51%|█████     | 373/730 [32:58<31:34,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.2666393001596424e-05, 'epoch': 5.11}

 51%|█████     | 373/730 [32:58<31:34,  5.31s/it]
                                                 
{'debug/num_tok_total': 2623.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2623.0, 'debug/num_lat_loss': 1777.0, 'epoch': 5.11}

 51%|█████     | 373/730 [33:00<31:34,  5.31s/it]
                                                 
{'train/ce_loss': 2.125, 'train/diffusion_loss': 0.4099808931350708, 'epoch': 5.11}

 51%|█████     | 373/730 [33:00<31:34,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.2666393001596424e-05, 'epoch': 5.11}

 51%|█████     | 373/730 [33:00<31:34,  5.31s/it]
                                                 
{'debug/num_tok_total': 2411.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2411.0, 'debug/num_lat_loss': 1780.0, 'epoch': 5.11}

 51%|█████     | 373/730 [33:01<31:34,  5.31s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.44482001662254333, 'epoch': 5.11}

 51%|█████     | 373/730 [33:01<31:34,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.2666393001596424e-05, 'epoch': 5.11}

 51%|█████     | 373/730 [33:01<31:34,  5.31s/it]
 51%|█████     | 374/730 [33:01<31:31,  5.31s/it]
                                                 
{'debug/num_tok_total': 2376.0, 'debug/num_tok_loss': 1754.0, 'debug/num_lat_total': 2376.0, 'debug/num_lat_loss': 1754.0, 'epoch': 5.12}

 51%|█████     | 374/730 [33:02<31:31,  5.31s/it]
                                                 
{'train/ce_loss': 1.875, 'train/diffusion_loss': 0.43012309074401855, 'epoch': 5.12}

 51%|█████     | 374/730 [33:02<31:31,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.2610930487863637e-05, 'epoch': 5.12}

 51%|█████     | 374/730 [33:02<31:31,  5.31s/it]
                                                 
{'debug/num_tok_total': 3108.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 3108.0, 'debug/num_lat_loss': 1794.0, 'epoch': 5.12}

 51%|█████     | 374/730 [33:04<31:31,  5.31s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.27529823780059814, 'epoch': 5.12}

 51%|█████     | 374/730 [33:04<31:31,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.2610930487863637e-05, 'epoch': 5.12}

 51%|█████     | 374/730 [33:04<31:31,  5.31s/it]
                                                 
{'debug/num_tok_total': 2882.0, 'debug/num_tok_loss': 1800.0, 'debug/num_lat_total': 2882.0, 'debug/num_lat_loss': 1800.0, 'epoch': 5.12}

 51%|█████     | 374/730 [33:05<31:31,  5.31s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.361605703830719, 'epoch': 5.12}

 51%|█████     | 374/730 [33:05<31:31,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.2610930487863637e-05, 'epoch': 5.12}

 51%|█████     | 374/730 [33:05<31:31,  5.31s/it]
                                                 
{'debug/num_tok_total': 2841.0, 'debug/num_tok_loss': 1760.0, 'debug/num_lat_total': 2841.0, 'debug/num_lat_loss': 1760.0, 'epoch': 5.12}

 51%|█████     | 374/730 [33:06<31:31,  5.31s/it]
                                                 
{'train/ce_loss': 1.9609375, 'train/diffusion_loss': 0.33355703949928284, 'epoch': 5.12}

 51%|█████     | 374/730 [33:06<31:31,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.2610930487863637e-05, 'epoch': 5.12}

 51%|█████     | 374/730 [33:06<31:31,  5.31s/it]
 51%|█████▏    | 375/730 [33:07<31:31,  5.33s/it]
                                                 
{'debug/num_tok_total': 2514.0, 'debug/num_tok_loss': 1639.0, 'debug/num_lat_total': 2514.0, 'debug/num_lat_loss': 1639.0, 'epoch': 5.14}

 51%|█████▏    | 375/730 [33:08<31:31,  5.33s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.3975639343261719, 'epoch': 5.14}

 51%|█████▏    | 375/730 [33:08<31:31,  5.33s/it]
                                                 
{'train/learning_rate_real': 1.2555465789975927e-05, 'epoch': 5.14}

 51%|█████▏    | 375/730 [33:08<31:31,  5.33s/it]
                                                 
{'debug/num_tok_total': 2899.0, 'debug/num_tok_loss': 1816.0, 'debug/num_lat_total': 2899.0, 'debug/num_lat_loss': 1816.0, 'epoch': 5.14}

 51%|█████▏    | 375/730 [33:09<31:31,  5.33s/it]
                                                 
{'train/ce_loss': 2.046875, 'train/diffusion_loss': 0.36548635363578796, 'epoch': 5.14}

 51%|█████▏    | 375/730 [33:09<31:31,  5.33s/it]
                                                 
{'train/learning_rate_real': 1.2555465789975927e-05, 'epoch': 5.14}

 51%|█████▏    | 375/730 [33:09<31:31,  5.33s/it]
                                                 
{'debug/num_tok_total': 3073.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 3073.0, 'debug/num_lat_loss': 1782.0, 'epoch': 5.14}

 51%|█████▏    | 375/730 [33:10<31:31,  5.33s/it]
                                                 
{'train/ce_loss': 1.8515625, 'train/diffusion_loss': 0.29560744762420654, 'epoch': 5.14}

 51%|█████▏    | 375/730 [33:10<31:31,  5.33s/it]
                                                 
{'train/learning_rate_real': 1.2555465789975927e-05, 'epoch': 5.14}

 51%|█████▏    | 375/730 [33:10<31:31,  5.33s/it]
                                                 
{'debug/num_tok_total': 2853.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2853.0, 'debug/num_lat_loss': 1776.0, 'epoch': 5.14}

 51%|█████▏    | 375/730 [33:12<31:31,  5.33s/it]
                                                 
{'train/ce_loss': 2.078125, 'train/diffusion_loss': 0.331356942653656, 'epoch': 5.14}

 51%|█████▏    | 375/730 [33:12<31:31,  5.33s/it]
                                                 
{'train/learning_rate_real': 1.2555465789975927e-05, 'epoch': 5.14}

 51%|█████▏    | 375/730 [33:12<31:31,  5.33s/it]
 52%|█████▏    | 376/730 [33:12<31:42,  5.37s/it]
                                                 
{'debug/num_tok_total': 2923.0, 'debug/num_tok_loss': 1712.0, 'debug/num_lat_total': 2923.0, 'debug/num_lat_loss': 1712.0, 'epoch': 5.15}

 52%|█████▏    | 376/730 [33:13<31:42,  5.37s/it]
                                                 
{'train/ce_loss': 1.875, 'train/diffusion_loss': 0.32675445079803467, 'epoch': 5.15}

 52%|█████▏    | 376/730 [33:13<31:42,  5.37s/it]
                                                 
{'train/learning_rate_real': 1.25e-05, 'epoch': 5.15}

 52%|█████▏    | 376/730 [33:13<31:42,  5.37s/it]
                                                 
{'debug/num_tok_total': 3063.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 3063.0, 'debug/num_lat_loss': 1780.0, 'epoch': 5.15}

 52%|█████▏    | 376/730 [33:15<31:42,  5.37s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.3256686329841614, 'epoch': 5.15}

 52%|█████▏    | 376/730 [33:15<31:42,  5.37s/it]
                                                 
{'train/learning_rate_real': 1.25e-05, 'epoch': 5.15}

 52%|█████▏    | 376/730 [33:15<31:42,  5.37s/it]
                                                 
{'debug/num_tok_total': 2843.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2843.0, 'debug/num_lat_loss': 1780.0, 'epoch': 5.15}

 52%|█████▏    | 376/730 [33:16<31:42,  5.37s/it]
                                                 
{'train/ce_loss': 1.875, 'train/diffusion_loss': 0.34669750928878784, 'epoch': 5.15}

 52%|█████▏    | 376/730 [33:16<31:42,  5.37s/it]
                                                 
{'train/learning_rate_real': 1.25e-05, 'epoch': 5.15}

 52%|█████▏    | 376/730 [33:16<31:42,  5.37s/it]
                                                 
{'debug/num_tok_total': 2408.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2408.0, 'debug/num_lat_loss': 1776.0, 'epoch': 5.15}

 52%|█████▏    | 376/730 [33:17<31:42,  5.37s/it]
                                                 
{'train/ce_loss': 2.234375, 'train/diffusion_loss': 0.45397236943244934, 'epoch': 5.15}

 52%|█████▏    | 376/730 [33:17<31:42,  5.37s/it]
                                                 
{'train/learning_rate_real': 1.25e-05, 'epoch': 5.15}

 52%|█████▏    | 376/730 [33:17<31:42,  5.37s/it]
 52%|█████▏    | 377/730 [33:17<31:31,  5.36s/it]
                                                 
{'debug/num_tok_total': 2463.0, 'debug/num_tok_loss': 1619.0, 'debug/num_lat_total': 2463.0, 'debug/num_lat_loss': 1619.0, 'epoch': 5.16}

 52%|█████▏    | 377/730 [33:19<31:31,  5.36s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.41008567810058594, 'epoch': 5.16}

 52%|█████▏    | 377/730 [33:19<31:31,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.2444534210024075e-05, 'epoch': 5.16}

 52%|█████▏    | 377/730 [33:19<31:31,  5.36s/it]
                                                 
{'debug/num_tok_total': 2644.0, 'debug/num_tok_loss': 1677.0, 'debug/num_lat_total': 2644.0, 'debug/num_lat_loss': 1677.0, 'epoch': 5.16}

 52%|█████▏    | 377/730 [33:20<31:31,  5.36s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.3623366355895996, 'epoch': 5.16}

 52%|█████▏    | 377/730 [33:20<31:31,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.2444534210024075e-05, 'epoch': 5.16}

 52%|█████▏    | 377/730 [33:20<31:31,  5.36s/it]
                                                 
{'debug/num_tok_total': 2409.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2409.0, 'debug/num_lat_loss': 1773.0, 'epoch': 5.16}

 52%|█████▏    | 377/730 [33:21<31:31,  5.36s/it]
                                                 
{'train/ce_loss': 1.890625, 'train/diffusion_loss': 0.4463941156864166, 'epoch': 5.16}

 52%|█████▏    | 377/730 [33:21<31:31,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.2444534210024075e-05, 'epoch': 5.16}

 52%|█████▏    | 377/730 [33:21<31:31,  5.36s/it]
                                                 
{'debug/num_tok_total': 2702.0, 'debug/num_tok_loss': 1622.0, 'debug/num_lat_total': 2702.0, 'debug/num_lat_loss': 1622.0, 'epoch': 5.16}

 52%|█████▏    | 377/730 [33:22<31:31,  5.36s/it]
                                                 
{'train/ce_loss': 2.375, 'train/diffusion_loss': 0.333406001329422, 'epoch': 5.16}

 52%|█████▏    | 377/730 [33:22<31:31,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.2444534210024075e-05, 'epoch': 5.16}

 52%|█████▏    | 377/730 [33:22<31:31,  5.36s/it]
 52%|█████▏    | 378/730 [33:23<31:11,  5.32s/it]
                                                 
{'debug/num_tok_total': 3121.0, 'debug/num_tok_loss': 1815.0, 'debug/num_lat_total': 3121.0, 'debug/num_lat_loss': 1815.0, 'epoch': 5.18}

 52%|█████▏    | 378/730 [33:24<31:11,  5.32s/it]
                                                 
{'train/ce_loss': 1.890625, 'train/diffusion_loss': 0.32326439023017883, 'epoch': 5.18}

 52%|█████▏    | 378/730 [33:24<31:11,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.2389069512136362e-05, 'epoch': 5.18}

 52%|█████▏    | 378/730 [33:24<31:11,  5.32s/it]
                                                 
{'debug/num_tok_total': 2611.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 2611.0, 'debug/num_lat_loss': 1768.0, 'epoch': 5.18}

 52%|█████▏    | 378/730 [33:25<31:11,  5.32s/it]
                                                 
{'train/ce_loss': 1.8984375, 'train/diffusion_loss': 0.45180970430374146, 'epoch': 5.18}

 52%|█████▏    | 378/730 [33:25<31:11,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.2389069512136362e-05, 'epoch': 5.18}

 52%|█████▏    | 378/730 [33:25<31:11,  5.32s/it]
                                                 
{'debug/num_tok_total': 2833.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2833.0, 'debug/num_lat_loss': 1774.0, 'epoch': 5.18}

 52%|█████▏    | 378/730 [33:27<31:11,  5.32s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.39444664120674133, 'epoch': 5.18}

 52%|█████▏    | 378/730 [33:27<31:11,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.2389069512136362e-05, 'epoch': 5.18}

 52%|█████▏    | 378/730 [33:27<31:11,  5.32s/it]
                                                 
{'debug/num_tok_total': 2673.0, 'debug/num_tok_loss': 1800.0, 'debug/num_lat_total': 2673.0, 'debug/num_lat_loss': 1800.0, 'epoch': 5.18}

 52%|█████▏    | 378/730 [33:28<31:11,  5.32s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.42039254307746887, 'epoch': 5.18}

 52%|█████▏    | 378/730 [33:28<31:11,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.2389069512136362e-05, 'epoch': 5.18}

 52%|█████▏    | 378/730 [33:28<31:11,  5.32s/it]
 52%|█████▏    | 379/730 [33:28<31:37,  5.41s/it]
                                                 
{'debug/num_tok_total': 2181.0, 'debug/num_tok_loss': 1760.0, 'debug/num_lat_total': 2181.0, 'debug/num_lat_loss': 1760.0, 'epoch': 5.19}

 52%|█████▏    | 379/730 [33:29<31:37,  5.41s/it]
                                                 
{'train/ce_loss': 1.9140625, 'train/diffusion_loss': 0.45792582631111145, 'epoch': 5.19}

 52%|█████▏    | 379/730 [33:29<31:37,  5.41s/it]
                                                 
{'train/learning_rate_real': 1.233360699840358e-05, 'epoch': 5.19}

 52%|█████▏    | 379/730 [33:29<31:37,  5.41s/it]
                                                 
{'debug/num_tok_total': 3102.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 3102.0, 'debug/num_lat_loss': 1796.0, 'epoch': 5.19}

 52%|█████▏    | 379/730 [33:31<31:37,  5.41s/it]
                                                 
{'train/ce_loss': 1.96875, 'train/diffusion_loss': 0.3423309624195099, 'epoch': 5.19}

 52%|█████▏    | 379/730 [33:31<31:37,  5.41s/it]
                                                 
{'train/learning_rate_real': 1.233360699840358e-05, 'epoch': 5.19}

 52%|█████▏    | 379/730 [33:31<31:37,  5.41s/it]
                                                 
{'debug/num_tok_total': 3306.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 3306.0, 'debug/num_lat_loss': 1795.0, 'epoch': 5.19}

 52%|█████▏    | 379/730 [33:32<31:37,  5.41s/it]
                                                 
{'train/ce_loss': 1.890625, 'train/diffusion_loss': 0.24859732389450073, 'epoch': 5.19}

 52%|█████▏    | 379/730 [33:32<31:37,  5.41s/it]
                                                 
{'train/learning_rate_real': 1.233360699840358e-05, 'epoch': 5.19}

 52%|█████▏    | 379/730 [33:32<31:37,  5.41s/it]
                                                 
{'debug/num_tok_total': 2675.0, 'debug/num_tok_loss': 1814.0, 'debug/num_lat_total': 2675.0, 'debug/num_lat_loss': 1814.0, 'epoch': 5.19}

 52%|█████▏    | 379/730 [33:33<31:37,  5.41s/it]
                                                 
{'train/ce_loss': 1.9140625, 'train/diffusion_loss': 0.3955957889556885, 'epoch': 5.19}

 52%|█████▏    | 379/730 [33:33<31:37,  5.41s/it]
                                                 
{'train/learning_rate_real': 1.233360699840358e-05, 'epoch': 5.19}

 52%|█████▏    | 379/730 [33:33<31:37,  5.41s/it]03/16/2026 07:21:06 - INFO - __main__ - LoRA debug step 380: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 52%|█████▏    | 380/730 [33:34<31:56,  5.48s/it]
                                                 
{'loss': 2.4153, 'grad_norm': 1.3502992391586304, 'learning_rate': 1.233360699840358e-05, 'epoch': 5.21}

 52%|█████▏    | 380/730 [33:34<31:56,  5.48s/it]
                                                 
{'debug/num_tok_total': 2815.0, 'debug/num_tok_loss': 1749.0, 'debug/num_lat_total': 2815.0, 'debug/num_lat_loss': 1749.0, 'epoch': 5.21}

 52%|█████▏    | 380/730 [33:35<31:56,  5.48s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.33071979880332947, 'epoch': 5.21}

 52%|█████▏    | 380/730 [33:35<31:56,  5.48s/it]
                                                 
{'train/learning_rate_real': 1.2278147760849427e-05, 'epoch': 5.21}

 52%|█████▏    | 380/730 [33:35<31:56,  5.48s/it]
                                                 
{'debug/num_tok_total': 3330.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 3330.0, 'debug/num_lat_loss': 1799.0, 'epoch': 5.21}

 52%|█████▏    | 380/730 [33:36<31:56,  5.48s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.2971540093421936, 'epoch': 5.21}

 52%|█████▏    | 380/730 [33:36<31:56,  5.48s/it]
                                                 
{'train/learning_rate_real': 1.2278147760849427e-05, 'epoch': 5.21}

 52%|█████▏    | 380/730 [33:36<31:56,  5.48s/it]
                                                 
{'debug/num_tok_total': 2170.0, 'debug/num_tok_loss': 1646.0, 'debug/num_lat_total': 2170.0, 'debug/num_lat_loss': 1646.0, 'epoch': 5.21}

 52%|█████▏    | 380/730 [33:38<31:56,  5.48s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.46348363161087036, 'epoch': 5.21}

 52%|█████▏    | 380/730 [33:38<31:56,  5.48s/it]
                                                 
{'train/learning_rate_real': 1.2278147760849427e-05, 'epoch': 5.21}

 52%|█████▏    | 380/730 [33:38<31:56,  5.48s/it]
                                                 
{'debug/num_tok_total': 2379.0, 'debug/num_tok_loss': 1652.0, 'debug/num_lat_total': 2379.0, 'debug/num_lat_loss': 1652.0, 'epoch': 5.21}

 52%|█████▏    | 380/730 [33:39<31:56,  5.48s/it]
                                                 
{'train/ce_loss': 1.984375, 'train/diffusion_loss': 0.41309061646461487, 'epoch': 5.21}

 52%|█████▏    | 380/730 [33:39<31:56,  5.48s/it]
                                                 
{'train/learning_rate_real': 1.2278147760849427e-05, 'epoch': 5.21}

 52%|█████▏    | 380/730 [33:39<31:56,  5.48s/it]
 52%|█████▏    | 381/730 [33:39<31:41,  5.45s/it]
                                                 
{'debug/num_tok_total': 3326.0, 'debug/num_tok_loss': 1798.0, 'debug/num_lat_total': 3326.0, 'debug/num_lat_loss': 1798.0, 'epoch': 5.22}

 52%|█████▏    | 381/730 [33:41<31:41,  5.45s/it]
                                                 
{'train/ce_loss': 1.90625, 'train/diffusion_loss': 0.28062987327575684, 'epoch': 5.22}

 52%|█████▏    | 381/730 [33:41<31:41,  5.45s/it]
                                                 
{'train/learning_rate_real': 1.2222692891433105e-05, 'epoch': 5.22}

 52%|█████▏    | 381/730 [33:41<31:41,  5.45s/it]
                                                 
{'debug/num_tok_total': 2901.0, 'debug/num_tok_loss': 1807.0, 'debug/num_lat_total': 2901.0, 'debug/num_lat_loss': 1807.0, 'epoch': 5.22}

 52%|█████▏    | 381/730 [33:42<31:41,  5.45s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.31966787576675415, 'epoch': 5.22}

 52%|█████▏    | 381/730 [33:42<31:41,  5.45s/it]
                                                 
{'train/learning_rate_real': 1.2222692891433105e-05, 'epoch': 5.22}

 52%|█████▏    | 381/730 [33:42<31:41,  5.45s/it]
                                                 
{'debug/num_tok_total': 2784.0, 'debug/num_tok_loss': 1719.0, 'debug/num_lat_total': 2784.0, 'debug/num_lat_loss': 1719.0, 'epoch': 5.22}

 52%|█████▏    | 381/730 [33:43<31:41,  5.45s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.34212782979011536, 'epoch': 5.22}

 52%|█████▏    | 381/730 [33:43<31:41,  5.45s/it]
                                                 
{'train/learning_rate_real': 1.2222692891433105e-05, 'epoch': 5.22}

 52%|█████▏    | 381/730 [33:43<31:41,  5.45s/it]
                                                 
{'debug/num_tok_total': 2858.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2858.0, 'debug/num_lat_loss': 1775.0, 'epoch': 5.22}

 52%|█████▏    | 381/730 [33:45<31:41,  5.45s/it]
                                                 
{'train/ce_loss': 2.03125, 'train/diffusion_loss': 0.36847585439682007, 'epoch': 5.22}

 52%|█████▏    | 381/730 [33:45<31:41,  5.45s/it]
                                                 
{'train/learning_rate_real': 1.2222692891433105e-05, 'epoch': 5.22}

 52%|█████▏    | 381/730 [33:45<31:41,  5.45s/it]
 52%|█████▏    | 382/730 [33:45<32:14,  5.56s/it]
                                                 
{'debug/num_tok_total': 2416.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 2416.0, 'debug/num_lat_loss': 1769.0, 'epoch': 5.23}

 52%|█████▏    | 382/730 [33:46<32:14,  5.56s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.445895254611969, 'epoch': 5.23}

 52%|█████▏    | 382/730 [33:46<32:14,  5.56s/it]
                                                 
{'train/learning_rate_real': 1.2167243482027816e-05, 'epoch': 5.23}

 52%|█████▏    | 382/730 [33:46<32:14,  5.56s/it]
                                                 
{'debug/num_tok_total': 2486.0, 'debug/num_tok_loss': 1603.0, 'debug/num_lat_total': 2486.0, 'debug/num_lat_loss': 1603.0, 'epoch': 5.23}

 52%|█████▏    | 382/730 [33:47<32:14,  5.56s/it]
                                                 
{'train/ce_loss': 2.140625, 'train/diffusion_loss': 0.36970221996307373, 'epoch': 5.23}

 52%|█████▏    | 382/730 [33:47<32:14,  5.56s/it]
                                                 
{'train/learning_rate_real': 1.2167243482027816e-05, 'epoch': 5.23}

 52%|█████▏    | 382/730 [33:47<32:14,  5.56s/it]
                                                 
{'debug/num_tok_total': 2218.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2218.0, 'debug/num_lat_loss': 1778.0, 'epoch': 5.23}

 52%|█████▏    | 382/730 [33:48<32:14,  5.56s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.4734830856323242, 'epoch': 5.23}

 52%|█████▏    | 382/730 [33:48<32:14,  5.56s/it]
                                                 
{'train/learning_rate_real': 1.2167243482027816e-05, 'epoch': 5.23}

 52%|█████▏    | 382/730 [33:48<32:14,  5.56s/it]
                                                 
{'debug/num_tok_total': 3087.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 3087.0, 'debug/num_lat_loss': 1783.0, 'epoch': 5.23}

 52%|█████▏    | 382/730 [33:50<32:14,  5.56s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.32064247131347656, 'epoch': 5.23}

 52%|█████▏    | 382/730 [33:50<32:14,  5.56s/it]
                                                 
{'train/learning_rate_real': 1.2167243482027816e-05, 'epoch': 5.23}

 52%|█████▏    | 382/730 [33:50<32:14,  5.56s/it]
 52%|█████▏    | 383/730 [33:50<31:34,  5.46s/it]
                                                 
{'debug/num_tok_total': 1936.0, 'debug/num_tok_loss': 1644.0, 'debug/num_lat_total': 1936.0, 'debug/num_lat_loss': 1644.0, 'epoch': 5.25}

 52%|█████▏    | 383/730 [33:51<31:34,  5.46s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.5105114579200745, 'epoch': 5.25}

 52%|█████▏    | 383/730 [33:51<31:34,  5.46s/it]
                                                 
{'train/learning_rate_real': 1.2111800624399244e-05, 'epoch': 5.25}

 52%|█████▏    | 383/730 [33:51<31:34,  5.46s/it]
                                                 
{'debug/num_tok_total': 2870.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2870.0, 'debug/num_lat_loss': 1779.0, 'epoch': 5.25}

 52%|█████▏    | 383/730 [33:53<31:34,  5.46s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.3411625623703003, 'epoch': 5.25}

 52%|█████▏    | 383/730 [33:53<31:34,  5.46s/it]
                                                 
{'train/learning_rate_real': 1.2111800624399244e-05, 'epoch': 5.25}

 52%|█████▏    | 383/730 [33:53<31:34,  5.46s/it]
                                                 
{'debug/num_tok_total': 3362.0, 'debug/num_tok_loss': 1821.0, 'debug/num_lat_total': 3362.0, 'debug/num_lat_loss': 1821.0, 'epoch': 5.25}

 52%|█████▏    | 383/730 [33:54<31:34,  5.46s/it]
                                                 
{'train/ce_loss': 1.875, 'train/diffusion_loss': 0.2604753375053406, 'epoch': 5.25}

 52%|█████▏    | 383/730 [33:54<31:34,  5.46s/it]
                                                 
{'train/learning_rate_real': 1.2111800624399244e-05, 'epoch': 5.25}

 52%|█████▏    | 383/730 [33:54<31:34,  5.46s/it]
                                                 
{'debug/num_tok_total': 2631.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2631.0, 'debug/num_lat_loss': 1773.0, 'epoch': 5.25}

 52%|█████▏    | 383/730 [33:55<31:34,  5.46s/it]
                                                 
{'train/ce_loss': 1.96875, 'train/diffusion_loss': 0.38157889246940613, 'epoch': 5.25}

 52%|█████▏    | 383/730 [33:55<31:34,  5.46s/it]
                                                 
{'train/learning_rate_real': 1.2111800624399244e-05, 'epoch': 5.25}

 52%|█████▏    | 383/730 [33:55<31:34,  5.46s/it]
 53%|█████▎    | 384/730 [33:56<31:29,  5.46s/it]
                                                 
{'debug/num_tok_total': 2421.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2421.0, 'debug/num_lat_loss': 1782.0, 'epoch': 5.26}

 53%|█████▎    | 384/730 [33:57<31:29,  5.46s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.4495629668235779, 'epoch': 5.26}

 53%|█████▎    | 384/730 [33:57<31:29,  5.46s/it]
                                                 
{'train/learning_rate_real': 1.2056365410184088e-05, 'epoch': 5.26}

 53%|█████▎    | 384/730 [33:57<31:29,  5.46s/it]
                                                 
{'debug/num_tok_total': 3148.0, 'debug/num_tok_loss': 1823.0, 'debug/num_lat_total': 3148.0, 'debug/num_lat_loss': 1823.0, 'epoch': 5.26}

 53%|█████▎    | 384/730 [33:58<31:29,  5.46s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.3212644159793854, 'epoch': 5.26}

 53%|█████▎    | 384/730 [33:58<31:29,  5.46s/it]
                                                 
{'train/learning_rate_real': 1.2056365410184088e-05, 'epoch': 5.26}

 53%|█████▎    | 384/730 [33:58<31:29,  5.46s/it]
                                                 
{'debug/num_tok_total': 2003.0, 'debug/num_tok_loss': 1573.0, 'debug/num_lat_total': 2003.0, 'debug/num_lat_loss': 1573.0, 'epoch': 5.26}

 53%|█████▎    | 384/730 [33:59<31:29,  5.46s/it]
                                                 
{'train/ce_loss': 1.96875, 'train/diffusion_loss': 0.46382641792297363, 'epoch': 5.26}

 53%|█████▎    | 384/730 [33:59<31:29,  5.46s/it]
                                                 
{'train/learning_rate_real': 1.2056365410184088e-05, 'epoch': 5.26}

 53%|█████▎    | 384/730 [33:59<31:29,  5.46s/it]
                                                 
{'debug/num_tok_total': 3060.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 3060.0, 'debug/num_lat_loss': 1779.0, 'epoch': 5.26}

 53%|█████▎    | 384/730 [34:01<31:29,  5.46s/it]
                                                 
{'train/ce_loss': 1.984375, 'train/diffusion_loss': 0.3454790711402893, 'epoch': 5.26}

 53%|█████▎    | 384/730 [34:01<31:29,  5.46s/it]
                                                 
{'train/learning_rate_real': 1.2056365410184088e-05, 'epoch': 5.26}

 53%|█████▎    | 384/730 [34:01<31:29,  5.46s/it]
 53%|█████▎    | 385/730 [34:01<31:19,  5.45s/it]
                                                 
{'debug/num_tok_total': 3061.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 3061.0, 'debug/num_lat_loss': 1774.0, 'epoch': 5.27}

 53%|█████▎    | 385/730 [34:02<31:19,  5.45s/it]
                                                 
{'train/ce_loss': 1.65625, 'train/diffusion_loss': 0.30337586998939514, 'epoch': 5.27}

 53%|█████▎    | 385/730 [34:02<31:19,  5.45s/it]
                                                 
{'train/learning_rate_real': 1.2000938930868534e-05, 'epoch': 5.27}

 53%|█████▎    | 385/730 [34:02<31:19,  5.45s/it]
                                                 
{'debug/num_tok_total': 1716.0, 'debug/num_tok_loss': 1594.0, 'debug/num_lat_total': 1716.0, 'debug/num_lat_loss': 1594.0, 'epoch': 5.27}

 53%|█████▎    | 385/730 [34:03<31:19,  5.45s/it]
                                                 
{'train/ce_loss': 1.96875, 'train/diffusion_loss': 0.5590569972991943, 'epoch': 5.27}

 53%|█████▎    | 385/730 [34:03<31:19,  5.45s/it]
                                                 
{'train/learning_rate_real': 1.2000938930868534e-05, 'epoch': 5.27}

 53%|█████▎    | 385/730 [34:03<31:19,  5.45s/it]
                                                 
{'debug/num_tok_total': 2871.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2871.0, 'debug/num_lat_loss': 1794.0, 'epoch': 5.27}

 53%|█████▎    | 385/730 [34:05<31:19,  5.45s/it]
                                                 
{'train/ce_loss': 1.9296875, 'train/diffusion_loss': 0.36918506026268005, 'epoch': 5.27}

 53%|█████▎    | 385/730 [34:05<31:19,  5.45s/it]
                                                 
{'train/learning_rate_real': 1.2000938930868534e-05, 'epoch': 5.27}

 53%|█████▎    | 385/730 [34:05<31:19,  5.45s/it]
                                                 
{'debug/num_tok_total': 3072.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 3072.0, 'debug/num_lat_loss': 1781.0, 'epoch': 5.27}

 53%|█████▎    | 385/730 [34:06<31:19,  5.45s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.3130257725715637, 'epoch': 5.27}

 53%|█████▎    | 385/730 [34:06<31:19,  5.45s/it]
                                                 
{'train/learning_rate_real': 1.2000938930868534e-05, 'epoch': 5.27}

 53%|█████▎    | 385/730 [34:06<31:19,  5.45s/it]
 53%|█████▎    | 386/730 [34:06<30:47,  5.37s/it]
                                                 
{'debug/num_tok_total': 2639.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2639.0, 'debug/num_lat_loss': 1782.0, 'epoch': 5.29}

 53%|█████▎    | 386/730 [34:08<30:47,  5.37s/it]
                                                 
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.39055678248405457, 'epoch': 5.29}

 53%|█████▎    | 386/730 [34:08<30:47,  5.37s/it]
                                                 
{'train/learning_rate_real': 1.1945522277766794e-05, 'epoch': 5.29}

 53%|█████▎    | 386/730 [34:08<30:47,  5.37s/it]
                                                 
{'debug/num_tok_total': 3075.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 3075.0, 'debug/num_lat_loss': 1784.0, 'epoch': 5.29}

 53%|█████▎    | 386/730 [34:09<30:47,  5.37s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.29906129837036133, 'epoch': 5.29}

 53%|█████▎    | 386/730 [34:09<30:47,  5.37s/it]
                                                 
{'train/learning_rate_real': 1.1945522277766794e-05, 'epoch': 5.29}

 53%|█████▎    | 386/730 [34:09<30:47,  5.37s/it]
                                                 
{'debug/num_tok_total': 2658.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2658.0, 'debug/num_lat_loss': 1789.0, 'epoch': 5.29}

 53%|█████▎    | 386/730 [34:10<30:47,  5.37s/it]
                                                 
{'train/ce_loss': 2.015625, 'train/diffusion_loss': 0.40184664726257324, 'epoch': 5.29}

 53%|█████▎    | 386/730 [34:10<30:47,  5.37s/it]
                                                 
{'train/learning_rate_real': 1.1945522277766794e-05, 'epoch': 5.29}

 53%|█████▎    | 386/730 [34:10<30:47,  5.37s/it]
                                                 
{'debug/num_tok_total': 2665.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2665.0, 'debug/num_lat_loss': 1788.0, 'epoch': 5.29}

 53%|█████▎    | 386/730 [34:11<30:47,  5.37s/it]
                                                 
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.3778179883956909, 'epoch': 5.29}

 53%|█████▎    | 386/730 [34:11<30:47,  5.37s/it]
                                                 
{'train/learning_rate_real': 1.1945522277766794e-05, 'epoch': 5.29}

 53%|█████▎    | 386/730 [34:11<30:47,  5.37s/it]
 53%|█████▎    | 387/730 [34:12<30:48,  5.39s/it]
                                                 
{'debug/num_tok_total': 2898.0, 'debug/num_tok_loss': 1808.0, 'debug/num_lat_total': 2898.0, 'debug/num_lat_loss': 1808.0, 'epoch': 5.3}

 53%|█████▎    | 387/730 [34:13<30:48,  5.39s/it]
                                                 
{'train/ce_loss': 2.078125, 'train/diffusion_loss': 0.3692452013492584, 'epoch': 5.3}

 53%|█████▎    | 387/730 [34:13<30:48,  5.39s/it]
                                                 
{'train/learning_rate_real': 1.1890116541999614e-05, 'epoch': 5.3}

 53%|█████▎    | 387/730 [34:13<30:48,  5.39s/it]
                                                 
{'debug/num_tok_total': 2728.0, 'debug/num_tok_loss': 1636.0, 'debug/num_lat_total': 2728.0, 'debug/num_lat_loss': 1636.0, 'epoch': 5.3}

 53%|█████▎    | 387/730 [34:14<30:48,  5.39s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.35613057017326355, 'epoch': 5.3}

 53%|█████▎    | 387/730 [34:14<30:48,  5.39s/it]
                                                 
{'train/learning_rate_real': 1.1890116541999614e-05, 'epoch': 5.3}

 53%|█████▎    | 387/730 [34:14<30:48,  5.39s/it]
                                                 
{'debug/num_tok_total': 2651.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2651.0, 'debug/num_lat_loss': 1787.0, 'epoch': 5.3}

 53%|█████▎    | 387/730 [34:16<30:48,  5.39s/it]
                                                 
{'train/ce_loss': 1.90625, 'train/diffusion_loss': 0.3847382962703705, 'epoch': 5.3}

 53%|█████▎    | 387/730 [34:16<30:48,  5.39s/it]
                                                 
{'train/learning_rate_real': 1.1890116541999614e-05, 'epoch': 5.3}

 53%|█████▎    | 387/730 [34:16<30:48,  5.39s/it]
                                                 
{'debug/num_tok_total': 2656.0, 'debug/num_tok_loss': 1684.0, 'debug/num_lat_total': 2656.0, 'debug/num_lat_loss': 1684.0, 'epoch': 5.3}

 53%|█████▎    | 387/730 [34:17<30:48,  5.39s/it]
                                                 
{'train/ce_loss': 2.015625, 'train/diffusion_loss': 0.3559465706348419, 'epoch': 5.3}

 53%|█████▎    | 387/730 [34:17<30:48,  5.39s/it]
                                                 
{'train/learning_rate_real': 1.1890116541999614e-05, 'epoch': 5.3}

 53%|█████▎    | 387/730 [34:17<30:48,  5.39s/it]
 53%|█████▎    | 388/730 [34:17<30:46,  5.40s/it]
                                                 
{'debug/num_tok_total': 2295.0, 'debug/num_tok_loss': 1714.0, 'debug/num_lat_total': 2295.0, 'debug/num_lat_loss': 1714.0, 'epoch': 5.32}

 53%|█████▎    | 388/730 [34:18<30:46,  5.40s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.47427451610565186, 'epoch': 5.32}

 53%|█████▎    | 388/730 [34:18<30:46,  5.40s/it]
                                                 
{'train/learning_rate_real': 1.1834722814472771e-05, 'epoch': 5.32}

 53%|█████▎    | 388/730 [34:18<30:46,  5.40s/it]
                                                 
{'debug/num_tok_total': 2667.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2667.0, 'debug/num_lat_loss': 1794.0, 'epoch': 5.32}

 53%|█████▎    | 388/730 [34:20<30:46,  5.40s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.3911558985710144, 'epoch': 5.32}

 53%|█████▎    | 388/730 [34:20<30:46,  5.40s/it]
                                                 
{'train/learning_rate_real': 1.1834722814472771e-05, 'epoch': 5.32}

 53%|█████▎    | 388/730 [34:20<30:46,  5.40s/it]
                                                 
{'debug/num_tok_total': 2853.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2853.0, 'debug/num_lat_loss': 1774.0, 'epoch': 5.32}

 53%|█████▎    | 388/730 [34:21<30:46,  5.40s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.36137279868125916, 'epoch': 5.32}

 53%|█████▎    | 388/730 [34:21<30:46,  5.40s/it]
                                                 
{'train/learning_rate_real': 1.1834722814472771e-05, 'epoch': 5.32}

 53%|█████▎    | 388/730 [34:21<30:46,  5.40s/it]
                                                 
{'debug/num_tok_total': 2274.0, 'debug/num_tok_loss': 1709.0, 'debug/num_lat_total': 2274.0, 'debug/num_lat_loss': 1709.0, 'epoch': 5.32}

 53%|█████▎    | 388/730 [34:22<30:46,  5.40s/it]
                                                 
{'train/ce_loss': 1.9765625, 'train/diffusion_loss': 0.4615236818790436, 'epoch': 5.32}

 53%|█████▎    | 388/730 [34:22<30:46,  5.40s/it]
                                                 
{'train/learning_rate_real': 1.1834722814472771e-05, 'epoch': 5.32}

 53%|█████▎    | 388/730 [34:22<30:46,  5.40s/it]
 53%|█████▎    | 389/730 [34:23<30:26,  5.36s/it]
                                                 
{'debug/num_tok_total': 2647.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2647.0, 'debug/num_lat_loss': 1784.0, 'epoch': 5.33}

 53%|█████▎    | 389/730 [34:24<30:26,  5.36s/it]
                                                 
{'train/ce_loss': 1.671875, 'train/diffusion_loss': 0.4180532693862915, 'epoch': 5.33}

 53%|█████▎    | 389/730 [34:24<30:26,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.1779342185855608e-05, 'epoch': 5.33}

 53%|█████▎    | 389/730 [34:24<30:26,  5.36s/it]
                                                 
{'debug/num_tok_total': 2653.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2653.0, 'debug/num_lat_loss': 1788.0, 'epoch': 5.33}

 53%|█████▎    | 389/730 [34:25<30:26,  5.36s/it]
                                                 
{'train/ce_loss': 1.96875, 'train/diffusion_loss': 0.40186429023742676, 'epoch': 5.33}

 53%|█████▎    | 389/730 [34:25<30:26,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.1779342185855608e-05, 'epoch': 5.33}

 53%|█████▎    | 389/730 [34:25<30:26,  5.36s/it]
                                                 
{'debug/num_tok_total': 1970.0, 'debug/num_tok_loss': 1760.0, 'debug/num_lat_total': 1970.0, 'debug/num_lat_loss': 1760.0, 'epoch': 5.33}

 53%|█████▎    | 389/730 [34:26<30:26,  5.36s/it]
                                                 
{'train/ce_loss': 1.90625, 'train/diffusion_loss': 0.5144637823104858, 'epoch': 5.33}

 53%|█████▎    | 389/730 [34:26<30:26,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.1779342185855608e-05, 'epoch': 5.33}

 53%|█████▎    | 389/730 [34:26<30:26,  5.36s/it]
                                                 
{'debug/num_tok_total': 2294.0, 'debug/num_tok_loss': 1606.0, 'debug/num_lat_total': 2294.0, 'debug/num_lat_loss': 1606.0, 'epoch': 5.33}

 53%|█████▎    | 389/730 [34:27<30:26,  5.36s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.4028507173061371, 'epoch': 5.33}

 53%|█████▎    | 389/730 [34:27<30:26,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.1779342185855608e-05, 'epoch': 5.33}

 53%|█████▎    | 389/730 [34:27<30:26,  5.36s/it]03/16/2026 07:22:00 - INFO - __main__ - LoRA debug step 390: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 53%|█████▎    | 390/730 [34:28<30:05,  5.31s/it]
                                                 
{'loss': 2.4553, 'grad_norm': 1.2754102945327759, 'learning_rate': 1.1779342185855608e-05, 'epoch': 5.34}

 53%|█████▎    | 390/730 [34:28<30:05,  5.31s/it]
                                                 
{'debug/num_tok_total': 2866.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2866.0, 'debug/num_lat_loss': 1786.0, 'epoch': 5.34}

 53%|█████▎    | 390/730 [34:29<30:05,  5.31s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.35515695810317993, 'epoch': 5.34}

 53%|█████▎    | 390/730 [34:29<30:05,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.1723975746559576e-05, 'epoch': 5.34}

 53%|█████▎    | 390/730 [34:29<30:05,  5.31s/it]
                                                 
{'debug/num_tok_total': 2440.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2440.0, 'debug/num_lat_loss': 1770.0, 'epoch': 5.34}

 53%|█████▎    | 390/730 [34:30<30:05,  5.31s/it]
                                                 
{'train/ce_loss': 2.015625, 'train/diffusion_loss': 0.4256676137447357, 'epoch': 5.34}

 53%|█████▎    | 390/730 [34:30<30:05,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.1723975746559576e-05, 'epoch': 5.34}

 53%|█████▎    | 390/730 [34:30<30:05,  5.31s/it]
                                                 
{'debug/num_tok_total': 2766.0, 'debug/num_tok_loss': 1736.0, 'debug/num_lat_total': 2766.0, 'debug/num_lat_loss': 1736.0, 'epoch': 5.34}

 53%|█████▎    | 390/730 [34:31<30:05,  5.31s/it]
                                                 
{'train/ce_loss': 1.9453125, 'train/diffusion_loss': 0.3603058457374573, 'epoch': 5.34}

 53%|█████▎    | 390/730 [34:31<30:05,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.1723975746559576e-05, 'epoch': 5.34}

 53%|█████▎    | 390/730 [34:31<30:05,  5.31s/it]
                                                 
{'debug/num_tok_total': 2494.0, 'debug/num_tok_loss': 1552.0, 'debug/num_lat_total': 2494.0, 'debug/num_lat_loss': 1552.0, 'epoch': 5.34}

 53%|█████▎    | 390/730 [34:33<30:05,  5.31s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.3582332134246826, 'epoch': 5.34}

 53%|█████▎    | 390/730 [34:33<30:05,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.1723975746559576e-05, 'epoch': 5.34}

 53%|█████▎    | 390/730 [34:33<30:05,  5.31s/it]
 54%|█████▎    | 391/730 [34:33<30:22,  5.38s/it]
                                                 
{'debug/num_tok_total': 2636.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2636.0, 'debug/num_lat_loss': 1772.0, 'epoch': 5.36}

 54%|█████▎    | 391/730 [34:34<30:22,  5.38s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.3876456618309021, 'epoch': 5.36}

 54%|█████▎    | 391/730 [34:34<30:22,  5.38s/it]
                                                 
{'train/learning_rate_real': 1.1668624586716723e-05, 'epoch': 5.36}

 54%|█████▎    | 391/730 [34:34<30:22,  5.38s/it]
                                                 
{'debug/num_tok_total': 2652.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2652.0, 'debug/num_lat_loss': 1788.0, 'epoch': 5.36}

 54%|█████▎    | 391/730 [34:36<30:22,  5.38s/it]
                                                 
{'train/ce_loss': 1.875, 'train/diffusion_loss': 0.40279945731163025, 'epoch': 5.36}

 54%|█████▎    | 391/730 [34:36<30:22,  5.38s/it]
                                                 
{'train/learning_rate_real': 1.1668624586716723e-05, 'epoch': 5.36}

 54%|█████▎    | 391/730 [34:36<30:22,  5.38s/it]
                                                 
{'debug/num_tok_total': 2617.0, 'debug/num_tok_loss': 1761.0, 'debug/num_lat_total': 2617.0, 'debug/num_lat_loss': 1761.0, 'epoch': 5.36}

 54%|█████▎    | 391/730 [34:37<30:22,  5.38s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.4103446900844574, 'epoch': 5.36}

 54%|█████▎    | 391/730 [34:37<30:22,  5.38s/it]
                                                 
{'train/learning_rate_real': 1.1668624586716723e-05, 'epoch': 5.36}

 54%|█████▎    | 391/730 [34:37<30:22,  5.38s/it]
                                                 
{'debug/num_tok_total': 1921.0, 'debug/num_tok_loss': 1697.0, 'debug/num_lat_total': 1921.0, 'debug/num_lat_loss': 1697.0, 'epoch': 5.36}

 54%|█████▎    | 391/730 [34:38<30:22,  5.38s/it]
                                                 
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.541977047920227, 'epoch': 5.36}

 54%|█████▎    | 391/730 [34:38<30:22,  5.38s/it]
                                                 
{'train/learning_rate_real': 1.1668624586716723e-05, 'epoch': 5.36}

 54%|█████▎    | 391/730 [34:38<30:22,  5.38s/it]
 54%|█████▎    | 392/730 [34:38<29:44,  5.28s/it]
                                                 
{'debug/num_tok_total': 2620.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2620.0, 'debug/num_lat_loss': 1764.0, 'epoch': 5.37}

 54%|█████▎    | 392/730 [34:39<29:44,  5.28s/it]
                                                 
{'train/ce_loss': 2.171875, 'train/diffusion_loss': 0.3768068253993988, 'epoch': 5.37}

 54%|█████▎    | 392/730 [34:39<29:44,  5.28s/it]
                                                 
{'train/learning_rate_real': 1.1613289796158269e-05, 'epoch': 5.37}

 54%|█████▎    | 392/730 [34:39<29:44,  5.28s/it]
                                                 
{'debug/num_tok_total': 2267.0, 'debug/num_tok_loss': 1596.0, 'debug/num_lat_total': 2267.0, 'debug/num_lat_loss': 1596.0, 'epoch': 5.37}

 54%|█████▎    | 392/730 [34:41<29:44,  5.28s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.3962506353855133, 'epoch': 5.37}

 54%|█████▎    | 392/730 [34:41<29:44,  5.28s/it]
                                                 
{'train/learning_rate_real': 1.1613289796158269e-05, 'epoch': 5.37}

 54%|█████▎    | 392/730 [34:41<29:44,  5.28s/it]
                                                 
{'debug/num_tok_total': 2892.0, 'debug/num_tok_loss': 1813.0, 'debug/num_lat_total': 2892.0, 'debug/num_lat_loss': 1813.0, 'epoch': 5.37}

 54%|█████▎    | 392/730 [34:42<29:44,  5.28s/it]
                                                 
{'train/ce_loss': 2.09375, 'train/diffusion_loss': 0.38966646790504456, 'epoch': 5.37}

 54%|█████▎    | 392/730 [34:42<29:44,  5.28s/it]
                                                 
{'train/learning_rate_real': 1.1613289796158269e-05, 'epoch': 5.37}

 54%|█████▎    | 392/730 [34:42<29:44,  5.28s/it]
                                                 
{'debug/num_tok_total': 2233.0, 'debug/num_tok_loss': 1693.0, 'debug/num_lat_total': 2233.0, 'debug/num_lat_loss': 1693.0, 'epoch': 5.37}

 54%|█████▎    | 392/730 [34:43<29:44,  5.28s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.4384276866912842, 'epoch': 5.37}

 54%|█████▎    | 392/730 [34:43<29:44,  5.28s/it]
                                                 
{'train/learning_rate_real': 1.1613289796158269e-05, 'epoch': 5.37}

 54%|█████▎    | 392/730 [34:43<29:44,  5.28s/it]
 54%|█████▍    | 393/730 [34:43<29:31,  5.26s/it]
                                                 
{'debug/num_tok_total': 2212.0, 'debug/num_tok_loss': 1622.0, 'debug/num_lat_total': 2212.0, 'debug/num_lat_loss': 1622.0, 'epoch': 5.38}

 54%|█████▍    | 393/730 [34:45<29:31,  5.26s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.4339134395122528, 'epoch': 5.38}

 54%|█████▍    | 393/730 [34:45<29:31,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.155797246439313e-05, 'epoch': 5.38}

 54%|█████▍    | 393/730 [34:45<29:31,  5.26s/it]
                                                 
{'debug/num_tok_total': 2844.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2844.0, 'debug/num_lat_loss': 1773.0, 'epoch': 5.38}

 54%|█████▍    | 393/730 [34:46<29:31,  5.26s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.32827895879745483, 'epoch': 5.38}

 54%|█████▍    | 393/730 [34:46<29:31,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.155797246439313e-05, 'epoch': 5.38}

 54%|█████▍    | 393/730 [34:46<29:31,  5.26s/it]
                                                 
{'debug/num_tok_total': 2872.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2872.0, 'debug/num_lat_loss': 1796.0, 'epoch': 5.38}

 54%|█████▍    | 393/730 [34:47<29:31,  5.26s/it]
                                                 
{'train/ce_loss': 1.9609375, 'train/diffusion_loss': 0.3501279056072235, 'epoch': 5.38}

 54%|█████▍    | 393/730 [34:47<29:31,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.155797246439313e-05, 'epoch': 5.38}

 54%|█████▍    | 393/730 [34:47<29:31,  5.26s/it]
                                                 
{'debug/num_tok_total': 2403.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 2403.0, 'debug/num_lat_loss': 1769.0, 'epoch': 5.38}

 54%|█████▍    | 393/730 [34:48<29:31,  5.26s/it]
                                                 
{'train/ce_loss': 1.9453125, 'train/diffusion_loss': 0.4598109722137451, 'epoch': 5.38}

 54%|█████▍    | 393/730 [34:48<29:31,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.155797246439313e-05, 'epoch': 5.38}

 54%|█████▍    | 393/730 [34:48<29:31,  5.26s/it]
 54%|█████▍    | 394/730 [34:49<29:39,  5.30s/it]
                                                 
{'debug/num_tok_total': 2820.0, 'debug/num_tok_loss': 1746.0, 'debug/num_lat_total': 2820.0, 'debug/num_lat_loss': 1746.0, 'epoch': 5.4}

 54%|█████▍    | 394/730 [34:50<29:39,  5.30s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.3456258177757263, 'epoch': 5.4}

 54%|█████▍    | 394/730 [34:50<29:39,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.150267368058646e-05, 'epoch': 5.4}

 54%|█████▍    | 394/730 [34:50<29:39,  5.30s/it]
                                                 
{'debug/num_tok_total': 2656.0, 'debug/num_tok_loss': 1798.0, 'debug/num_lat_total': 2656.0, 'debug/num_lat_loss': 1798.0, 'epoch': 5.4}

 54%|█████▍    | 394/730 [34:51<29:39,  5.30s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.43467411398887634, 'epoch': 5.4}

 54%|█████▍    | 394/730 [34:51<29:39,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.150267368058646e-05, 'epoch': 5.4}

 54%|█████▍    | 394/730 [34:51<29:39,  5.30s/it]
                                                 
{'debug/num_tok_total': 2838.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 2838.0, 'debug/num_lat_loss': 1769.0, 'epoch': 5.4}

 54%|█████▍    | 394/730 [34:53<29:39,  5.30s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.365997850894928, 'epoch': 5.4}

 54%|█████▍    | 394/730 [34:53<29:39,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.150267368058646e-05, 'epoch': 5.4}

 54%|█████▍    | 394/730 [34:53<29:39,  5.30s/it]
                                                 
{'debug/num_tok_total': 2659.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 2659.0, 'debug/num_lat_loss': 1801.0, 'epoch': 5.4}

 54%|█████▍    | 394/730 [34:54<29:39,  5.30s/it]
                                                 
{'train/ce_loss': 1.953125, 'train/diffusion_loss': 0.39845091104507446, 'epoch': 5.4}

 54%|█████▍    | 394/730 [34:54<29:39,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.150267368058646e-05, 'epoch': 5.4}

 54%|█████▍    | 394/730 [34:54<29:39,  5.30s/it]
 54%|█████▍    | 395/730 [34:55<30:06,  5.39s/it]
                                                 
{'debug/num_tok_total': 3332.0, 'debug/num_tok_loss': 1806.0, 'debug/num_lat_total': 3332.0, 'debug/num_lat_loss': 1806.0, 'epoch': 5.41}

 54%|█████▍    | 395/730 [34:56<30:06,  5.39s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.28146234154701233, 'epoch': 5.41}

 54%|█████▍    | 395/730 [34:56<30:06,  5.39s/it]
                                                 
{'train/learning_rate_real': 1.1447394533538234e-05, 'epoch': 5.41}

 54%|█████▍    | 395/730 [34:56<30:06,  5.39s/it]
                                                 
{'debug/num_tok_total': 2464.0, 'debug/num_tok_loss': 1807.0, 'debug/num_lat_total': 2464.0, 'debug/num_lat_loss': 1807.0, 'epoch': 5.41}

 54%|█████▍    | 395/730 [34:57<30:06,  5.39s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.45914003252983093, 'epoch': 5.41}

 54%|█████▍    | 395/730 [34:57<30:06,  5.39s/it]
                                                 
{'train/learning_rate_real': 1.1447394533538234e-05, 'epoch': 5.41}

 54%|█████▍    | 395/730 [34:57<30:06,  5.39s/it]
                                                 
{'debug/num_tok_total': 2831.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 2831.0, 'debug/num_lat_loss': 1767.0, 'epoch': 5.41}

 54%|█████▍    | 395/730 [34:58<30:06,  5.39s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.34179848432540894, 'epoch': 5.41}

 54%|█████▍    | 395/730 [34:58<30:06,  5.39s/it]
                                                 
{'train/learning_rate_real': 1.1447394533538234e-05, 'epoch': 5.41}

 54%|█████▍    | 395/730 [34:58<30:06,  5.39s/it]
                                                 
{'debug/num_tok_total': 2622.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2622.0, 'debug/num_lat_loss': 1780.0, 'epoch': 5.41}

 54%|█████▍    | 395/730 [35:00<30:06,  5.39s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.39029791951179504, 'epoch': 5.41}

 54%|█████▍    | 395/730 [35:00<30:06,  5.39s/it]
                                                 
{'train/learning_rate_real': 1.1447394533538234e-05, 'epoch': 5.41}

 54%|█████▍    | 395/730 [35:00<30:06,  5.39s/it]
 54%|█████▍    | 396/730 [35:00<30:17,  5.44s/it]
                                                 
{'debug/num_tok_total': 2038.0, 'debug/num_tok_loss': 1815.0, 'debug/num_lat_total': 2038.0, 'debug/num_lat_loss': 1815.0, 'epoch': 5.42}

 54%|█████▍    | 396/730 [35:01<30:17,  5.44s/it]
                                                 
{'train/ce_loss': 1.9375, 'train/diffusion_loss': 0.5618933439254761, 'epoch': 5.42}

 54%|█████▍    | 396/730 [35:01<30:17,  5.44s/it]
                                                 
{'train/learning_rate_real': 1.1392136111661766e-05, 'epoch': 5.42}

 54%|█████▍    | 396/730 [35:01<30:17,  5.44s/it]
                                                 
{'debug/num_tok_total': 2883.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2883.0, 'debug/num_lat_loss': 1799.0, 'epoch': 5.42}

 54%|█████▍    | 396/730 [35:02<30:17,  5.44s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.36794179677963257, 'epoch': 5.42}

 54%|█████▍    | 396/730 [35:02<30:17,  5.44s/it]
                                                 
{'train/learning_rate_real': 1.1392136111661766e-05, 'epoch': 5.42}

 54%|█████▍    | 396/730 [35:02<30:17,  5.44s/it]
                                                 
{'debug/num_tok_total': 2683.0, 'debug/num_tok_loss': 1811.0, 'debug/num_lat_total': 2683.0, 'debug/num_lat_loss': 1811.0, 'epoch': 5.42}

 54%|█████▍    | 396/730 [35:04<30:17,  5.44s/it]
                                                 
{'train/ce_loss': 1.9140625, 'train/diffusion_loss': 0.3997327983379364, 'epoch': 5.42}

 54%|█████▍    | 396/730 [35:04<30:17,  5.44s/it]
                                                 
{'train/learning_rate_real': 1.1392136111661766e-05, 'epoch': 5.42}

 54%|█████▍    | 396/730 [35:04<30:17,  5.44s/it]
                                                 
{'debug/num_tok_total': 2890.0, 'debug/num_tok_loss': 1804.0, 'debug/num_lat_total': 2890.0, 'debug/num_lat_loss': 1804.0, 'epoch': 5.42}

 54%|█████▍    | 396/730 [35:05<30:17,  5.44s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.34589481353759766, 'epoch': 5.42}

 54%|█████▍    | 396/730 [35:05<30:17,  5.44s/it]
                                                 
{'train/learning_rate_real': 1.1392136111661766e-05, 'epoch': 5.42}

 54%|█████▍    | 396/730 [35:05<30:17,  5.44s/it]
 54%|█████▍    | 397/730 [35:05<29:58,  5.40s/it]
                                                 
{'debug/num_tok_total': 2631.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2631.0, 'debug/num_lat_loss': 1780.0, 'epoch': 5.44}

 54%|█████▍    | 397/730 [35:07<29:58,  5.40s/it]
                                                 
{'train/ce_loss': 1.953125, 'train/diffusion_loss': 0.40771564841270447, 'epoch': 5.44}

 54%|█████▍    | 397/730 [35:07<29:58,  5.40s/it]
                                                 
{'train/learning_rate_real': 1.133689950296232e-05, 'epoch': 5.44}

 54%|█████▍    | 397/730 [35:07<29:58,  5.40s/it]
                                                 
{'debug/num_tok_total': 2290.0, 'debug/num_tok_loss': 1708.0, 'debug/num_lat_total': 2290.0, 'debug/num_lat_loss': 1708.0, 'epoch': 5.44}

 54%|█████▍    | 397/730 [35:08<29:58,  5.40s/it]
                                                 
{'train/ce_loss': 1.875, 'train/diffusion_loss': 0.4897439777851105, 'epoch': 5.44}

 54%|█████▍    | 397/730 [35:08<29:58,  5.40s/it]
                                                 
{'train/learning_rate_real': 1.133689950296232e-05, 'epoch': 5.44}

 54%|█████▍    | 397/730 [35:08<29:58,  5.40s/it]
                                                 
{'debug/num_tok_total': 3070.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 3070.0, 'debug/num_lat_loss': 1787.0, 'epoch': 5.44}

 54%|█████▍    | 397/730 [35:09<29:58,  5.40s/it]
                                                 
{'train/ce_loss': 1.9296875, 'train/diffusion_loss': 0.3240067660808563, 'epoch': 5.44}

 54%|█████▍    | 397/730 [35:09<29:58,  5.40s/it]
                                                 
{'train/learning_rate_real': 1.133689950296232e-05, 'epoch': 5.44}

 54%|█████▍    | 397/730 [35:09<29:58,  5.40s/it]
                                                 
{'debug/num_tok_total': 2659.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2659.0, 'debug/num_lat_loss': 1788.0, 'epoch': 5.44}

 54%|█████▍    | 397/730 [35:10<29:58,  5.40s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.43447256088256836, 'epoch': 5.44}

 54%|█████▍    | 397/730 [35:10<29:58,  5.40s/it]
                                                 
{'train/learning_rate_real': 1.133689950296232e-05, 'epoch': 5.44}

 54%|█████▍    | 397/730 [35:10<29:58,  5.40s/it]
 55%|█████▍    | 398/730 [35:11<29:55,  5.41s/it]
                                                 
{'debug/num_tok_total': 2654.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2654.0, 'debug/num_lat_loss': 1802.0, 'epoch': 5.45}

 55%|█████▍    | 398/730 [35:12<29:55,  5.41s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.41866445541381836, 'epoch': 5.45}

 55%|█████▍    | 398/730 [35:12<29:55,  5.41s/it]
                                                 
{'train/learning_rate_real': 1.128168579501567e-05, 'epoch': 5.45}

 55%|█████▍    | 398/730 [35:12<29:55,  5.41s/it]
                                                 
{'debug/num_tok_total': 2206.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2206.0, 'debug/num_lat_loss': 1790.0, 'epoch': 5.45}

 55%|█████▍    | 398/730 [35:13<29:55,  5.41s/it]
                                                 
{'train/ce_loss': 1.8984375, 'train/diffusion_loss': 0.47042030096054077, 'epoch': 5.45}

 55%|█████▍    | 398/730 [35:13<29:55,  5.41s/it]
                                                 
{'train/learning_rate_real': 1.128168579501567e-05, 'epoch': 5.45}

 55%|█████▍    | 398/730 [35:13<29:55,  5.41s/it]
                                                 
{'debug/num_tok_total': 2436.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2436.0, 'debug/num_lat_loss': 1784.0, 'epoch': 5.45}

 55%|█████▍    | 398/730 [35:14<29:55,  5.41s/it]
                                                 
{'train/ce_loss': 1.9765625, 'train/diffusion_loss': 0.44332772493362427, 'epoch': 5.45}

 55%|█████▍    | 398/730 [35:14<29:55,  5.41s/it]
                                                 
{'train/learning_rate_real': 1.128168579501567e-05, 'epoch': 5.45}

 55%|█████▍    | 398/730 [35:14<29:55,  5.41s/it]
                                                 
{'debug/num_tok_total': 2612.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 2612.0, 'debug/num_lat_loss': 1767.0, 'epoch': 5.45}

 55%|█████▍    | 398/730 [35:15<29:55,  5.41s/it]
                                                 
{'train/ce_loss': 1.921875, 'train/diffusion_loss': 0.44071370363235474, 'epoch': 5.45}

 55%|█████▍    | 398/730 [35:15<29:55,  5.41s/it]
                                                 
{'train/learning_rate_real': 1.128168579501567e-05, 'epoch': 5.45}

 55%|█████▍    | 398/730 [35:15<29:55,  5.41s/it]
 55%|█████▍    | 399/730 [35:16<29:20,  5.32s/it]
                                                 
{'debug/num_tok_total': 2898.0, 'debug/num_tok_loss': 1806.0, 'debug/num_lat_total': 2898.0, 'debug/num_lat_loss': 1806.0, 'epoch': 5.47}

 55%|█████▍    | 399/730 [35:17<29:20,  5.32s/it]
                                                 
{'train/ce_loss': 1.90625, 'train/diffusion_loss': 0.3398825526237488, 'epoch': 5.47}

 55%|█████▍    | 399/730 [35:17<29:20,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.122649607494669e-05, 'epoch': 5.47}

 55%|█████▍    | 399/730 [35:17<29:20,  5.32s/it]
                                                 
{'debug/num_tok_total': 2816.0, 'debug/num_tok_loss': 1756.0, 'debug/num_lat_total': 2816.0, 'debug/num_lat_loss': 1756.0, 'epoch': 5.47}

 55%|█████▍    | 399/730 [35:18<29:20,  5.32s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.3638966679573059, 'epoch': 5.47}

 55%|█████▍    | 399/730 [35:18<29:20,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.122649607494669e-05, 'epoch': 5.47}

 55%|█████▍    | 399/730 [35:18<29:20,  5.32s/it]
                                                 
{'debug/num_tok_total': 2657.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2657.0, 'debug/num_lat_loss': 1788.0, 'epoch': 5.47}

 55%|█████▍    | 399/730 [35:20<29:20,  5.32s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.4017951786518097, 'epoch': 5.47}

 55%|█████▍    | 399/730 [35:20<29:20,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.122649607494669e-05, 'epoch': 5.47}

 55%|█████▍    | 399/730 [35:20<29:20,  5.32s/it]
                                                 
{'debug/num_tok_total': 2875.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2875.0, 'debug/num_lat_loss': 1790.0, 'epoch': 5.47}

 55%|█████▍    | 399/730 [35:21<29:20,  5.32s/it]
                                                 
{'train/ce_loss': 1.8984375, 'train/diffusion_loss': 0.3224497437477112, 'epoch': 5.47}

 55%|█████▍    | 399/730 [35:21<29:20,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.122649607494669e-05, 'epoch': 5.47}

 55%|█████▍    | 399/730 [35:21<29:20,  5.32s/it]03/16/2026 07:22:53 - INFO - __main__ - LoRA debug step 400: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 55%|█████▍    | 400/730 [35:21<29:29,  5.36s/it]
                                                 
{'loss': 2.5371, 'grad_norm': 1.3492817878723145, 'learning_rate': 1.122649607494669e-05, 'epoch': 5.48}

 55%|█████▍    | 400/730 [35:21<29:29,  5.36s/it]/home/ubuntu/.local/lib/python3.10/site-packages/peft/utils/save_and_load.py:295: UserWarning: Could not find a config file in  - will assume that the vocabulary was not modified.
  warnings.warn(

                                                 
{'debug/num_tok_total': 2621.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2621.0, 'debug/num_lat_loss': 1764.0, 'epoch': 5.48}

 55%|█████▍    | 400/730 [35:24<29:29,  5.36s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.40925201773643494, 'epoch': 5.48}

 55%|█████▍    | 400/730 [35:24<29:29,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.1171331429407934e-05, 'epoch': 5.48}

 55%|█████▍    | 400/730 [35:24<29:29,  5.36s/it]
                                                 
{'debug/num_tok_total': 2433.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2433.0, 'debug/num_lat_loss': 1783.0, 'epoch': 5.48}

 55%|█████▍    | 400/730 [35:25<29:29,  5.36s/it]
                                                 
{'train/ce_loss': 2.09375, 'train/diffusion_loss': 0.45490315556526184, 'epoch': 5.48}

 55%|█████▍    | 400/730 [35:25<29:29,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.1171331429407934e-05, 'epoch': 5.48}

 55%|█████▍    | 400/730 [35:25<29:29,  5.36s/it]
                                                 
{'debug/num_tok_total': 2635.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2635.0, 'debug/num_lat_loss': 1775.0, 'epoch': 5.48}

 55%|█████▍    | 400/730 [35:26<29:29,  5.36s/it]
                                                 
{'train/ce_loss': 1.9140625, 'train/diffusion_loss': 0.4224199950695038, 'epoch': 5.48}

 55%|█████▍    | 400/730 [35:26<29:29,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.1171331429407934e-05, 'epoch': 5.48}

 55%|█████▍    | 400/730 [35:26<29:29,  5.36s/it]
                                                 
{'debug/num_tok_total': 2399.0, 'debug/num_tok_loss': 1758.0, 'debug/num_lat_total': 2399.0, 'debug/num_lat_loss': 1758.0, 'epoch': 5.48}

 55%|█████▍    | 400/730 [35:28<29:29,  5.36s/it]
                                                 
{'train/ce_loss': 1.96875, 'train/diffusion_loss': 0.4582572877407074, 'epoch': 5.48}

 55%|█████▍    | 400/730 [35:28<29:29,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.1171331429407934e-05, 'epoch': 5.48}

 55%|█████▍    | 400/730 [35:28<29:29,  5.36s/it]
 55%|█████▍    | 401/730 [35:28<31:35,  5.76s/it]
                                                 
{'debug/num_tok_total': 2405.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2405.0, 'debug/num_lat_loss': 1763.0, 'epoch': 5.49}

 55%|█████▍    | 401/730 [35:29<31:35,  5.76s/it]
                                                 
{'train/ce_loss': 1.8515625, 'train/diffusion_loss': 0.4656890034675598, 'epoch': 5.49}

 55%|█████▍    | 401/730 [35:29<31:35,  5.76s/it]
                                                 
{'train/learning_rate_real': 1.111619294455827e-05, 'epoch': 5.49}

 55%|█████▍    | 401/730 [35:29<31:35,  5.76s/it]
                                                 
{'debug/num_tok_total': 2256.0, 'debug/num_tok_loss': 1812.0, 'debug/num_lat_total': 2256.0, 'debug/num_lat_loss': 1812.0, 'epoch': 5.49}

 55%|█████▍    | 401/730 [35:30<31:35,  5.76s/it]
                                                 
{'train/ce_loss': 2.53125, 'train/diffusion_loss': 0.49542978405952454, 'epoch': 5.49}

 55%|█████▍    | 401/730 [35:30<31:35,  5.76s/it]
                                                 
{'train/learning_rate_real': 1.111619294455827e-05, 'epoch': 5.49}

 55%|█████▍    | 401/730 [35:30<31:35,  5.76s/it]
                                                 
{'debug/num_tok_total': 2453.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2453.0, 'debug/num_lat_loss': 1802.0, 'epoch': 5.49}

 55%|█████▍    | 401/730 [35:31<31:35,  5.76s/it]
                                                 
{'train/ce_loss': 1.8984375, 'train/diffusion_loss': 0.4494922459125519, 'epoch': 5.49}

 55%|█████▍    | 401/730 [35:31<31:35,  5.76s/it]
                                                 
{'train/learning_rate_real': 1.111619294455827e-05, 'epoch': 5.49}

 55%|█████▍    | 401/730 [35:31<31:35,  5.76s/it]
                                                 
{'debug/num_tok_total': 2447.0, 'debug/num_tok_loss': 1800.0, 'debug/num_lat_total': 2447.0, 'debug/num_lat_loss': 1800.0, 'epoch': 5.49}

 55%|█████▍    | 401/730 [35:33<31:35,  5.76s/it]
                                                 
{'train/ce_loss': 1.8984375, 'train/diffusion_loss': 0.463359534740448, 'epoch': 5.49}

 55%|█████▍    | 401/730 [35:33<31:35,  5.76s/it]
                                                 
{'train/learning_rate_real': 1.111619294455827e-05, 'epoch': 5.49}

 55%|█████▍    | 401/730 [35:33<31:35,  5.76s/it]
 55%|█████▌    | 402/730 [35:33<30:11,  5.52s/it]
                                                 
{'debug/num_tok_total': 2218.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2218.0, 'debug/num_lat_loss': 1777.0, 'epoch': 5.51}

 55%|█████▌    | 402/730 [35:34<30:11,  5.52s/it]
                                                 
{'train/ce_loss': 1.9375, 'train/diffusion_loss': 0.4930035173892975, 'epoch': 5.51}

 55%|█████▌    | 402/730 [35:34<30:11,  5.52s/it]
                                                 
{'train/learning_rate_real': 1.1061081706041463e-05, 'epoch': 5.51}

 55%|█████▌    | 402/730 [35:34<30:11,  5.52s/it]
                                                 
{'debug/num_tok_total': 2627.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2627.0, 'debug/num_lat_loss': 1777.0, 'epoch': 5.51}

 55%|█████▌    | 402/730 [35:35<30:11,  5.52s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.37381109595298767, 'epoch': 5.51}

 55%|█████▌    | 402/730 [35:35<30:11,  5.52s/it]
                                                 
{'train/learning_rate_real': 1.1061081706041463e-05, 'epoch': 5.51}

 55%|█████▌    | 402/730 [35:35<30:11,  5.52s/it]
                                                 
{'debug/num_tok_total': 2177.0, 'debug/num_tok_loss': 1752.0, 'debug/num_lat_total': 2177.0, 'debug/num_lat_loss': 1752.0, 'epoch': 5.51}

 55%|█████▌    | 402/730 [35:37<30:11,  5.52s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.4848552346229553, 'epoch': 5.51}

 55%|█████▌    | 402/730 [35:37<30:11,  5.52s/it]
                                                 
{'train/learning_rate_real': 1.1061081706041463e-05, 'epoch': 5.51}

 55%|█████▌    | 402/730 [35:37<30:11,  5.52s/it]
                                                 
{'debug/num_tok_total': 2536.0, 'debug/num_tok_loss': 1680.0, 'debug/num_lat_total': 2536.0, 'debug/num_lat_loss': 1680.0, 'epoch': 5.51}

 55%|█████▌    | 402/730 [35:38<30:11,  5.52s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.3972345292568207, 'epoch': 5.51}

 55%|█████▌    | 402/730 [35:38<30:11,  5.52s/it]
                                                 
{'train/learning_rate_real': 1.1061081706041463e-05, 'epoch': 5.51}

 55%|█████▌    | 402/730 [35:38<30:11,  5.52s/it]
 55%|█████▌    | 403/730 [35:38<29:31,  5.42s/it]
                                                 
{'debug/num_tok_total': 2685.0, 'debug/num_tok_loss': 1814.0, 'debug/num_lat_total': 2685.0, 'debug/num_lat_loss': 1814.0, 'epoch': 5.52}

 55%|█████▌    | 403/730 [35:39<29:31,  5.42s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.39466866850852966, 'epoch': 5.52}

 55%|█████▌    | 403/730 [35:39<29:31,  5.42s/it]
                                                 
{'train/learning_rate_real': 1.100599879896483e-05, 'epoch': 5.52}

 55%|█████▌    | 403/730 [35:39<29:31,  5.42s/it]
                                                 
{'debug/num_tok_total': 2626.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2626.0, 'debug/num_lat_loss': 1764.0, 'epoch': 5.52}

 55%|█████▌    | 403/730 [35:41<29:31,  5.42s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.42149344086647034, 'epoch': 5.52}

 55%|█████▌    | 403/730 [35:41<29:31,  5.42s/it]
                                                 
{'train/learning_rate_real': 1.100599879896483e-05, 'epoch': 5.52}

 55%|█████▌    | 403/730 [35:41<29:31,  5.42s/it]
                                                 
{'debug/num_tok_total': 2658.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2658.0, 'debug/num_lat_loss': 1794.0, 'epoch': 5.52}

 55%|█████▌    | 403/730 [35:42<29:31,  5.42s/it]
                                                 
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.4101671576499939, 'epoch': 5.52}

 55%|█████▌    | 403/730 [35:42<29:31,  5.42s/it]
                                                 
{'train/learning_rate_real': 1.100599879896483e-05, 'epoch': 5.52}

 55%|█████▌    | 403/730 [35:42<29:31,  5.42s/it]
                                                 
{'debug/num_tok_total': 2395.0, 'debug/num_tok_loss': 1757.0, 'debug/num_lat_total': 2395.0, 'debug/num_lat_loss': 1757.0, 'epoch': 5.52}

 55%|█████▌    | 403/730 [35:43<29:31,  5.42s/it]
                                                 
{'train/ce_loss': 1.9453125, 'train/diffusion_loss': 0.45404744148254395, 'epoch': 5.52}

 55%|█████▌    | 403/730 [35:43<29:31,  5.42s/it]
                                                 
{'train/learning_rate_real': 1.100599879896483e-05, 'epoch': 5.52}

 55%|█████▌    | 403/730 [35:43<29:31,  5.42s/it]
 55%|█████▌    | 404/730 [35:43<29:08,  5.36s/it]
                                                 
{'debug/num_tok_total': 3121.0, 'debug/num_tok_loss': 1808.0, 'debug/num_lat_total': 3121.0, 'debug/num_lat_loss': 1808.0, 'epoch': 5.53}

 55%|█████▌    | 404/730 [35:45<29:08,  5.36s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.3312734365463257, 'epoch': 5.53}

 55%|█████▌    | 404/730 [35:45<29:08,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.0950945307877847e-05, 'epoch': 5.53}

 55%|█████▌    | 404/730 [35:45<29:08,  5.36s/it]
                                                 
{'debug/num_tok_total': 2857.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2857.0, 'debug/num_lat_loss': 1787.0, 'epoch': 5.53}

 55%|█████▌    | 404/730 [35:46<29:08,  5.36s/it]
                                                 
{'train/ce_loss': 1.8984375, 'train/diffusion_loss': 0.36310604214668274, 'epoch': 5.53}

 55%|█████▌    | 404/730 [35:46<29:08,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.0950945307877847e-05, 'epoch': 5.53}

 55%|█████▌    | 404/730 [35:46<29:08,  5.36s/it]
                                                 
{'debug/num_tok_total': 2387.0, 'debug/num_tok_loss': 1749.0, 'debug/num_lat_total': 2387.0, 'debug/num_lat_loss': 1749.0, 'epoch': 5.53}

 55%|█████▌    | 404/730 [35:47<29:08,  5.36s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.44408565759658813, 'epoch': 5.53}

 55%|█████▌    | 404/730 [35:47<29:08,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.0950945307877847e-05, 'epoch': 5.53}

 55%|█████▌    | 404/730 [35:47<29:08,  5.36s/it]
                                                 
{'debug/num_tok_total': 2657.0, 'debug/num_tok_loss': 1584.0, 'debug/num_lat_total': 2657.0, 'debug/num_lat_loss': 1584.0, 'epoch': 5.53}

 55%|█████▌    | 404/730 [35:48<29:08,  5.36s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.3366656005382538, 'epoch': 5.53}

 55%|█████▌    | 404/730 [35:48<29:08,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.0950945307877847e-05, 'epoch': 5.53}

 55%|█████▌    | 404/730 [35:48<29:08,  5.36s/it]
 55%|█████▌    | 405/730 [35:49<29:17,  5.41s/it]
                                                 
{'debug/num_tok_total': 2174.0, 'debug/num_tok_loss': 1754.0, 'debug/num_lat_total': 2174.0, 'debug/num_lat_loss': 1754.0, 'epoch': 5.55}

 55%|█████▌    | 405/730 [35:50<29:17,  5.41s/it]
                                                 
{'train/ce_loss': 1.9375, 'train/diffusion_loss': 0.4885052740573883, 'epoch': 5.55}

 55%|█████▌    | 405/730 [35:50<29:17,  5.41s/it]
                                                 
{'train/learning_rate_real': 1.08959223167508e-05, 'epoch': 5.55}

 55%|█████▌    | 405/730 [35:50<29:17,  5.41s/it]
                                                 
{'debug/num_tok_total': 2598.0, 'debug/num_tok_loss': 1655.0, 'debug/num_lat_total': 2598.0, 'debug/num_lat_loss': 1655.0, 'epoch': 5.55}

 55%|█████▌    | 405/730 [35:51<29:17,  5.41s/it]
                                                 
{'train/ce_loss': 1.9453125, 'train/diffusion_loss': 0.38954296708106995, 'epoch': 5.55}

 55%|█████▌    | 405/730 [35:51<29:17,  5.41s/it]
                                                 
{'train/learning_rate_real': 1.08959223167508e-05, 'epoch': 5.55}

 55%|█████▌    | 405/730 [35:51<29:17,  5.41s/it]
                                                 
{'debug/num_tok_total': 2650.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2650.0, 'debug/num_lat_loss': 1785.0, 'epoch': 5.55}

 55%|█████▌    | 405/730 [35:52<29:17,  5.41s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.3949190080165863, 'epoch': 5.55}

 55%|█████▌    | 405/730 [35:52<29:17,  5.41s/it]
                                                 
{'train/learning_rate_real': 1.08959223167508e-05, 'epoch': 5.55}

 55%|█████▌    | 405/730 [35:52<29:17,  5.41s/it]
                                                 
{'debug/num_tok_total': 2420.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2420.0, 'debug/num_lat_loss': 1771.0, 'epoch': 5.55}

 55%|█████▌    | 405/730 [35:54<29:17,  5.41s/it]
                                                 
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.42888349294662476, 'epoch': 5.55}

 55%|█████▌    | 405/730 [35:54<29:17,  5.41s/it]
                                                 
{'train/learning_rate_real': 1.08959223167508e-05, 'epoch': 5.55}

 55%|█████▌    | 405/730 [35:54<29:17,  5.41s/it]
 56%|█████▌    | 406/730 [35:54<28:46,  5.33s/it]
                                                 
{'debug/num_tok_total': 2642.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2642.0, 'debug/num_lat_loss': 1780.0, 'epoch': 5.56}

 56%|█████▌    | 406/730 [35:55<28:46,  5.33s/it]
                                                 
{'train/ce_loss': 1.96875, 'train/diffusion_loss': 0.3879479467868805, 'epoch': 5.56}

 56%|█████▌    | 406/730 [35:55<28:46,  5.33s/it]
                                                 
{'train/learning_rate_real': 1.0840930908953477e-05, 'epoch': 5.56}

 56%|█████▌    | 406/730 [35:55<28:46,  5.33s/it]
                                                 
{'debug/num_tok_total': 2872.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2872.0, 'debug/num_lat_loss': 1795.0, 'epoch': 5.56}

 56%|█████▌    | 406/730 [35:57<28:46,  5.33s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.37297752499580383, 'epoch': 5.56}

 56%|█████▌    | 406/730 [35:57<28:46,  5.33s/it]
                                                 
{'train/learning_rate_real': 1.0840930908953477e-05, 'epoch': 5.56}

 56%|█████▌    | 406/730 [35:57<28:46,  5.33s/it]
                                                 
{'debug/num_tok_total': 2645.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2645.0, 'debug/num_lat_loss': 1794.0, 'epoch': 5.56}

 56%|█████▌    | 406/730 [35:58<28:46,  5.33s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.40125662088394165, 'epoch': 5.56}

 56%|█████▌    | 406/730 [35:58<28:46,  5.33s/it]
                                                 
{'train/learning_rate_real': 1.0840930908953477e-05, 'epoch': 5.56}

 56%|█████▌    | 406/730 [35:58<28:46,  5.33s/it]
                                                 
{'debug/num_tok_total': 2186.0, 'debug/num_tok_loss': 1742.0, 'debug/num_lat_total': 2186.0, 'debug/num_lat_loss': 1742.0, 'epoch': 5.56}

 56%|█████▌    | 406/730 [35:59<28:46,  5.33s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.49036484956741333, 'epoch': 5.56}

 56%|█████▌    | 406/730 [35:59<28:46,  5.33s/it]
                                                 
{'train/learning_rate_real': 1.0840930908953477e-05, 'epoch': 5.56}

 56%|█████▌    | 406/730 [35:59<28:46,  5.33s/it]
 56%|█████▌    | 407/730 [35:59<28:37,  5.32s/it]
                                                 
{'debug/num_tok_total': 2942.0, 'debug/num_tok_loss': 1664.0, 'debug/num_lat_total': 2942.0, 'debug/num_lat_loss': 1664.0, 'epoch': 5.58}

 56%|█████▌    | 407/730 [36:01<28:37,  5.32s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.3033895194530487, 'epoch': 5.58}

 56%|█████▌    | 407/730 [36:01<28:37,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.0785972167233779e-05, 'epoch': 5.58}

 56%|█████▌    | 407/730 [36:01<28:37,  5.32s/it]
                                                 
{'debug/num_tok_total': 2663.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2663.0, 'debug/num_lat_loss': 1795.0, 'epoch': 5.58}

 56%|█████▌    | 407/730 [36:02<28:37,  5.32s/it]
                                                 
{'train/ce_loss': 1.984375, 'train/diffusion_loss': 0.4089197516441345, 'epoch': 5.58}

 56%|█████▌    | 407/730 [36:02<28:37,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.0785972167233779e-05, 'epoch': 5.58}

 56%|█████▌    | 407/730 [36:02<28:37,  5.32s/it]
                                                 
{'debug/num_tok_total': 2856.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2856.0, 'debug/num_lat_loss': 1780.0, 'epoch': 5.58}

 56%|█████▌    | 407/730 [36:03<28:37,  5.32s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.36866289377212524, 'epoch': 5.58}

 56%|█████▌    | 407/730 [36:03<28:37,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.0785972167233779e-05, 'epoch': 5.58}

 56%|█████▌    | 407/730 [36:03<28:37,  5.32s/it]
                                                 
{'debug/num_tok_total': 2231.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2231.0, 'debug/num_lat_loss': 1791.0, 'epoch': 5.58}

 56%|█████▌    | 407/730 [36:04<28:37,  5.32s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.4723320007324219, 'epoch': 5.58}

 56%|█████▌    | 407/730 [36:04<28:37,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.0785972167233779e-05, 'epoch': 5.58}

 56%|█████▌    | 407/730 [36:04<28:37,  5.32s/it]
 56%|█████▌    | 408/730 [36:05<28:45,  5.36s/it]
                                                 
{'debug/num_tok_total': 2045.0, 'debug/num_tok_loss': 1588.0, 'debug/num_lat_total': 2045.0, 'debug/num_lat_loss': 1588.0, 'epoch': 5.59}

 56%|█████▌    | 408/730 [36:06<28:45,  5.36s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.4886169731616974, 'epoch': 5.59}

 56%|█████▌    | 408/730 [36:06<28:45,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.0731047173696443e-05, 'epoch': 5.59}

 56%|█████▌    | 408/730 [36:06<28:45,  5.36s/it]
                                                 
{'debug/num_tok_total': 3300.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 3300.0, 'debug/num_lat_loss': 1791.0, 'epoch': 5.59}

 56%|█████▌    | 408/730 [36:07<28:45,  5.36s/it]
                                                 
{'train/ce_loss': 2.09375, 'train/diffusion_loss': 0.2729327082633972, 'epoch': 5.59}

 56%|█████▌    | 408/730 [36:07<28:45,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.0731047173696443e-05, 'epoch': 5.59}

 56%|█████▌    | 408/730 [36:07<28:45,  5.36s/it]
                                                 
{'debug/num_tok_total': 2852.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2852.0, 'debug/num_lat_loss': 1789.0, 'epoch': 5.59}

 56%|█████▌    | 408/730 [36:09<28:45,  5.36s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.38415369391441345, 'epoch': 5.59}

 56%|█████▌    | 408/730 [36:09<28:45,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.0731047173696443e-05, 'epoch': 5.59}

 56%|█████▌    | 408/730 [36:09<28:45,  5.36s/it]
                                                 
{'debug/num_tok_total': 2173.0, 'debug/num_tok_loss': 1640.0, 'debug/num_lat_total': 2173.0, 'debug/num_lat_loss': 1640.0, 'epoch': 5.59}

 56%|█████▌    | 408/730 [36:10<28:45,  5.36s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.4712621867656708, 'epoch': 5.59}

 56%|█████▌    | 408/730 [36:10<28:45,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.0731047173696443e-05, 'epoch': 5.59}

 56%|█████▌    | 408/730 [36:10<28:45,  5.36s/it]
 56%|█████▌    | 409/730 [36:10<28:46,  5.38s/it]
                                                 
{'debug/num_tok_total': 2895.0, 'debug/num_tok_loss': 1806.0, 'debug/num_lat_total': 2895.0, 'debug/num_lat_loss': 1806.0, 'epoch': 5.6}

 56%|█████▌    | 409/730 [36:11<28:46,  5.38s/it]
                                                 
{'train/ce_loss': 1.96875, 'train/diffusion_loss': 0.39387190341949463, 'epoch': 5.6}

 56%|█████▌    | 409/730 [36:11<28:46,  5.38s/it]
                                                 
{'train/learning_rate_real': 1.067615700978173e-05, 'epoch': 5.6}

 56%|█████▌    | 409/730 [36:11<28:46,  5.38s/it]
                                                 
{'debug/num_tok_total': 2017.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2017.0, 'debug/num_lat_loss': 1793.0, 'epoch': 5.6}

 56%|█████▌    | 409/730 [36:13<28:46,  5.38s/it]
                                                 
{'train/ce_loss': 1.890625, 'train/diffusion_loss': 0.515536904335022, 'epoch': 5.6}

 56%|█████▌    | 409/730 [36:13<28:46,  5.38s/it]
                                                 
{'train/learning_rate_real': 1.067615700978173e-05, 'epoch': 5.6}

 56%|█████▌    | 409/730 [36:13<28:46,  5.38s/it]
                                                 
{'debug/num_tok_total': 2665.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2665.0, 'debug/num_lat_loss': 1802.0, 'epoch': 5.6}

 56%|█████▌    | 409/730 [36:14<28:46,  5.38s/it]
                                                 
{'train/ce_loss': 1.9921875, 'train/diffusion_loss': 0.4048593044281006, 'epoch': 5.6}

 56%|█████▌    | 409/730 [36:14<28:46,  5.38s/it]
                                                 
{'train/learning_rate_real': 1.067615700978173e-05, 'epoch': 5.6}

 56%|█████▌    | 409/730 [36:14<28:46,  5.38s/it]
                                                 
{'debug/num_tok_total': 2180.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 2180.0, 'debug/num_lat_loss': 1765.0, 'epoch': 5.6}

 56%|█████▌    | 409/730 [36:15<28:46,  5.38s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.46583789587020874, 'epoch': 5.6}

 56%|█████▌    | 409/730 [36:15<28:46,  5.38s/it]
                                                 
{'train/learning_rate_real': 1.067615700978173e-05, 'epoch': 5.6}

 56%|█████▌    | 409/730 [36:15<28:46,  5.38s/it]03/16/2026 07:23:47 - INFO - __main__ - LoRA debug step 410: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 56%|█████▌    | 410/730 [36:15<28:15,  5.30s/it]
                                                 
{'loss': 2.6585, 'grad_norm': 2.299288034439087, 'learning_rate': 1.067615700978173e-05, 'epoch': 5.62}

 56%|█████▌    | 410/730 [36:15<28:15,  5.30s/it]
                                                 
{'debug/num_tok_total': 3094.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 3094.0, 'debug/num_lat_loss': 1790.0, 'epoch': 5.62}

 56%|█████▌    | 410/730 [36:17<28:15,  5.30s/it]
                                                 
{'train/ce_loss': 1.984375, 'train/diffusion_loss': 0.3149610459804535, 'epoch': 5.62}

 56%|█████▌    | 410/730 [36:17<28:15,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.0621302756244124e-05, 'epoch': 5.62}

 56%|█████▌    | 410/730 [36:17<28:15,  5.30s/it]
                                                 
{'debug/num_tok_total': 2864.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2864.0, 'debug/num_lat_loss': 1787.0, 'epoch': 5.62}

 56%|█████▌    | 410/730 [36:18<28:15,  5.30s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.3782121539115906, 'epoch': 5.62}

 56%|█████▌    | 410/730 [36:18<28:15,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.0621302756244124e-05, 'epoch': 5.62}

 56%|█████▌    | 410/730 [36:18<28:15,  5.30s/it]
                                                 
{'debug/num_tok_total': 2214.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2214.0, 'debug/num_lat_loss': 1788.0, 'epoch': 5.62}

 56%|█████▌    | 410/730 [36:19<28:15,  5.30s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.5015697479248047, 'epoch': 5.62}

 56%|█████▌    | 410/730 [36:19<28:15,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.0621302756244124e-05, 'epoch': 5.62}

 56%|█████▌    | 410/730 [36:19<28:15,  5.30s/it]
                                                 
{'debug/num_tok_total': 2451.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 2451.0, 'debug/num_lat_loss': 1797.0, 'epoch': 5.62}

 56%|█████▌    | 410/730 [36:20<28:15,  5.30s/it]
                                                 
{'train/ce_loss': 1.6875, 'train/diffusion_loss': 0.44823160767555237, 'epoch': 5.62}

 56%|█████▌    | 410/730 [36:20<28:15,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.0621302756244124e-05, 'epoch': 5.62}

 56%|█████▌    | 410/730 [36:20<28:15,  5.30s/it]
 56%|█████▋    | 411/730 [36:21<28:18,  5.32s/it]
                                                 
{'debug/num_tok_total': 2861.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2861.0, 'debug/num_lat_loss': 1772.0, 'epoch': 5.63}

 56%|█████▋    | 411/730 [36:22<28:18,  5.32s/it]
                                                 
{'train/ce_loss': 1.984375, 'train/diffusion_loss': 0.3828701376914978, 'epoch': 5.63}

 56%|█████▋    | 411/730 [36:22<28:18,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.0566485493131042e-05, 'epoch': 5.63}

 56%|█████▋    | 411/730 [36:22<28:18,  5.32s/it]
                                                 
{'debug/num_tok_total': 3306.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 3306.0, 'debug/num_lat_loss': 1791.0, 'epoch': 5.63}

 56%|█████▋    | 411/730 [36:23<28:18,  5.32s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.29569825530052185, 'epoch': 5.63}

 56%|█████▋    | 411/730 [36:23<28:18,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.0566485493131042e-05, 'epoch': 5.63}

 56%|█████▋    | 411/730 [36:23<28:18,  5.32s/it]
                                                 
{'debug/num_tok_total': 2877.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2877.0, 'debug/num_lat_loss': 1794.0, 'epoch': 5.63}

 56%|█████▋    | 411/730 [36:25<28:18,  5.32s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.36439186334609985, 'epoch': 5.63}

 56%|█████▋    | 411/730 [36:25<28:18,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.0566485493131042e-05, 'epoch': 5.63}

 56%|█████▋    | 411/730 [36:25<28:18,  5.32s/it]
                                                 
{'debug/num_tok_total': 2667.0, 'debug/num_tok_loss': 1690.0, 'debug/num_lat_total': 2667.0, 'debug/num_lat_loss': 1690.0, 'epoch': 5.63}

 56%|█████▋    | 411/730 [36:26<28:18,  5.32s/it]
                                                 
{'train/ce_loss': 1.875, 'train/diffusion_loss': 0.3739621043205261, 'epoch': 5.63}

 56%|█████▋    | 411/730 [36:26<28:18,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.0566485493131042e-05, 'epoch': 5.63}

 56%|█████▋    | 411/730 [36:26<28:18,  5.32s/it]
 56%|█████▋    | 412/730 [36:26<28:40,  5.41s/it]
                                                 
{'debug/num_tok_total': 2886.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2886.0, 'debug/num_lat_loss': 1796.0, 'epoch': 5.64}

 56%|█████▋    | 412/730 [36:28<28:40,  5.41s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.38727277517318726, 'epoch': 5.64}

 56%|█████▋    | 412/730 [36:28<28:40,  5.41s/it]
                                                 
{'train/learning_rate_real': 1.0511706299761602e-05, 'epoch': 5.64}

 56%|█████▋    | 412/730 [36:28<28:40,  5.41s/it]
                                                 
{'debug/num_tok_total': 2417.0, 'debug/num_tok_loss': 1543.0, 'debug/num_lat_total': 2417.0, 'debug/num_lat_loss': 1543.0, 'epoch': 5.64}

 56%|█████▋    | 412/730 [36:29<28:40,  5.41s/it]
                                                 
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.34425994753837585, 'epoch': 5.64}

 56%|█████▋    | 412/730 [36:29<28:40,  5.41s/it]
                                                 
{'train/learning_rate_real': 1.0511706299761602e-05, 'epoch': 5.64}

 56%|█████▋    | 412/730 [36:29<28:40,  5.41s/it]
                                                 
{'debug/num_tok_total': 2907.0, 'debug/num_tok_loss': 1803.0, 'debug/num_lat_total': 2907.0, 'debug/num_lat_loss': 1803.0, 'epoch': 5.64}

 56%|█████▋    | 412/730 [36:30<28:40,  5.41s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.3672166168689728, 'epoch': 5.64}

 56%|█████▋    | 412/730 [36:30<28:40,  5.41s/it]
                                                 
{'train/learning_rate_real': 1.0511706299761602e-05, 'epoch': 5.64}

 56%|█████▋    | 412/730 [36:30<28:40,  5.41s/it]
                                                 
{'debug/num_tok_total': 3199.0, 'debug/num_tok_loss': 1736.0, 'debug/num_lat_total': 3199.0, 'debug/num_lat_loss': 1736.0, 'epoch': 5.64}

 56%|█████▋    | 412/730 [36:32<28:40,  5.41s/it]
                                                 
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.27881699800491333, 'epoch': 5.64}

 56%|█████▋    | 412/730 [36:32<28:40,  5.41s/it]
                                                 
{'train/learning_rate_real': 1.0511706299761602e-05, 'epoch': 5.64}

 56%|█████▋    | 412/730 [36:32<28:40,  5.41s/it]
 57%|█████▋    | 413/730 [36:32<29:00,  5.49s/it]
                                                 
{'debug/num_tok_total': 3317.0, 'debug/num_tok_loss': 1798.0, 'debug/num_lat_total': 3317.0, 'debug/num_lat_loss': 1798.0, 'epoch': 5.66}

 57%|█████▋    | 413/730 [36:33<29:00,  5.49s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.28067412972450256, 'epoch': 5.66}

 57%|█████▋    | 413/730 [36:33<29:00,  5.49s/it]
                                                 
{'train/learning_rate_real': 1.0456966254705337e-05, 'epoch': 5.66}

 57%|█████▋    | 413/730 [36:33<29:00,  5.49s/it]
                                                 
{'debug/num_tok_total': 2728.0, 'debug/num_tok_loss': 1632.0, 'debug/num_lat_total': 2728.0, 'debug/num_lat_loss': 1632.0, 'epoch': 5.66}

 57%|█████▋    | 413/730 [36:35<29:00,  5.49s/it]
                                                 
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.3231348693370819, 'epoch': 5.66}

 57%|█████▋    | 413/730 [36:35<29:00,  5.49s/it]
                                                 
{'train/learning_rate_real': 1.0456966254705337e-05, 'epoch': 5.66}

 57%|█████▋    | 413/730 [36:35<29:00,  5.49s/it]
                                                 
{'debug/num_tok_total': 2597.0, 'debug/num_tok_loss': 1745.0, 'debug/num_lat_total': 2597.0, 'debug/num_lat_loss': 1745.0, 'epoch': 5.66}

 57%|█████▋    | 413/730 [36:36<29:00,  5.49s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.4259370267391205, 'epoch': 5.66}

 57%|█████▋    | 413/730 [36:36<29:00,  5.49s/it]
                                                 
{'train/learning_rate_real': 1.0456966254705337e-05, 'epoch': 5.66}

 57%|█████▋    | 413/730 [36:36<29:00,  5.49s/it]
                                                 
{'debug/num_tok_total': 2638.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2638.0, 'debug/num_lat_loss': 1773.0, 'epoch': 5.66}

 57%|█████▋    | 413/730 [36:37<29:00,  5.49s/it]
                                                 
{'train/ce_loss': 1.921875, 'train/diffusion_loss': 0.3921055495738983, 'epoch': 5.66}

 57%|█████▋    | 413/730 [36:37<29:00,  5.49s/it]
                                                 
{'train/learning_rate_real': 1.0456966254705337e-05, 'epoch': 5.66}

 57%|█████▋    | 413/730 [36:37<29:00,  5.49s/it]
 57%|█████▋    | 414/730 [36:38<29:08,  5.53s/it]
                                                 
{'debug/num_tok_total': 2273.0, 'debug/num_tok_loss': 1650.0, 'debug/num_lat_total': 2273.0, 'debug/num_lat_loss': 1650.0, 'epoch': 5.67}

 57%|█████▋    | 414/730 [36:39<29:08,  5.53s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.419040709733963, 'epoch': 5.67}

 57%|█████▋    | 414/730 [36:39<29:08,  5.53s/it]
                                                 
{'train/learning_rate_real': 1.040226643576098e-05, 'epoch': 5.67}

 57%|█████▋    | 414/730 [36:39<29:08,  5.53s/it]
                                                 
{'debug/num_tok_total': 2145.0, 'debug/num_tok_loss': 1717.0, 'debug/num_lat_total': 2145.0, 'debug/num_lat_loss': 1717.0, 'epoch': 5.67}

 57%|█████▋    | 414/730 [36:40<29:08,  5.53s/it]
                                                 
{'train/ce_loss': 1.9921875, 'train/diffusion_loss': 0.48385387659072876, 'epoch': 5.67}

 57%|█████▋    | 414/730 [36:40<29:08,  5.53s/it]
                                                 
{'train/learning_rate_real': 1.040226643576098e-05, 'epoch': 5.67}

 57%|█████▋    | 414/730 [36:40<29:08,  5.53s/it]
                                                 
{'debug/num_tok_total': 2241.0, 'debug/num_tok_loss': 1811.0, 'debug/num_lat_total': 2241.0, 'debug/num_lat_loss': 1811.0, 'epoch': 5.67}

 57%|█████▋    | 414/730 [36:41<29:08,  5.53s/it]
                                                 
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.4667074680328369, 'epoch': 5.67}

 57%|█████▋    | 414/730 [36:41<29:08,  5.53s/it]
                                                 
{'train/learning_rate_real': 1.040226643576098e-05, 'epoch': 5.67}

 57%|█████▋    | 414/730 [36:41<29:08,  5.53s/it]
                                                 
{'debug/num_tok_total': 2210.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2210.0, 'debug/num_lat_loss': 1772.0, 'epoch': 5.67}

 57%|█████▋    | 414/730 [36:42<29:08,  5.53s/it]
                                                 
{'train/ce_loss': 1.984375, 'train/diffusion_loss': 0.47779127955436707, 'epoch': 5.67}

 57%|█████▋    | 414/730 [36:42<29:08,  5.53s/it]
                                                 
{'train/learning_rate_real': 1.040226643576098e-05, 'epoch': 5.67}

 57%|█████▋    | 414/730 [36:42<29:08,  5.53s/it]
 57%|█████▋    | 415/730 [36:43<28:05,  5.35s/it]
                                                 
{'debug/num_tok_total': 2829.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2829.0, 'debug/num_lat_loss': 1763.0, 'epoch': 5.68}

 57%|█████▋    | 415/730 [36:44<28:05,  5.35s/it]
                                                 
{'train/ce_loss': 1.984375, 'train/diffusion_loss': 0.3481243848800659, 'epoch': 5.68}

 57%|█████▋    | 415/730 [36:44<28:05,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.0347607919935238e-05, 'epoch': 5.68}

 57%|█████▋    | 415/730 [36:44<28:05,  5.35s/it]
                                                 
{'debug/num_tok_total': 2040.0, 'debug/num_tok_loss': 1606.0, 'debug/num_lat_total': 2040.0, 'debug/num_lat_loss': 1606.0, 'epoch': 5.68}

 57%|█████▋    | 415/730 [36:45<28:05,  5.35s/it]
                                                 
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.46735480427742004, 'epoch': 5.68}

 57%|█████▋    | 415/730 [36:45<28:05,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.0347607919935238e-05, 'epoch': 5.68}

 57%|█████▋    | 415/730 [36:45<28:05,  5.35s/it]
                                                 
{'debug/num_tok_total': 2873.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2873.0, 'debug/num_lat_loss': 1789.0, 'epoch': 5.68}

 57%|█████▋    | 415/730 [36:46<28:05,  5.35s/it]
                                                 
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.3558003604412079, 'epoch': 5.68}

 57%|█████▋    | 415/730 [36:46<28:05,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.0347607919935238e-05, 'epoch': 5.68}

 57%|█████▋    | 415/730 [36:46<28:05,  5.35s/it]
                                                 
{'debug/num_tok_total': 2433.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2433.0, 'debug/num_lat_loss': 1785.0, 'epoch': 5.68}

 57%|█████▋    | 415/730 [36:47<28:05,  5.35s/it]
                                                 
{'train/ce_loss': 1.921875, 'train/diffusion_loss': 0.4475365877151489, 'epoch': 5.68}

 57%|█████▋    | 415/730 [36:47<28:05,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.0347607919935238e-05, 'epoch': 5.68}

 57%|█████▋    | 415/730 [36:47<28:05,  5.35s/it]
 57%|█████▋    | 416/730 [36:48<27:51,  5.32s/it]
                                                 
{'debug/num_tok_total': 2872.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 2872.0, 'debug/num_lat_loss': 1797.0, 'epoch': 5.7}

 57%|█████▋    | 416/730 [36:49<27:51,  5.32s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.3516913056373596, 'epoch': 5.7}

 57%|█████▋    | 416/730 [36:49<27:51,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.0292991783421572e-05, 'epoch': 5.7}

 57%|█████▋    | 416/730 [36:49<27:51,  5.32s/it]
                                                 
{'debug/num_tok_total': 2879.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2879.0, 'debug/num_lat_loss': 1796.0, 'epoch': 5.7}

 57%|█████▋    | 416/730 [36:50<27:51,  5.32s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.35891106724739075, 'epoch': 5.7}

 57%|█████▋    | 416/730 [36:50<27:51,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.0292991783421572e-05, 'epoch': 5.7}

 57%|█████▋    | 416/730 [36:50<27:51,  5.32s/it]
                                                 
{'debug/num_tok_total': 2015.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2015.0, 'debug/num_lat_loss': 1793.0, 'epoch': 5.7}

 57%|█████▋    | 416/730 [36:51<27:51,  5.32s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.5592963099479675, 'epoch': 5.7}

 57%|█████▋    | 416/730 [36:51<27:51,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.0292991783421572e-05, 'epoch': 5.7}

 57%|█████▋    | 416/730 [36:51<27:51,  5.32s/it]
                                                 
{'debug/num_tok_total': 2617.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2617.0, 'debug/num_lat_loss': 1773.0, 'epoch': 5.7}

 57%|█████▋    | 416/730 [36:53<27:51,  5.32s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.39423584938049316, 'epoch': 5.7}

 57%|█████▋    | 416/730 [36:53<27:51,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.0292991783421572e-05, 'epoch': 5.7}

 57%|█████▋    | 416/730 [36:53<27:51,  5.32s/it]
 57%|█████▋    | 417/730 [36:53<27:40,  5.31s/it]
                                                 
{'debug/num_tok_total': 2638.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2638.0, 'debug/num_lat_loss': 1778.0, 'epoch': 5.71}

 57%|█████▋    | 417/730 [36:54<27:40,  5.31s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.39942896366119385, 'epoch': 5.71}

 57%|█████▋    | 417/730 [36:54<27:40,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.0238419101579034e-05, 'epoch': 5.71}

 57%|█████▋    | 417/730 [36:54<27:40,  5.31s/it]
                                                 
{'debug/num_tok_total': 2681.0, 'debug/num_tok_loss': 1812.0, 'debug/num_lat_total': 2681.0, 'debug/num_lat_loss': 1812.0, 'epoch': 5.71}

 57%|█████▋    | 417/730 [36:55<27:40,  5.31s/it]
                                                 
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.3966110646724701, 'epoch': 5.71}

 57%|█████▋    | 417/730 [36:55<27:40,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.0238419101579034e-05, 'epoch': 5.71}

 57%|█████▋    | 417/730 [36:55<27:40,  5.31s/it]
                                                 
{'debug/num_tok_total': 2607.0, 'debug/num_tok_loss': 1752.0, 'debug/num_lat_total': 2607.0, 'debug/num_lat_loss': 1752.0, 'epoch': 5.71}

 57%|█████▋    | 417/730 [36:57<27:40,  5.31s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.4123537838459015, 'epoch': 5.71}

 57%|█████▋    | 417/730 [36:57<27:40,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.0238419101579034e-05, 'epoch': 5.71}

 57%|█████▋    | 417/730 [36:57<27:40,  5.31s/it]
                                                 
{'debug/num_tok_total': 2666.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2666.0, 'debug/num_lat_loss': 1802.0, 'epoch': 5.71}

 57%|█████▋    | 417/730 [36:58<27:40,  5.31s/it]
                                                 
{'train/ce_loss': 1.875, 'train/diffusion_loss': 0.4007384181022644, 'epoch': 5.71}

 57%|█████▋    | 417/730 [36:58<27:40,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.0238419101579034e-05, 'epoch': 5.71}

 57%|█████▋    | 417/730 [36:58<27:40,  5.31s/it]
 57%|█████▋    | 418/730 [36:58<27:28,  5.28s/it]
                                                 
{'debug/num_tok_total': 2419.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2419.0, 'debug/num_lat_loss': 1773.0, 'epoch': 5.73}

 57%|█████▋    | 418/730 [36:59<27:28,  5.28s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.46635276079177856, 'epoch': 5.73}

 57%|█████▋    | 418/730 [36:59<27:28,  5.28s/it]
                                                 
{'train/learning_rate_real': 1.0183890948911074e-05, 'epoch': 5.73}

 57%|█████▋    | 418/730 [36:59<27:28,  5.28s/it]
                                                 
{'debug/num_tok_total': 2212.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2212.0, 'debug/num_lat_loss': 1783.0, 'epoch': 5.73}

 57%|█████▋    | 418/730 [37:01<27:28,  5.28s/it]
                                                 
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.4827675521373749, 'epoch': 5.73}

 57%|█████▋    | 418/730 [37:01<27:28,  5.28s/it]
                                                 
{'train/learning_rate_real': 1.0183890948911074e-05, 'epoch': 5.73}

 57%|█████▋    | 418/730 [37:01<27:28,  5.28s/it]
                                                 
{'debug/num_tok_total': 2686.0, 'debug/num_tok_loss': 1806.0, 'debug/num_lat_total': 2686.0, 'debug/num_lat_loss': 1806.0, 'epoch': 5.73}

 57%|█████▋    | 418/730 [37:02<27:28,  5.28s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.36070430278778076, 'epoch': 5.73}

 57%|█████▋    | 418/730 [37:02<27:28,  5.28s/it]
                                                 
{'train/learning_rate_real': 1.0183890948911074e-05, 'epoch': 5.73}

 57%|█████▋    | 418/730 [37:02<27:28,  5.28s/it]
                                                 
{'debug/num_tok_total': 2612.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 2612.0, 'debug/num_lat_loss': 1767.0, 'epoch': 5.73}

 57%|█████▋    | 418/730 [37:03<27:28,  5.28s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.4003145098686218, 'epoch': 5.73}

 57%|█████▋    | 418/730 [37:03<27:28,  5.28s/it]
                                                 
{'train/learning_rate_real': 1.0183890948911074e-05, 'epoch': 5.73}

 57%|█████▋    | 418/730 [37:03<27:28,  5.28s/it]
 57%|█████▋    | 419/730 [37:04<27:11,  5.24s/it]
                                                 
{'debug/num_tok_total': 2613.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 2613.0, 'debug/num_lat_loss': 1765.0, 'epoch': 5.74}

 57%|█████▋    | 419/730 [37:05<27:11,  5.24s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.39310944080352783, 'epoch': 5.74}

 57%|█████▋    | 419/730 [37:05<27:11,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.0129408399044382e-05, 'epoch': 5.74}

 57%|█████▋    | 419/730 [37:05<27:11,  5.24s/it]
                                                 
{'debug/num_tok_total': 2743.0, 'debug/num_tok_loss': 1615.0, 'debug/num_lat_total': 2743.0, 'debug/num_lat_loss': 1615.0, 'epoch': 5.74}

 57%|█████▋    | 419/730 [37:06<27:11,  5.24s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.340116024017334, 'epoch': 5.74}

 57%|█████▋    | 419/730 [37:06<27:11,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.0129408399044382e-05, 'epoch': 5.74}

 57%|█████▋    | 419/730 [37:06<27:11,  5.24s/it]
                                                 
{'debug/num_tok_total': 2223.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2223.0, 'debug/num_lat_loss': 1790.0, 'epoch': 5.74}

 57%|█████▋    | 419/730 [37:07<27:11,  5.24s/it]
                                                 
{'train/ce_loss': 1.9140625, 'train/diffusion_loss': 0.49937084317207336, 'epoch': 5.74}

 57%|█████▋    | 419/730 [37:07<27:11,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.0129408399044382e-05, 'epoch': 5.74}

 57%|█████▋    | 419/730 [37:07<27:11,  5.24s/it]
                                                 
{'debug/num_tok_total': 3087.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 3087.0, 'debug/num_lat_loss': 1784.0, 'epoch': 5.74}

 57%|█████▋    | 419/730 [37:08<27:11,  5.24s/it]
                                                 
{'train/ce_loss': 1.9140625, 'train/diffusion_loss': 0.3471311628818512, 'epoch': 5.74}

 57%|█████▋    | 419/730 [37:08<27:11,  5.24s/it]
                                                 
{'train/learning_rate_real': 1.0129408399044382e-05, 'epoch': 5.74}

 57%|█████▋    | 419/730 [37:08<27:11,  5.24s/it]03/16/2026 07:24:41 - INFO - __main__ - LoRA debug step 420: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 58%|█████▊    | 420/730 [37:09<27:12,  5.26s/it]
                                                 
{'loss': 2.5173, 'grad_norm': 1.40345299243927, 'learning_rate': 1.0129408399044382e-05, 'epoch': 5.75}

 58%|█████▊    | 420/730 [37:09<27:12,  5.26s/it]
                                                 
{'debug/num_tok_total': 3086.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 3086.0, 'debug/num_lat_loss': 1791.0, 'epoch': 5.75}

 58%|█████▊    | 420/730 [37:10<27:12,  5.26s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.30306240916252136, 'epoch': 5.75}

 58%|█████▊    | 420/730 [37:10<27:12,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.0074972524707774e-05, 'epoch': 5.75}

 58%|█████▊    | 420/730 [37:10<27:12,  5.26s/it]
                                                 
{'debug/num_tok_total': 2223.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2223.0, 'debug/num_lat_loss': 1784.0, 'epoch': 5.75}

 58%|█████▊    | 420/730 [37:11<27:12,  5.26s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.48005959391593933, 'epoch': 5.75}

 58%|█████▊    | 420/730 [37:11<27:12,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.0074972524707774e-05, 'epoch': 5.75}

 58%|█████▊    | 420/730 [37:11<27:12,  5.26s/it]
                                                 
{'debug/num_tok_total': 2406.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 2406.0, 'debug/num_lat_loss': 1765.0, 'epoch': 5.75}

 58%|█████▊    | 420/730 [37:12<27:12,  5.26s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.4770980179309845, 'epoch': 5.75}

 58%|█████▊    | 420/730 [37:12<27:12,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.0074972524707774e-05, 'epoch': 5.75}

 58%|█████▊    | 420/730 [37:12<27:12,  5.26s/it]
                                                 
{'debug/num_tok_total': 3271.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 3271.0, 'debug/num_lat_loss': 1774.0, 'epoch': 5.75}

 58%|█████▊    | 420/730 [37:14<27:12,  5.26s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.25065702199935913, 'epoch': 5.75}

 58%|█████▊    | 420/730 [37:14<27:12,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.0074972524707774e-05, 'epoch': 5.75}

 58%|█████▊    | 420/730 [37:14<27:12,  5.26s/it]
 58%|█████▊    | 421/730 [37:14<27:14,  5.29s/it]
                                                 
{'debug/num_tok_total': 2469.0, 'debug/num_tok_loss': 1617.0, 'debug/num_lat_total': 2469.0, 'debug/num_lat_loss': 1617.0, 'epoch': 5.77}

 58%|█████▊    | 421/730 [37:15<27:14,  5.29s/it]
                                                 
{'train/ce_loss': 1.6015625, 'train/diffusion_loss': 0.4105099141597748, 'epoch': 5.77}

 58%|█████▊    | 421/730 [37:15<27:14,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.0020584397711027e-05, 'epoch': 5.77}

 58%|█████▊    | 421/730 [37:15<27:14,  5.29s/it]
                                                 
{'debug/num_tok_total': 2429.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2429.0, 'debug/num_lat_loss': 1779.0, 'epoch': 5.77}

 58%|█████▊    | 421/730 [37:16<27:14,  5.29s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.4395619332790375, 'epoch': 5.77}

 58%|█████▊    | 421/730 [37:17<27:14,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.0020584397711027e-05, 'epoch': 5.77}

 58%|█████▊    | 421/730 [37:17<27:14,  5.29s/it]
                                                 
{'debug/num_tok_total': 2394.0, 'debug/num_tok_loss': 1756.0, 'debug/num_lat_total': 2394.0, 'debug/num_lat_loss': 1756.0, 'epoch': 5.77}

 58%|█████▊    | 421/730 [37:18<27:14,  5.29s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.4365805983543396, 'epoch': 5.77}

 58%|█████▊    | 421/730 [37:18<27:14,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.0020584397711027e-05, 'epoch': 5.77}

 58%|█████▊    | 421/730 [37:18<27:14,  5.29s/it]
                                                 
{'debug/num_tok_total': 2151.0, 'debug/num_tok_loss': 1436.0, 'debug/num_lat_total': 2151.0, 'debug/num_lat_loss': 1436.0, 'epoch': 5.77}

 58%|█████▊    | 421/730 [37:19<27:14,  5.29s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.39506635069847107, 'epoch': 5.77}

 58%|█████▊    | 421/730 [37:19<27:14,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.0020584397711027e-05, 'epoch': 5.77}

 58%|█████▊    | 421/730 [37:19<27:14,  5.29s/it]
 58%|█████▊    | 422/730 [37:19<27:00,  5.26s/it]
                                                 
{'debug/num_tok_total': 3092.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 3092.0, 'debug/num_lat_loss': 1784.0, 'epoch': 5.78}

 58%|█████▊    | 422/730 [37:21<27:00,  5.26s/it]
                                                 
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.3334876298904419, 'epoch': 5.78}

 58%|█████▊    | 422/730 [37:21<27:00,  5.26s/it]
                                                 
{'train/learning_rate_real': 9.96624508892383e-06, 'epoch': 5.78}

 58%|█████▊    | 422/730 [37:21<27:00,  5.26s/it]
                                                 
{'debug/num_tok_total': 2447.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2447.0, 'debug/num_lat_loss': 1793.0, 'epoch': 5.78}

 58%|█████▊    | 422/730 [37:22<27:00,  5.26s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.4442253112792969, 'epoch': 5.78}

 58%|█████▊    | 422/730 [37:22<27:00,  5.26s/it]
                                                 
{'train/learning_rate_real': 9.96624508892383e-06, 'epoch': 5.78}

 58%|█████▊    | 422/730 [37:22<27:00,  5.26s/it]
                                                 
{'debug/num_tok_total': 2873.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2873.0, 'debug/num_lat_loss': 1788.0, 'epoch': 5.78}

 58%|█████▊    | 422/730 [37:23<27:00,  5.26s/it]
                                                 
{'train/ce_loss': 2.109375, 'train/diffusion_loss': 0.3480319380760193, 'epoch': 5.78}

 58%|█████▊    | 422/730 [37:23<27:00,  5.26s/it]
                                                 
{'train/learning_rate_real': 9.96624508892383e-06, 'epoch': 5.78}

 58%|█████▊    | 422/730 [37:23<27:00,  5.26s/it]
                                                 
{'debug/num_tok_total': 2713.0, 'debug/num_tok_loss': 1692.0, 'debug/num_lat_total': 2713.0, 'debug/num_lat_loss': 1692.0, 'epoch': 5.78}

 58%|█████▊    | 422/730 [37:24<27:00,  5.26s/it]
                                                 
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.34820449352264404, 'epoch': 5.78}

 58%|█████▊    | 422/730 [37:24<27:00,  5.26s/it]
                                                 
{'train/learning_rate_real': 9.96624508892383e-06, 'epoch': 5.78}

 58%|█████▊    | 422/730 [37:24<27:00,  5.26s/it]
 58%|█████▊    | 423/730 [37:25<27:18,  5.34s/it]
                                                 
{'debug/num_tok_total': 2473.0, 'debug/num_tok_loss': 1820.0, 'debug/num_lat_total': 2473.0, 'debug/num_lat_loss': 1820.0, 'epoch': 5.79}

 58%|█████▊    | 423/730 [37:26<27:18,  5.34s/it]
                                                 
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.4679969549179077, 'epoch': 5.79}

 58%|█████▊    | 423/730 [37:26<27:18,  5.34s/it]
                                                 
{'train/learning_rate_real': 9.911955668254653e-06, 'epoch': 5.79}

 58%|█████▊    | 423/730 [37:26<27:18,  5.34s/it]
                                                 
{'debug/num_tok_total': 2232.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2232.0, 'debug/num_lat_loss': 1792.0, 'epoch': 5.79}

 58%|█████▊    | 423/730 [37:27<27:18,  5.34s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.4877637028694153, 'epoch': 5.79}

 58%|█████▊    | 423/730 [37:27<27:18,  5.34s/it]
                                                 
{'train/learning_rate_real': 9.911955668254653e-06, 'epoch': 5.79}

 58%|█████▊    | 423/730 [37:27<27:18,  5.34s/it]
                                                 
{'debug/num_tok_total': 3066.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 3066.0, 'debug/num_lat_loss': 1772.0, 'epoch': 5.79}

 58%|█████▊    | 423/730 [37:29<27:18,  5.34s/it]
                                                 
{'train/ce_loss': 1.890625, 'train/diffusion_loss': 0.34702908992767334, 'epoch': 5.79}

 58%|█████▊    | 423/730 [37:29<27:18,  5.34s/it]
                                                 
{'train/learning_rate_real': 9.911955668254653e-06, 'epoch': 5.79}

 58%|█████▊    | 423/730 [37:29<27:18,  5.34s/it]
                                                 
{'debug/num_tok_total': 2393.0, 'debug/num_tok_loss': 1752.0, 'debug/num_lat_total': 2393.0, 'debug/num_lat_loss': 1752.0, 'epoch': 5.79}

 58%|█████▊    | 423/730 [37:30<27:18,  5.34s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.4244561493396759, 'epoch': 5.79}

 58%|█████▊    | 423/730 [37:30<27:18,  5.34s/it]
                                                 
{'train/learning_rate_real': 9.911955668254653e-06, 'epoch': 5.79}

 58%|█████▊    | 423/730 [37:30<27:18,  5.34s/it]
 58%|█████▊    | 424/730 [37:30<27:15,  5.34s/it]
                                                 
{'debug/num_tok_total': 2291.0, 'debug/num_tok_loss': 1598.0, 'debug/num_lat_total': 2291.0, 'debug/num_lat_loss': 1598.0, 'epoch': 5.81}

 58%|█████▊    | 424/730 [37:31<27:15,  5.34s/it]
                                                 
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.39606958627700806, 'epoch': 5.81}

 58%|█████▊    | 424/730 [37:31<27:15,  5.34s/it]
                                                 
{'train/learning_rate_real': 9.857717204629696e-06, 'epoch': 5.81}

 58%|█████▊    | 424/730 [37:31<27:15,  5.34s/it]
                                                 
{'debug/num_tok_total': 2893.0, 'debug/num_tok_loss': 1804.0, 'debug/num_lat_total': 2893.0, 'debug/num_lat_loss': 1804.0, 'epoch': 5.81}

 58%|█████▊    | 424/730 [37:33<27:15,  5.34s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.3706355094909668, 'epoch': 5.81}

 58%|█████▊    | 424/730 [37:33<27:15,  5.34s/it]
                                                 
{'train/learning_rate_real': 9.857717204629696e-06, 'epoch': 5.81}

 58%|█████▊    | 424/730 [37:33<27:15,  5.34s/it]
                                                 
{'debug/num_tok_total': 1973.0, 'debug/num_tok_loss': 1757.0, 'debug/num_lat_total': 1973.0, 'debug/num_lat_loss': 1757.0, 'epoch': 5.81}

 58%|█████▊    | 424/730 [37:34<27:15,  5.34s/it]
                                                 
{'train/ce_loss': 2.125, 'train/diffusion_loss': 0.5174368023872375, 'epoch': 5.81}

 58%|█████▊    | 424/730 [37:34<27:15,  5.34s/it]
                                                 
{'train/learning_rate_real': 9.857717204629696e-06, 'epoch': 5.81}

 58%|█████▊    | 424/730 [37:34<27:15,  5.34s/it]
                                                 
{'debug/num_tok_total': 2181.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 2181.0, 'debug/num_lat_loss': 1768.0, 'epoch': 5.81}

 58%|█████▊    | 424/730 [37:35<27:15,  5.34s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.47832831740379333, 'epoch': 5.81}

 58%|█████▊    | 424/730 [37:35<27:15,  5.34s/it]
                                                 
{'train/learning_rate_real': 9.857717204629696e-06, 'epoch': 5.81}

 58%|█████▊    | 424/730 [37:35<27:15,  5.34s/it]
 58%|█████▊    | 425/730 [37:35<26:27,  5.20s/it]
                                                 
{'debug/num_tok_total': 2634.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2634.0, 'debug/num_lat_loss': 1776.0, 'epoch': 5.82}

 58%|█████▊    | 425/730 [37:36<26:27,  5.20s/it]
                                                 
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.42211970686912537, 'epoch': 5.82}

 58%|█████▊    | 425/730 [37:36<26:27,  5.20s/it]
                                                 
{'train/learning_rate_real': 9.80353076597186e-06, 'epoch': 5.82}

 58%|█████▊    | 425/730 [37:36<26:27,  5.20s/it]
                                                 
{'debug/num_tok_total': 2644.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2644.0, 'debug/num_lat_loss': 1777.0, 'epoch': 5.82}

 58%|█████▊    | 425/730 [37:38<26:27,  5.20s/it]
                                                 
{'train/ce_loss': 1.96875, 'train/diffusion_loss': 0.4115304946899414, 'epoch': 5.82}

 58%|█████▊    | 425/730 [37:38<26:27,  5.20s/it]
                                                 
{'train/learning_rate_real': 9.80353076597186e-06, 'epoch': 5.82}

 58%|█████▊    | 425/730 [37:38<26:27,  5.20s/it]
                                                 
{'debug/num_tok_total': 3047.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 3047.0, 'debug/num_lat_loss': 1774.0, 'epoch': 5.82}

 58%|█████▊    | 425/730 [37:39<26:27,  5.20s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.31300896406173706, 'epoch': 5.82}

 58%|█████▊    | 425/730 [37:39<26:27,  5.20s/it]
                                                 
{'train/learning_rate_real': 9.80353076597186e-06, 'epoch': 5.82}

 58%|█████▊    | 425/730 [37:39<26:27,  5.20s/it]
                                                 
{'debug/num_tok_total': 1979.0, 'debug/num_tok_loss': 1757.0, 'debug/num_lat_total': 1979.0, 'debug/num_lat_loss': 1757.0, 'epoch': 5.82}

 58%|█████▊    | 425/730 [37:40<26:27,  5.20s/it]
                                                 
{'train/ce_loss': 2.046875, 'train/diffusion_loss': 0.548903226852417, 'epoch': 5.82}

 58%|█████▊    | 425/730 [37:40<26:27,  5.20s/it]
                                                 
{'train/learning_rate_real': 9.80353076597186e-06, 'epoch': 5.82}

 58%|█████▊    | 425/730 [37:40<26:27,  5.20s/it]
 58%|█████▊    | 426/730 [37:40<26:20,  5.20s/it]
                                                 
{'debug/num_tok_total': 2319.0, 'debug/num_tok_loss': 1669.0, 'debug/num_lat_total': 2319.0, 'debug/num_lat_loss': 1669.0, 'epoch': 5.84}

 58%|█████▊    | 426/730 [37:41<26:20,  5.20s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.44378024339675903, 'epoch': 5.84}

 58%|█████▊    | 426/730 [37:41<26:20,  5.20s/it]
                                                 
{'train/learning_rate_real': 9.749397419179696e-06, 'epoch': 5.84}

 58%|█████▊    | 426/730 [37:41<26:20,  5.20s/it]
                                                 
{'debug/num_tok_total': 1983.0, 'debug/num_tok_loss': 1665.0, 'debug/num_lat_total': 1983.0, 'debug/num_lat_loss': 1665.0, 'epoch': 5.84}

 58%|█████▊    | 426/730 [37:42<26:20,  5.20s/it]
                                                 
{'train/ce_loss': 1.875, 'train/diffusion_loss': 0.4928912818431854, 'epoch': 5.84}

 58%|█████▊    | 426/730 [37:43<26:20,  5.20s/it]
                                                 
{'train/learning_rate_real': 9.749397419179696e-06, 'epoch': 5.84}

 58%|█████▊    | 426/730 [37:43<26:20,  5.20s/it]
                                                 
{'debug/num_tok_total': 3098.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 3098.0, 'debug/num_lat_loss': 1795.0, 'epoch': 5.84}

 58%|█████▊    | 426/730 [37:44<26:20,  5.20s/it]
                                                 
{'train/ce_loss': 1.9921875, 'train/diffusion_loss': 0.3590208888053894, 'epoch': 5.84}

 58%|█████▊    | 426/730 [37:44<26:20,  5.20s/it]
                                                 
{'train/learning_rate_real': 9.749397419179696e-06, 'epoch': 5.84}

 58%|█████▊    | 426/730 [37:44<26:20,  5.20s/it]
                                                 
{'debug/num_tok_total': 2617.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 2617.0, 'debug/num_lat_loss': 1767.0, 'epoch': 5.84}

 58%|█████▊    | 426/730 [37:45<26:20,  5.20s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.410087913274765, 'epoch': 5.84}

 58%|█████▊    | 426/730 [37:45<26:20,  5.20s/it]
                                                 
{'train/learning_rate_real': 9.749397419179696e-06, 'epoch': 5.84}

 58%|█████▊    | 426/730 [37:45<26:20,  5.20s/it]
 58%|█████▊    | 427/730 [37:45<26:13,  5.19s/it]
                                                 
{'debug/num_tok_total': 2384.0, 'debug/num_tok_loss': 1746.0, 'debug/num_lat_total': 2384.0, 'debug/num_lat_loss': 1746.0, 'epoch': 5.85}

 58%|█████▊    | 427/730 [37:47<26:13,  5.19s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.4360010623931885, 'epoch': 5.85}

 58%|█████▊    | 427/730 [37:47<26:13,  5.19s/it]
                                                 
{'train/learning_rate_real': 9.695318230106401e-06, 'epoch': 5.85}

 58%|█████▊    | 427/730 [37:47<26:13,  5.19s/it]
                                                 
{'debug/num_tok_total': 2665.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2665.0, 'debug/num_lat_loss': 1796.0, 'epoch': 5.85}

 58%|█████▊    | 427/730 [37:48<26:13,  5.19s/it]
                                                 
{'train/ce_loss': 1.9140625, 'train/diffusion_loss': 0.4289091229438782, 'epoch': 5.85}

 58%|█████▊    | 427/730 [37:48<26:13,  5.19s/it]
                                                 
{'train/learning_rate_real': 9.695318230106401e-06, 'epoch': 5.85}

 58%|█████▊    | 427/730 [37:48<26:13,  5.19s/it]
                                                 
{'debug/num_tok_total': 2814.0, 'debug/num_tok_loss': 1756.0, 'debug/num_lat_total': 2814.0, 'debug/num_lat_loss': 1756.0, 'epoch': 5.85}

 58%|█████▊    | 427/730 [37:49<26:13,  5.19s/it]
                                                 
{'train/ce_loss': 2.34375, 'train/diffusion_loss': 0.3798215985298157, 'epoch': 5.85}

 58%|█████▊    | 427/730 [37:49<26:13,  5.19s/it]
                                                 
{'train/learning_rate_real': 9.695318230106401e-06, 'epoch': 5.85}

 58%|█████▊    | 427/730 [37:49<26:13,  5.19s/it]
                                                 
{'debug/num_tok_total': 2337.0, 'debug/num_tok_loss': 1716.0, 'debug/num_lat_total': 2337.0, 'debug/num_lat_loss': 1716.0, 'epoch': 5.85}

 58%|█████▊    | 427/730 [37:50<26:13,  5.19s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.42668646574020386, 'epoch': 5.85}

 58%|█████▊    | 427/730 [37:50<26:13,  5.19s/it]
                                                 
{'train/learning_rate_real': 9.695318230106401e-06, 'epoch': 5.85}

 58%|█████▊    | 427/730 [37:50<26:13,  5.19s/it]
 59%|█████▊    | 428/730 [37:51<26:08,  5.19s/it]
                                                 
{'debug/num_tok_total': 2633.0, 'debug/num_tok_loss': 1676.0, 'debug/num_lat_total': 2633.0, 'debug/num_lat_loss': 1676.0, 'epoch': 5.86}

 59%|█████▊    | 428/730 [37:52<26:08,  5.19s/it]
                                                 
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.3663332462310791, 'epoch': 5.86}

 59%|█████▊    | 428/730 [37:52<26:08,  5.19s/it]
                                                 
{'train/learning_rate_real': 9.641294263538857e-06, 'epoch': 5.86}

 59%|█████▊    | 428/730 [37:52<26:08,  5.19s/it]
                                                 
{'debug/num_tok_total': 2181.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2181.0, 'debug/num_lat_loss': 1763.0, 'epoch': 5.86}

 59%|█████▊    | 428/730 [37:53<26:08,  5.19s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.4830867052078247, 'epoch': 5.86}

 59%|█████▊    | 428/730 [37:53<26:08,  5.19s/it]
                                                 
{'train/learning_rate_real': 9.641294263538857e-06, 'epoch': 5.86}

 59%|█████▊    | 428/730 [37:53<26:08,  5.19s/it]
                                                 
{'debug/num_tok_total': 2420.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2420.0, 'debug/num_lat_loss': 1778.0, 'epoch': 5.86}

 59%|█████▊    | 428/730 [37:54<26:08,  5.19s/it]
                                                 
{'train/ce_loss': 2.03125, 'train/diffusion_loss': 0.4264194369316101, 'epoch': 5.86}

 59%|█████▊    | 428/730 [37:54<26:08,  5.19s/it]
                                                 
{'train/learning_rate_real': 9.641294263538857e-06, 'epoch': 5.86}

 59%|█████▊    | 428/730 [37:54<26:08,  5.19s/it]
                                                 
{'debug/num_tok_total': 2433.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2433.0, 'debug/num_lat_loss': 1790.0, 'epoch': 5.86}

 59%|█████▊    | 428/730 [37:56<26:08,  5.19s/it]
                                                 
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.4411565065383911, 'epoch': 5.86}

 59%|█████▊    | 428/730 [37:56<26:08,  5.19s/it]
                                                 
{'train/learning_rate_real': 9.641294263538857e-06, 'epoch': 5.86}

 59%|█████▊    | 428/730 [37:56<26:08,  5.19s/it]
 59%|█████▉    | 429/730 [37:56<26:13,  5.23s/it]
                                                 
{'debug/num_tok_total': 3076.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 3076.0, 'debug/num_lat_loss': 1787.0, 'epoch': 5.88}

 59%|█████▉    | 429/730 [37:57<26:13,  5.23s/it]
                                                 
{'train/ce_loss': 1.6875, 'train/diffusion_loss': 0.34062421321868896, 'epoch': 5.88}

 59%|█████▉    | 429/730 [37:57<26:13,  5.23s/it]
                                                 
{'train/learning_rate_real': 9.587326583176636e-06, 'epoch': 5.88}

 59%|█████▉    | 429/730 [37:57<26:13,  5.23s/it]
                                                 
{'debug/num_tok_total': 2787.0, 'debug/num_tok_loss': 1735.0, 'debug/num_lat_total': 2787.0, 'debug/num_lat_loss': 1735.0, 'epoch': 5.88}

 59%|█████▉    | 429/730 [37:59<26:13,  5.23s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.352632611989975, 'epoch': 5.88}

 59%|█████▉    | 429/730 [37:59<26:13,  5.23s/it]
                                                 
{'train/learning_rate_real': 9.587326583176636e-06, 'epoch': 5.88}

 59%|█████▉    | 429/730 [37:59<26:13,  5.23s/it]
                                                 
{'debug/num_tok_total': 2177.0, 'debug/num_tok_loss': 1749.0, 'debug/num_lat_total': 2177.0, 'debug/num_lat_loss': 1749.0, 'epoch': 5.88}

 59%|█████▉    | 429/730 [38:00<26:13,  5.23s/it]
                                                 
{'train/ce_loss': 1.8828125, 'train/diffusion_loss': 0.48058760166168213, 'epoch': 5.88}

 59%|█████▉    | 429/730 [38:00<26:13,  5.23s/it]
                                                 
{'train/learning_rate_real': 9.587326583176636e-06, 'epoch': 5.88}

 59%|█████▉    | 429/730 [38:00<26:13,  5.23s/it]
                                                 
{'debug/num_tok_total': 2654.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2654.0, 'debug/num_lat_loss': 1793.0, 'epoch': 5.88}

 59%|█████▉    | 429/730 [38:01<26:13,  5.23s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.4013981521129608, 'epoch': 5.88}

 59%|█████▉    | 429/730 [38:01<26:13,  5.23s/it]
                                                 
{'train/learning_rate_real': 9.587326583176636e-06, 'epoch': 5.88}

 59%|█████▉    | 429/730 [38:01<26:13,  5.23s/it]03/16/2026 07:25:33 - INFO - __main__ - LoRA debug step 430: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 59%|█████▉    | 430/730 [38:01<26:28,  5.29s/it]
                                                 
{'loss': 2.6093, 'grad_norm': 1.3288816213607788, 'learning_rate': 9.587326583176636e-06, 'epoch': 5.89}

 59%|█████▉    | 430/730 [38:01<26:28,  5.29s/it]
                                                 
{'debug/num_tok_total': 1991.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 1991.0, 'debug/num_lat_loss': 1781.0, 'epoch': 5.89}

 59%|█████▉    | 430/730 [38:02<26:28,  5.29s/it]
                                                 
{'train/ce_loss': 2.296875, 'train/diffusion_loss': 0.5637247562408447, 'epoch': 5.89}

 59%|█████▉    | 430/730 [38:02<26:28,  5.29s/it]
                                                 
{'train/learning_rate_real': 9.533416251611064e-06, 'epoch': 5.89}

 59%|█████▉    | 430/730 [38:02<26:28,  5.29s/it]
                                                 
{'debug/num_tok_total': 2602.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2602.0, 'debug/num_lat_loss': 1770.0, 'epoch': 5.89}

 59%|█████▉    | 430/730 [38:04<26:28,  5.29s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.4268699884414673, 'epoch': 5.89}

 59%|█████▉    | 430/730 [38:04<26:28,  5.29s/it]
                                                 
{'train/learning_rate_real': 9.533416251611064e-06, 'epoch': 5.89}

 59%|█████▉    | 430/730 [38:04<26:28,  5.29s/it]
                                                 
{'debug/num_tok_total': 2204.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2204.0, 'debug/num_lat_loss': 1771.0, 'epoch': 5.89}

 59%|█████▉    | 430/730 [38:05<26:28,  5.29s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.5015642046928406, 'epoch': 5.89}

 59%|█████▉    | 430/730 [38:05<26:28,  5.29s/it]
                                                 
{'train/learning_rate_real': 9.533416251611064e-06, 'epoch': 5.89}

 59%|█████▉    | 430/730 [38:05<26:28,  5.29s/it]
                                                 
{'debug/num_tok_total': 2621.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2621.0, 'debug/num_lat_loss': 1778.0, 'epoch': 5.89}

 59%|█████▉    | 430/730 [38:06<26:28,  5.29s/it]
                                                 
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.4430067837238312, 'epoch': 5.89}

 59%|█████▉    | 430/730 [38:06<26:28,  5.29s/it]
                                                 
{'train/learning_rate_real': 9.533416251611064e-06, 'epoch': 5.89}

 59%|█████▉    | 430/730 [38:06<26:28,  5.29s/it]
 59%|█████▉    | 431/730 [38:07<26:07,  5.24s/it]
                                                 
{'debug/num_tok_total': 2416.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2416.0, 'debug/num_lat_loss': 1785.0, 'epoch': 5.9}

 59%|█████▉    | 431/730 [38:08<26:07,  5.24s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.4236304461956024, 'epoch': 5.9}

 59%|█████▉    | 431/730 [38:08<26:07,  5.24s/it]
                                                 
{'train/learning_rate_real': 9.479564330304323e-06, 'epoch': 5.9}

 59%|█████▉    | 431/730 [38:08<26:07,  5.24s/it]
                                                 
{'debug/num_tok_total': 2431.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2431.0, 'debug/num_lat_loss': 1778.0, 'epoch': 5.9}

 59%|█████▉    | 431/730 [38:09<26:07,  5.24s/it]
                                                 
{'train/ce_loss': 2.03125, 'train/diffusion_loss': 0.44115564227104187, 'epoch': 5.9}

 59%|█████▉    | 431/730 [38:09<26:07,  5.24s/it]
                                                 
{'train/learning_rate_real': 9.479564330304323e-06, 'epoch': 5.9}

 59%|█████▉    | 431/730 [38:09<26:07,  5.24s/it]
                                                 
{'debug/num_tok_total': 2603.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 2603.0, 'debug/num_lat_loss': 1769.0, 'epoch': 5.9}

 59%|█████▉    | 431/730 [38:10<26:07,  5.24s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.401658833026886, 'epoch': 5.9}

 59%|█████▉    | 431/730 [38:10<26:07,  5.24s/it]
                                                 
{'train/learning_rate_real': 9.479564330304323e-06, 'epoch': 5.9}

 59%|█████▉    | 431/730 [38:10<26:07,  5.24s/it]
                                                 
{'debug/num_tok_total': 2638.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2638.0, 'debug/num_lat_loss': 1772.0, 'epoch': 5.9}

 59%|█████▉    | 431/730 [38:11<26:07,  5.24s/it]
                                                 
{'train/ce_loss': 1.8984375, 'train/diffusion_loss': 0.3867921233177185, 'epoch': 5.9}

 59%|█████▉    | 431/730 [38:11<26:07,  5.24s/it]
                                                 
{'train/learning_rate_real': 9.479564330304323e-06, 'epoch': 5.9}

 59%|█████▉    | 431/730 [38:11<26:07,  5.24s/it]
 59%|█████▉    | 432/730 [38:12<25:54,  5.22s/it]
                                                 
{'debug/num_tok_total': 2190.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2190.0, 'debug/num_lat_loss': 1777.0, 'epoch': 5.92}

 59%|█████▉    | 432/730 [38:13<25:54,  5.22s/it]
                                                 
{'train/ce_loss': 1.9765625, 'train/diffusion_loss': 0.4907861649990082, 'epoch': 5.92}

 59%|█████▉    | 432/730 [38:13<25:54,  5.22s/it]
                                                 
{'train/learning_rate_real': 9.425771879568504e-06, 'epoch': 5.92}

 59%|█████▉    | 432/730 [38:13<25:54,  5.22s/it]
                                                 
{'debug/num_tok_total': 2646.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2646.0, 'debug/num_lat_loss': 1785.0, 'epoch': 5.92}

 59%|█████▉    | 432/730 [38:14<25:54,  5.22s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.39678844809532166, 'epoch': 5.92}

 59%|█████▉    | 432/730 [38:14<25:54,  5.22s/it]
                                                 
{'train/learning_rate_real': 9.425771879568504e-06, 'epoch': 5.92}

 59%|█████▉    | 432/730 [38:14<25:54,  5.22s/it]
                                                 
{'debug/num_tok_total': 2217.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2217.0, 'debug/num_lat_loss': 1780.0, 'epoch': 5.92}

 59%|█████▉    | 432/730 [38:15<25:54,  5.22s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.4618002474308014, 'epoch': 5.92}

 59%|█████▉    | 432/730 [38:15<25:54,  5.22s/it]
                                                 
{'train/learning_rate_real': 9.425771879568504e-06, 'epoch': 5.92}

 59%|█████▉    | 432/730 [38:15<25:54,  5.22s/it]
                                                 
{'debug/num_tok_total': 2857.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2857.0, 'debug/num_lat_loss': 1794.0, 'epoch': 5.92}

 59%|█████▉    | 432/730 [38:16<25:54,  5.22s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.36687618494033813, 'epoch': 5.92}

 59%|█████▉    | 432/730 [38:16<25:54,  5.22s/it]
                                                 
{'train/learning_rate_real': 9.425771879568504e-06, 'epoch': 5.92}

 59%|█████▉    | 432/730 [38:16<25:54,  5.22s/it]
 59%|█████▉    | 433/730 [38:17<25:41,  5.19s/it]
                                                 
{'debug/num_tok_total': 3089.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 3089.0, 'debug/num_lat_loss': 1795.0, 'epoch': 5.93}

 59%|█████▉    | 433/730 [38:18<25:41,  5.19s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.3093504011631012, 'epoch': 5.93}

 59%|█████▉    | 433/730 [38:18<25:41,  5.19s/it]
                                                 
{'train/learning_rate_real': 9.372039958544785e-06, 'epoch': 5.93}

 59%|█████▉    | 433/730 [38:18<25:41,  5.19s/it]
                                                 
{'debug/num_tok_total': 3294.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 3294.0, 'debug/num_lat_loss': 1789.0, 'epoch': 5.93}

 59%|█████▉    | 433/730 [38:20<25:41,  5.19s/it]
                                                 
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.2859404683113098, 'epoch': 5.93}

 59%|█████▉    | 433/730 [38:20<25:41,  5.19s/it]
                                                 
{'train/learning_rate_real': 9.372039958544785e-06, 'epoch': 5.93}

 59%|█████▉    | 433/730 [38:20<25:41,  5.19s/it]
                                                 
{'debug/num_tok_total': 2284.0, 'debug/num_tok_loss': 1595.0, 'debug/num_lat_total': 2284.0, 'debug/num_lat_loss': 1595.0, 'epoch': 5.93}

 59%|█████▉    | 433/730 [38:21<25:41,  5.19s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.39802590012550354, 'epoch': 5.93}

 59%|█████▉    | 433/730 [38:21<25:41,  5.19s/it]
                                                 
{'train/learning_rate_real': 9.372039958544785e-06, 'epoch': 5.93}

 59%|█████▉    | 433/730 [38:21<25:41,  5.19s/it]
                                                 
{'debug/num_tok_total': 2738.0, 'debug/num_tok_loss': 1652.0, 'debug/num_lat_total': 2738.0, 'debug/num_lat_loss': 1652.0, 'epoch': 5.93}

 59%|█████▉    | 433/730 [38:22<25:41,  5.19s/it]
                                                 
{'train/ce_loss': 1.9453125, 'train/diffusion_loss': 0.3217955231666565, 'epoch': 5.93}

 59%|█████▉    | 433/730 [38:22<25:41,  5.19s/it]
                                                 
{'train/learning_rate_real': 9.372039958544785e-06, 'epoch': 5.93}

 59%|█████▉    | 433/730 [38:22<25:41,  5.19s/it]
 59%|█████▉    | 434/730 [38:23<26:24,  5.35s/it]
                                                 
{'debug/num_tok_total': 2196.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2196.0, 'debug/num_lat_loss': 1766.0, 'epoch': 5.95}

 59%|█████▉    | 434/730 [38:24<26:24,  5.35s/it]
                                                 
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.4787895381450653, 'epoch': 5.95}

 59%|█████▉    | 434/730 [38:24<26:24,  5.35s/it]
                                                 
{'train/learning_rate_real': 9.31836962518253e-06, 'epoch': 5.95}

 59%|█████▉    | 434/730 [38:24<26:24,  5.35s/it]
                                                 
{'debug/num_tok_total': 2410.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 2410.0, 'debug/num_lat_loss': 1767.0, 'epoch': 5.95}

 59%|█████▉    | 434/730 [38:25<26:24,  5.35s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.4418469965457916, 'epoch': 5.95}

 59%|█████▉    | 434/730 [38:25<26:24,  5.35s/it]
                                                 
{'train/learning_rate_real': 9.31836962518253e-06, 'epoch': 5.95}

 59%|█████▉    | 434/730 [38:25<26:24,  5.35s/it]
                                                 
{'debug/num_tok_total': 2229.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2229.0, 'debug/num_lat_loss': 1799.0, 'epoch': 5.95}

 59%|█████▉    | 434/730 [38:26<26:24,  5.35s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.4865027368068695, 'epoch': 5.95}

 59%|█████▉    | 434/730 [38:26<26:24,  5.35s/it]
                                                 
{'train/learning_rate_real': 9.31836962518253e-06, 'epoch': 5.95}

 59%|█████▉    | 434/730 [38:26<26:24,  5.35s/it]
                                                 
{'debug/num_tok_total': 2276.0, 'debug/num_tok_loss': 1506.0, 'debug/num_lat_total': 2276.0, 'debug/num_lat_loss': 1506.0, 'epoch': 5.95}

 59%|█████▉    | 434/730 [38:27<26:24,  5.35s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.4015607237815857, 'epoch': 5.95}

 59%|█████▉    | 434/730 [38:27<26:24,  5.35s/it]
                                                 
{'train/learning_rate_real': 9.31836962518253e-06, 'epoch': 5.95}

 59%|█████▉    | 434/730 [38:27<26:24,  5.35s/it]
 60%|█████▉    | 435/730 [38:27<25:36,  5.21s/it]
                                                 
{'debug/num_tok_total': 2225.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2225.0, 'debug/num_lat_loss': 1789.0, 'epoch': 5.96}

 60%|█████▉    | 435/730 [38:29<25:36,  5.21s/it]
                                                 
{'train/ce_loss': 1.9296875, 'train/diffusion_loss': 0.5153000354766846, 'epoch': 5.96}

 60%|█████▉    | 435/730 [38:29<25:36,  5.21s/it]
                                                 
{'train/learning_rate_real': 9.26476193621849e-06, 'epoch': 5.96}

 60%|█████▉    | 435/730 [38:29<25:36,  5.21s/it]
                                                 
{'debug/num_tok_total': 2656.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2656.0, 'debug/num_lat_loss': 1793.0, 'epoch': 5.96}

 60%|█████▉    | 435/730 [38:30<25:36,  5.21s/it]
                                                 
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.37765416502952576, 'epoch': 5.96}

 60%|█████▉    | 435/730 [38:30<25:36,  5.21s/it]
                                                 
{'train/learning_rate_real': 9.26476193621849e-06, 'epoch': 5.96}

 60%|█████▉    | 435/730 [38:30<25:36,  5.21s/it]
                                                 
{'debug/num_tok_total': 2928.0, 'debug/num_tok_loss': 1820.0, 'debug/num_lat_total': 2928.0, 'debug/num_lat_loss': 1820.0, 'epoch': 5.96}

 60%|█████▉    | 435/730 [38:31<25:36,  5.21s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.3662756085395813, 'epoch': 5.96}

 60%|█████▉    | 435/730 [38:31<25:36,  5.21s/it]
                                                 
{'train/learning_rate_real': 9.26476193621849e-06, 'epoch': 5.96}

 60%|█████▉    | 435/730 [38:31<25:36,  5.21s/it]
                                                 
{'debug/num_tok_total': 2647.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2647.0, 'debug/num_lat_loss': 1779.0, 'epoch': 5.96}

 60%|█████▉    | 435/730 [38:32<25:36,  5.21s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.38642314076423645, 'epoch': 5.96}

 60%|█████▉    | 435/730 [38:32<25:36,  5.21s/it]
                                                 
{'train/learning_rate_real': 9.26476193621849e-06, 'epoch': 5.96}

 60%|█████▉    | 435/730 [38:32<25:36,  5.21s/it]
 60%|█████▉    | 436/730 [38:33<25:39,  5.24s/it]
                                                 
{'debug/num_tok_total': 2848.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2848.0, 'debug/num_lat_loss': 1775.0, 'epoch': 5.97}

 60%|█████▉    | 436/730 [38:34<25:39,  5.24s/it]
                                                 
{'train/ce_loss': 1.96875, 'train/diffusion_loss': 0.35744509100914, 'epoch': 5.97}

 60%|█████▉    | 436/730 [38:34<25:39,  5.24s/it]
                                                 
{'train/learning_rate_real': 9.211217947155986e-06, 'epoch': 5.97}

 60%|█████▉    | 436/730 [38:34<25:39,  5.24s/it]
                                                 
{'debug/num_tok_total': 2850.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2850.0, 'debug/num_lat_loss': 1776.0, 'epoch': 5.97}

 60%|█████▉    | 436/730 [38:35<25:39,  5.24s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.3617651164531708, 'epoch': 5.97}

 60%|█████▉    | 436/730 [38:35<25:39,  5.24s/it]
                                                 
{'train/learning_rate_real': 9.211217947155986e-06, 'epoch': 5.97}

 60%|█████▉    | 436/730 [38:35<25:39,  5.24s/it]
                                                 
{'debug/num_tok_total': 2870.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2870.0, 'debug/num_lat_loss': 1794.0, 'epoch': 5.97}

 60%|█████▉    | 436/730 [38:37<25:39,  5.24s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.34287673234939575, 'epoch': 5.97}

 60%|█████▉    | 436/730 [38:37<25:39,  5.24s/it]
                                                 
{'train/learning_rate_real': 9.211217947155986e-06, 'epoch': 5.97}

 60%|█████▉    | 436/730 [38:37<25:39,  5.24s/it]
                                                 
{'debug/num_tok_total': 2419.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2419.0, 'debug/num_lat_loss': 1784.0, 'epoch': 5.97}

 60%|█████▉    | 436/730 [38:38<25:39,  5.24s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.4326181411743164, 'epoch': 5.97}

 60%|█████▉    | 436/730 [38:38<25:39,  5.24s/it]
                                                 
{'train/learning_rate_real': 9.211217947155986e-06, 'epoch': 5.97}

 60%|█████▉    | 436/730 [38:38<25:39,  5.24s/it]
 60%|█████▉    | 437/730 [38:38<25:54,  5.30s/it]
                                                 
{'debug/num_tok_total': 1994.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 1994.0, 'debug/num_lat_loss': 1779.0, 'epoch': 5.99}

 60%|█████▉    | 437/730 [38:39<25:54,  5.30s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.5338225960731506, 'epoch': 5.99}

 60%|█████▉    | 437/730 [38:39<25:54,  5.30s/it]
                                                 
{'train/learning_rate_real': 9.157738712244118e-06, 'epoch': 5.99}

 60%|█████▉    | 437/730 [38:39<25:54,  5.30s/it]
                                                 
{'debug/num_tok_total': 2836.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2836.0, 'debug/num_lat_loss': 1773.0, 'epoch': 5.99}

 60%|█████▉    | 437/730 [38:40<25:54,  5.30s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.3926565647125244, 'epoch': 5.99}

 60%|█████▉    | 437/730 [38:40<25:54,  5.30s/it]
                                                 
{'train/learning_rate_real': 9.157738712244118e-06, 'epoch': 5.99}

 60%|█████▉    | 437/730 [38:40<25:54,  5.30s/it]
                                                 
{'debug/num_tok_total': 2445.0, 'debug/num_tok_loss': 1586.0, 'debug/num_lat_total': 2445.0, 'debug/num_lat_loss': 1586.0, 'epoch': 5.99}

 60%|█████▉    | 437/730 [38:42<25:54,  5.30s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.3961816132068634, 'epoch': 5.99}

 60%|█████▉    | 437/730 [38:42<25:54,  5.30s/it]
                                                 
{'train/learning_rate_real': 9.157738712244118e-06, 'epoch': 5.99}

 60%|█████▉    | 437/730 [38:42<25:54,  5.30s/it]
                                                 
{'debug/num_tok_total': 455.0, 'debug/num_tok_loss': 455.0, 'debug/num_lat_total': 455.0, 'debug/num_lat_loss': 455.0, 'epoch': 5.99}

 60%|█████▉    | 437/730 [38:42<25:54,  5.30s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.6469253897666931, 'epoch': 5.99}

 60%|█████▉    | 437/730 [38:42<25:54,  5.30s/it]
                                                 
{'train/learning_rate_real': 9.157738712244118e-06, 'epoch': 5.99}

 60%|█████▉    | 437/730 [38:42<25:54,  5.30s/it]
 60%|██████    | 438/730 [38:42<24:03,  4.94s/it]
                                                 
{'debug/num_tok_total': 2496.0, 'debug/num_tok_loss': 1833.0, 'debug/num_lat_total': 2496.0, 'debug/num_lat_loss': 1833.0, 'epoch': 6.0}

 60%|██████    | 438/730 [38:44<24:03,  4.94s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.445485919713974, 'epoch': 6.0}

 60%|██████    | 438/730 [38:44<24:03,  4.94s/it]
                                                 
{'train/learning_rate_real': 9.104325284457017e-06, 'epoch': 6.0}

 60%|██████    | 438/730 [38:44<24:03,  4.94s/it]
                                                 
{'debug/num_tok_total': 2445.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2445.0, 'debug/num_lat_loss': 1792.0, 'epoch': 6.0}

 60%|██████    | 438/730 [38:45<24:03,  4.94s/it]
                                                 
{'train/ce_loss': 1.9375, 'train/diffusion_loss': 0.4131440818309784, 'epoch': 6.0}

 60%|██████    | 438/730 [38:45<24:03,  4.94s/it]
                                                 
{'train/learning_rate_real': 9.104325284457017e-06, 'epoch': 6.0}

 60%|██████    | 438/730 [38:45<24:03,  4.94s/it]
                                                 
{'debug/num_tok_total': 2602.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2602.0, 'debug/num_lat_loss': 1771.0, 'epoch': 6.0}

 60%|██████    | 438/730 [38:46<24:03,  4.94s/it]
                                                 
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.3955443501472473, 'epoch': 6.0}

 60%|██████    | 438/730 [38:46<24:03,  4.94s/it]
                                                 
{'train/learning_rate_real': 9.104325284457017e-06, 'epoch': 6.0}

 60%|██████    | 438/730 [38:46<24:03,  4.94s/it]
                                                 
{'debug/num_tok_total': 3104.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 3104.0, 'debug/num_lat_loss': 1797.0, 'epoch': 6.0}

 60%|██████    | 438/730 [38:47<24:03,  4.94s/it]
                                                 
{'train/ce_loss': 2.109375, 'train/diffusion_loss': 0.29404738545417786, 'epoch': 6.0}

 60%|██████    | 438/730 [38:47<24:03,  4.94s/it]
                                                 
{'train/learning_rate_real': 9.104325284457017e-06, 'epoch': 6.0}

 60%|██████    | 438/730 [38:47<24:03,  4.94s/it]
 60%|██████    | 439/730 [38:48<24:35,  5.07s/it]
                                                 
{'debug/num_tok_total': 2856.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2856.0, 'debug/num_lat_loss': 1781.0, 'epoch': 6.01}

 60%|██████    | 439/730 [38:49<24:35,  5.07s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.375998854637146, 'epoch': 6.01}

 60%|██████    | 439/730 [38:49<24:35,  5.07s/it]
                                                 
{'train/learning_rate_real': 9.05097871547312e-06, 'epoch': 6.01}

 60%|██████    | 439/730 [38:49<24:35,  5.07s/it]
                                                 
{'debug/num_tok_total': 3116.0, 'debug/num_tok_loss': 1807.0, 'debug/num_lat_total': 3116.0, 'debug/num_lat_loss': 1807.0, 'epoch': 6.01}

 60%|██████    | 439/730 [38:50<24:35,  5.07s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.31189578771591187, 'epoch': 6.01}

 60%|██████    | 439/730 [38:50<24:35,  5.07s/it]
                                                 
{'train/learning_rate_real': 9.05097871547312e-06, 'epoch': 6.01}

 60%|██████    | 439/730 [38:50<24:35,  5.07s/it]
                                                 
{'debug/num_tok_total': 3095.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 3095.0, 'debug/num_lat_loss': 1799.0, 'epoch': 6.01}

 60%|██████    | 439/730 [38:52<24:35,  5.07s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.32597991824150085, 'epoch': 6.01}

 60%|██████    | 439/730 [38:52<24:35,  5.07s/it]
                                                 
{'train/learning_rate_real': 9.05097871547312e-06, 'epoch': 6.01}

 60%|██████    | 439/730 [38:52<24:35,  5.07s/it]
                                                 
{'debug/num_tok_total': 2469.0, 'debug/num_tok_loss': 1698.0, 'debug/num_lat_total': 2469.0, 'debug/num_lat_loss': 1698.0, 'epoch': 6.01}

 60%|██████    | 439/730 [38:53<24:35,  5.07s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.411367267370224, 'epoch': 6.01}

 60%|██████    | 439/730 [38:53<24:35,  5.07s/it]
                                                 
{'train/learning_rate_real': 9.05097871547312e-06, 'epoch': 6.01}

 60%|██████    | 439/730 [38:53<24:35,  5.07s/it]03/16/2026 07:26:26 - INFO - __main__ - LoRA debug step 440: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 60%|██████    | 440/730 [38:54<25:50,  5.35s/it]
                                                 
{'loss': 2.6077, 'grad_norm': 1.3773133754730225, 'learning_rate': 9.05097871547312e-06, 'epoch': 6.03}

 60%|██████    | 440/730 [38:54<25:50,  5.35s/it]
                                                 
{'debug/num_tok_total': 2733.0, 'debug/num_tok_loss': 1656.0, 'debug/num_lat_total': 2733.0, 'debug/num_lat_loss': 1656.0, 'epoch': 6.03}

 60%|██████    | 440/730 [38:55<25:50,  5.35s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.3330347239971161, 'epoch': 6.03}

 60%|██████    | 440/730 [38:55<25:50,  5.35s/it]
                                                 
{'train/learning_rate_real': 8.997700055654442e-06, 'epoch': 6.03}

 60%|██████    | 440/730 [38:55<25:50,  5.35s/it]
                                                 
{'debug/num_tok_total': 2431.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2431.0, 'debug/num_lat_loss': 1799.0, 'epoch': 6.03}

 60%|██████    | 440/730 [38:56<25:50,  5.35s/it]
                                                 
{'train/ce_loss': 1.9296875, 'train/diffusion_loss': 0.4132092595100403, 'epoch': 6.03}

 60%|██████    | 440/730 [38:56<25:50,  5.35s/it]
                                                 
{'train/learning_rate_real': 8.997700055654442e-06, 'epoch': 6.03}

 60%|██████    | 440/730 [38:56<25:50,  5.35s/it]
                                                 
{'debug/num_tok_total': 2311.0, 'debug/num_tok_loss': 1661.0, 'debug/num_lat_total': 2311.0, 'debug/num_lat_loss': 1661.0, 'epoch': 6.03}

 60%|██████    | 440/730 [38:57<25:50,  5.35s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.4317696690559387, 'epoch': 6.03}

 60%|██████    | 440/730 [38:57<25:50,  5.35s/it]
                                                 
{'train/learning_rate_real': 8.997700055654442e-06, 'epoch': 6.03}

 60%|██████    | 440/730 [38:57<25:50,  5.35s/it]
                                                 
{'debug/num_tok_total': 2841.0, 'debug/num_tok_loss': 1666.0, 'debug/num_lat_total': 2841.0, 'debug/num_lat_loss': 1666.0, 'epoch': 6.03}

 60%|██████    | 440/730 [38:59<25:50,  5.35s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.31207218766212463, 'epoch': 6.03}

 60%|██████    | 440/730 [38:59<25:50,  5.35s/it]
                                                 
{'train/learning_rate_real': 8.997700055654442e-06, 'epoch': 6.03}

 60%|██████    | 440/730 [38:59<25:50,  5.35s/it]
 60%|██████    | 441/730 [38:59<26:12,  5.44s/it]
                                                 
{'debug/num_tok_total': 2262.0, 'debug/num_tok_loss': 1622.0, 'debug/num_lat_total': 2262.0, 'debug/num_lat_loss': 1622.0, 'epoch': 6.04}

 60%|██████    | 441/730 [39:00<26:12,  5.44s/it]
                                                 
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.4449695348739624, 'epoch': 6.04}

 60%|██████    | 441/730 [39:00<26:12,  5.44s/it]
                                                 
{'train/learning_rate_real': 8.944490354025923e-06, 'epoch': 6.04}

 60%|██████    | 441/730 [39:00<26:12,  5.44s/it]
                                                 
{'debug/num_tok_total': 2896.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2896.0, 'debug/num_lat_loss': 1802.0, 'epoch': 6.04}

 60%|██████    | 441/730 [39:02<26:12,  5.44s/it]
                                                 
{'train/ce_loss': 1.9921875, 'train/diffusion_loss': 0.3752349615097046, 'epoch': 6.04}

 60%|██████    | 441/730 [39:02<26:12,  5.44s/it]
                                                 
{'train/learning_rate_real': 8.944490354025923e-06, 'epoch': 6.04}

 60%|██████    | 441/730 [39:02<26:12,  5.44s/it]
                                                 
{'debug/num_tok_total': 2463.0, 'debug/num_tok_loss': 1683.0, 'debug/num_lat_total': 2463.0, 'debug/num_lat_loss': 1683.0, 'epoch': 6.04}

 60%|██████    | 441/730 [39:03<26:12,  5.44s/it]
                                                 
{'train/ce_loss': 1.6171875, 'train/diffusion_loss': 0.3932560086250305, 'epoch': 6.04}

 60%|██████    | 441/730 [39:03<26:12,  5.44s/it]
                                                 
{'train/learning_rate_real': 8.944490354025923e-06, 'epoch': 6.04}

 60%|██████    | 441/730 [39:03<26:12,  5.44s/it]
                                                 
{'debug/num_tok_total': 2672.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2672.0, 'debug/num_lat_loss': 1802.0, 'epoch': 6.04}

 60%|██████    | 441/730 [39:04<26:12,  5.44s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.41405415534973145, 'epoch': 6.04}

 60%|██████    | 441/730 [39:04<26:12,  5.44s/it]
                                                 
{'train/learning_rate_real': 8.944490354025923e-06, 'epoch': 6.04}

 60%|██████    | 441/730 [39:04<26:12,  5.44s/it]
 61%|██████    | 442/730 [39:05<25:44,  5.36s/it]
                                                 
{'debug/num_tok_total': 2855.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2855.0, 'debug/num_lat_loss': 1775.0, 'epoch': 6.05}

 61%|██████    | 442/730 [39:06<25:44,  5.36s/it]
                                                 
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.3460316061973572, 'epoch': 6.05}

 61%|██████    | 442/730 [39:06<25:44,  5.36s/it]
                                                 
{'train/learning_rate_real': 8.89135065825474e-06, 'epoch': 6.05}

 61%|██████    | 442/730 [39:06<25:44,  5.36s/it]
                                                 
{'debug/num_tok_total': 2649.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2649.0, 'debug/num_lat_loss': 1786.0, 'epoch': 6.05}

 61%|██████    | 442/730 [39:07<25:44,  5.36s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.39352497458457947, 'epoch': 6.05}

 61%|██████    | 442/730 [39:07<25:44,  5.36s/it]
                                                 
{'train/learning_rate_real': 8.89135065825474e-06, 'epoch': 6.05}

 61%|██████    | 442/730 [39:07<25:44,  5.36s/it]
                                                 
{'debug/num_tok_total': 2205.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2205.0, 'debug/num_lat_loss': 1779.0, 'epoch': 6.05}

 61%|██████    | 442/730 [39:08<25:44,  5.36s/it]
                                                 
{'train/ce_loss': 1.8515625, 'train/diffusion_loss': 0.5082336664199829, 'epoch': 6.05}

 61%|██████    | 442/730 [39:08<25:44,  5.36s/it]
                                                 
{'train/learning_rate_real': 8.89135065825474e-06, 'epoch': 6.05}

 61%|██████    | 442/730 [39:08<25:44,  5.36s/it]
                                                 
{'debug/num_tok_total': 3324.0, 'debug/num_tok_loss': 1798.0, 'debug/num_lat_total': 3324.0, 'debug/num_lat_loss': 1798.0, 'epoch': 6.05}

 61%|██████    | 442/730 [39:09<25:44,  5.36s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.2610774338245392, 'epoch': 6.05}

 61%|██████    | 442/730 [39:09<25:44,  5.36s/it]
                                                 
{'train/learning_rate_real': 8.89135065825474e-06, 'epoch': 6.05}

 61%|██████    | 442/730 [39:09<25:44,  5.36s/it]
 61%|██████    | 443/730 [39:10<25:43,  5.38s/it]
                                                 
{'debug/num_tok_total': 2647.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2647.0, 'debug/num_lat_loss': 1782.0, 'epoch': 6.07}

 61%|██████    | 443/730 [39:11<25:43,  5.38s/it]
                                                 
{'train/ce_loss': 1.9609375, 'train/diffusion_loss': 0.40903881192207336, 'epoch': 6.07}

 61%|██████    | 443/730 [39:11<25:43,  5.38s/it]
                                                 
{'train/learning_rate_real': 8.838282014629703e-06, 'epoch': 6.07}

 61%|██████    | 443/730 [39:11<25:43,  5.38s/it]
                                                 
{'debug/num_tok_total': 2447.0, 'debug/num_tok_loss': 1586.0, 'debug/num_lat_total': 2447.0, 'debug/num_lat_loss': 1586.0, 'epoch': 6.07}

 61%|██████    | 443/730 [39:12<25:43,  5.38s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.36582663655281067, 'epoch': 6.07}

 61%|██████    | 443/730 [39:12<25:43,  5.38s/it]
                                                 
{'train/learning_rate_real': 8.838282014629703e-06, 'epoch': 6.07}

 61%|██████    | 443/730 [39:12<25:43,  5.38s/it]
                                                 
{'debug/num_tok_total': 2901.0, 'debug/num_tok_loss': 1804.0, 'debug/num_lat_total': 2901.0, 'debug/num_lat_loss': 1804.0, 'epoch': 6.07}

 61%|██████    | 443/730 [39:14<25:43,  5.38s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.3556272089481354, 'epoch': 6.07}

 61%|██████    | 443/730 [39:14<25:43,  5.38s/it]
                                                 
{'train/learning_rate_real': 8.838282014629703e-06, 'epoch': 6.07}

 61%|██████    | 443/730 [39:14<25:43,  5.38s/it]
                                                 
{'debug/num_tok_total': 2079.0, 'debug/num_tok_loss': 1633.0, 'debug/num_lat_total': 2079.0, 'debug/num_lat_loss': 1633.0, 'epoch': 6.07}

 61%|██████    | 443/730 [39:15<25:43,  5.38s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.5122529864311218, 'epoch': 6.07}

 61%|██████    | 443/730 [39:15<25:43,  5.38s/it]
                                                 
{'train/learning_rate_real': 8.838282014629703e-06, 'epoch': 6.07}

 61%|██████    | 443/730 [39:15<25:43,  5.38s/it]
 61%|██████    | 444/730 [39:15<25:29,  5.35s/it]
                                                 
{'debug/num_tok_total': 3063.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 3063.0, 'debug/num_lat_loss': 1771.0, 'epoch': 6.08}

 61%|██████    | 444/730 [39:16<25:29,  5.35s/it]
                                                 
{'train/ce_loss': 1.8515625, 'train/diffusion_loss': 0.3180014193058014, 'epoch': 6.08}

 61%|██████    | 444/730 [39:16<25:29,  5.35s/it]
                                                 
{'train/learning_rate_real': 8.785285468040653e-06, 'epoch': 6.08}

 61%|██████    | 444/730 [39:16<25:29,  5.35s/it]
                                                 
{'debug/num_tok_total': 2871.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2871.0, 'debug/num_lat_loss': 1799.0, 'epoch': 6.08}

 61%|██████    | 444/730 [39:18<25:29,  5.35s/it]
                                                 
{'train/ce_loss': 1.90625, 'train/diffusion_loss': 0.3840492069721222, 'epoch': 6.08}

 61%|██████    | 444/730 [39:18<25:29,  5.35s/it]
                                                 
{'train/learning_rate_real': 8.785285468040653e-06, 'epoch': 6.08}

 61%|██████    | 444/730 [39:18<25:29,  5.35s/it]
                                                 
{'debug/num_tok_total': 2432.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2432.0, 'debug/num_lat_loss': 1783.0, 'epoch': 6.08}

 61%|██████    | 444/730 [39:19<25:29,  5.35s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.4349561035633087, 'epoch': 6.08}

 61%|██████    | 444/730 [39:19<25:29,  5.35s/it]
                                                 
{'train/learning_rate_real': 8.785285468040653e-06, 'epoch': 6.08}

 61%|██████    | 444/730 [39:19<25:29,  5.35s/it]
                                                 
{'debug/num_tok_total': 2222.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2222.0, 'debug/num_lat_loss': 1792.0, 'epoch': 6.08}

 61%|██████    | 444/730 [39:20<25:29,  5.35s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.49201059341430664, 'epoch': 6.08}

 61%|██████    | 444/730 [39:20<25:29,  5.35s/it]
                                                 
{'train/learning_rate_real': 8.785285468040653e-06, 'epoch': 6.08}

 61%|██████    | 444/730 [39:20<25:29,  5.35s/it]
 61%|██████    | 445/730 [39:21<25:29,  5.37s/it]
                                                 
{'debug/num_tok_total': 1802.0, 'debug/num_tok_loss': 1594.0, 'debug/num_lat_total': 1802.0, 'debug/num_lat_loss': 1594.0, 'epoch': 6.1}

 61%|██████    | 445/730 [39:22<25:29,  5.37s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.5246633291244507, 'epoch': 6.1}

 61%|██████    | 445/730 [39:22<25:29,  5.37s/it]
                                                 
{'train/learning_rate_real': 8.73236206195788e-06, 'epoch': 6.1}

 61%|██████    | 445/730 [39:22<25:29,  5.37s/it]
                                                 
{'debug/num_tok_total': 3082.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 3082.0, 'debug/num_lat_loss': 1786.0, 'epoch': 6.1}

 61%|██████    | 445/730 [39:23<25:29,  5.37s/it]
                                                 
{'train/ce_loss': 1.921875, 'train/diffusion_loss': 0.32416772842407227, 'epoch': 6.1}

 61%|██████    | 445/730 [39:23<25:29,  5.37s/it]
                                                 
{'train/learning_rate_real': 8.73236206195788e-06, 'epoch': 6.1}

 61%|██████    | 445/730 [39:23<25:29,  5.37s/it]
                                                 
{'debug/num_tok_total': 2193.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2193.0, 'debug/num_lat_loss': 1783.0, 'epoch': 6.1}

 61%|██████    | 445/730 [39:24<25:29,  5.37s/it]
                                                 
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.47183868288993835, 'epoch': 6.1}

 61%|██████    | 445/730 [39:24<25:29,  5.37s/it]
                                                 
{'train/learning_rate_real': 8.73236206195788e-06, 'epoch': 6.1}

 61%|██████    | 445/730 [39:24<25:29,  5.37s/it]
                                                 
{'debug/num_tok_total': 3336.0, 'debug/num_tok_loss': 1812.0, 'debug/num_lat_total': 3336.0, 'debug/num_lat_loss': 1812.0, 'epoch': 6.1}

 61%|██████    | 445/730 [39:25<25:29,  5.37s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.2603754997253418, 'epoch': 6.1}

 61%|██████    | 445/730 [39:26<25:29,  5.37s/it]
                                                 
{'train/learning_rate_real': 8.73236206195788e-06, 'epoch': 6.1}

 61%|██████    | 445/730 [39:26<25:29,  5.37s/it]
 61%|██████    | 446/730 [39:26<25:22,  5.36s/it]
                                                 
{'debug/num_tok_total': 2910.0, 'debug/num_tok_loss': 1807.0, 'debug/num_lat_total': 2910.0, 'debug/num_lat_loss': 1807.0, 'epoch': 6.11}

 61%|██████    | 446/730 [39:27<25:22,  5.36s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.3468431234359741, 'epoch': 6.11}

 61%|██████    | 446/730 [39:27<25:22,  5.36s/it]
                                                 
{'train/learning_rate_real': 8.679512838411575e-06, 'epoch': 6.11}

 61%|██████    | 446/730 [39:27<25:22,  5.36s/it]
                                                 
{'debug/num_tok_total': 2422.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2422.0, 'debug/num_lat_loss': 1781.0, 'epoch': 6.11}

 61%|██████    | 446/730 [39:28<25:22,  5.36s/it]
                                                 
{'train/ce_loss': 1.8984375, 'train/diffusion_loss': 0.44013121724128723, 'epoch': 6.11}

 61%|██████    | 446/730 [39:28<25:22,  5.36s/it]
                                                 
{'train/learning_rate_real': 8.679512838411575e-06, 'epoch': 6.11}

 61%|██████    | 446/730 [39:28<25:22,  5.36s/it]
                                                 
{'debug/num_tok_total': 2643.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2643.0, 'debug/num_lat_loss': 1786.0, 'epoch': 6.11}

 61%|██████    | 446/730 [39:30<25:22,  5.36s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.3910481333732605, 'epoch': 6.11}

 61%|██████    | 446/730 [39:30<25:22,  5.36s/it]
                                                 
{'train/learning_rate_real': 8.679512838411575e-06, 'epoch': 6.11}

 61%|██████    | 446/730 [39:30<25:22,  5.36s/it]
                                                 
{'debug/num_tok_total': 2646.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2646.0, 'debug/num_lat_loss': 1781.0, 'epoch': 6.11}

 61%|██████    | 446/730 [39:31<25:22,  5.36s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.41522571444511414, 'epoch': 6.11}

 61%|██████    | 446/730 [39:31<25:22,  5.36s/it]
                                                 
{'train/learning_rate_real': 8.679512838411575e-06, 'epoch': 6.11}

 61%|██████    | 446/730 [39:31<25:22,  5.36s/it]
 61%|██████    | 447/730 [39:31<25:21,  5.38s/it]
                                                 
{'debug/num_tok_total': 3008.0, 'debug/num_tok_loss': 1746.0, 'debug/num_lat_total': 3008.0, 'debug/num_lat_loss': 1746.0, 'epoch': 6.12}

 61%|██████    | 447/730 [39:33<25:21,  5.38s/it]
                                                 
{'train/ce_loss': 1.671875, 'train/diffusion_loss': 0.33093979954719543, 'epoch': 6.12}

 61%|██████    | 447/730 [39:33<25:21,  5.38s/it]
                                                 
{'train/learning_rate_real': 8.626738837971337e-06, 'epoch': 6.12}

 61%|██████    | 447/730 [39:33<25:21,  5.38s/it]
                                                 
{'debug/num_tok_total': 2621.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2621.0, 'debug/num_lat_loss': 1771.0, 'epoch': 6.12}

 61%|██████    | 447/730 [39:34<25:21,  5.38s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.40083298087120056, 'epoch': 6.12}

 61%|██████    | 447/730 [39:34<25:21,  5.38s/it]
                                                 
{'train/learning_rate_real': 8.626738837971337e-06, 'epoch': 6.12}

 61%|██████    | 447/730 [39:34<25:21,  5.38s/it]
                                                 
{'debug/num_tok_total': 2239.0, 'debug/num_tok_loss': 1598.0, 'debug/num_lat_total': 2239.0, 'debug/num_lat_loss': 1598.0, 'epoch': 6.12}

 61%|██████    | 447/730 [39:35<25:21,  5.38s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.41199246048927307, 'epoch': 6.12}

 61%|██████    | 447/730 [39:35<25:21,  5.38s/it]
                                                 
{'train/learning_rate_real': 8.626738837971337e-06, 'epoch': 6.12}

 61%|██████    | 447/730 [39:35<25:21,  5.38s/it]
                                                 
{'debug/num_tok_total': 2399.0, 'debug/num_tok_loss': 1756.0, 'debug/num_lat_total': 2399.0, 'debug/num_lat_loss': 1756.0, 'epoch': 6.12}

 61%|██████    | 447/730 [39:36<25:21,  5.38s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.4235708713531494, 'epoch': 6.12}

 61%|██████    | 447/730 [39:36<25:21,  5.38s/it]
                                                 
{'train/learning_rate_real': 8.626738837971337e-06, 'epoch': 6.12}

 61%|██████    | 447/730 [39:36<25:21,  5.38s/it]
 61%|██████▏   | 448/730 [39:37<25:00,  5.32s/it]
                                                 
{'debug/num_tok_total': 2203.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2203.0, 'debug/num_lat_loss': 1773.0, 'epoch': 6.14}

 61%|██████▏   | 448/730 [39:38<25:00,  5.32s/it]
                                                 
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.496637761592865, 'epoch': 6.14}

 61%|██████▏   | 448/730 [39:38<25:00,  5.32s/it]
                                                 
{'train/learning_rate_real': 8.574041099725644e-06, 'epoch': 6.14}

 61%|██████▏   | 448/730 [39:38<25:00,  5.32s/it]
                                                 
{'debug/num_tok_total': 3048.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 3048.0, 'debug/num_lat_loss': 1763.0, 'epoch': 6.14}

 61%|██████▏   | 448/730 [39:39<25:00,  5.32s/it]
                                                 
{'train/ce_loss': 1.625, 'train/diffusion_loss': 0.3144676089286804, 'epoch': 6.14}

 61%|██████▏   | 448/730 [39:39<25:00,  5.32s/it]
                                                 
{'train/learning_rate_real': 8.574041099725644e-06, 'epoch': 6.14}

 61%|██████▏   | 448/730 [39:39<25:00,  5.32s/it]
                                                 
{'debug/num_tok_total': 2488.0, 'debug/num_tok_loss': 1710.0, 'debug/num_lat_total': 2488.0, 'debug/num_lat_loss': 1710.0, 'epoch': 6.14}

 61%|██████▏   | 448/730 [39:40<25:00,  5.32s/it]
                                                 
{'train/ce_loss': 2.03125, 'train/diffusion_loss': 0.4127914607524872, 'epoch': 6.14}

 61%|██████▏   | 448/730 [39:40<25:00,  5.32s/it]
                                                 
{'train/learning_rate_real': 8.574041099725644e-06, 'epoch': 6.14}

 61%|██████▏   | 448/730 [39:40<25:00,  5.32s/it]
                                                 
{'debug/num_tok_total': 2658.0, 'debug/num_tok_loss': 1798.0, 'debug/num_lat_total': 2658.0, 'debug/num_lat_loss': 1798.0, 'epoch': 6.14}

 61%|██████▏   | 448/730 [39:41<25:00,  5.32s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.3918304443359375, 'epoch': 6.14}

 61%|██████▏   | 448/730 [39:41<25:00,  5.32s/it]
                                                 
{'train/learning_rate_real': 8.574041099725644e-06, 'epoch': 6.14}

 61%|██████▏   | 448/730 [39:41<25:00,  5.32s/it]
 62%|██████▏   | 449/730 [39:42<24:57,  5.33s/it]
                                                 
{'debug/num_tok_total': 2634.0, 'debug/num_tok_loss': 1666.0, 'debug/num_lat_total': 2634.0, 'debug/num_lat_loss': 1666.0, 'epoch': 6.15}

 62%|██████▏   | 449/730 [39:43<24:57,  5.33s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.3659103512763977, 'epoch': 6.15}

 62%|██████▏   | 449/730 [39:43<24:57,  5.33s/it]
                                                 
{'train/learning_rate_real': 8.521420661261426e-06, 'epoch': 6.15}

 62%|██████▏   | 449/730 [39:43<24:57,  5.33s/it]
                                                 
{'debug/num_tok_total': 2656.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2656.0, 'debug/num_lat_loss': 1791.0, 'epoch': 6.15}

 62%|██████▏   | 449/730 [39:44<24:57,  5.33s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.3985714018344879, 'epoch': 6.15}

 62%|██████▏   | 449/730 [39:44<24:57,  5.33s/it]
                                                 
{'train/learning_rate_real': 8.521420661261426e-06, 'epoch': 6.15}

 62%|██████▏   | 449/730 [39:44<24:57,  5.33s/it]
                                                 
{'debug/num_tok_total': 2245.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2245.0, 'debug/num_lat_loss': 1799.0, 'epoch': 6.15}

 62%|██████▏   | 449/730 [39:46<24:57,  5.33s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.4882946312427521, 'epoch': 6.15}

 62%|██████▏   | 449/730 [39:46<24:57,  5.33s/it]
                                                 
{'train/learning_rate_real': 8.521420661261426e-06, 'epoch': 6.15}

 62%|██████▏   | 449/730 [39:46<24:57,  5.33s/it]
                                                 
{'debug/num_tok_total': 2494.0, 'debug/num_tok_loss': 1827.0, 'debug/num_lat_total': 2494.0, 'debug/num_lat_loss': 1827.0, 'epoch': 6.15}

 62%|██████▏   | 449/730 [39:47<24:57,  5.33s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.4460631012916565, 'epoch': 6.15}

 62%|██████▏   | 449/730 [39:47<24:57,  5.33s/it]
                                                 
{'train/learning_rate_real': 8.521420661261426e-06, 'epoch': 6.15}

 62%|██████▏   | 449/730 [39:47<24:57,  5.33s/it]03/16/2026 07:27:19 - INFO - __main__ - LoRA debug step 450: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 62%|██████▏   | 450/730 [39:47<24:53,  5.33s/it]
                                                 
{'loss': 2.5244, 'grad_norm': 1.266939401626587, 'learning_rate': 8.521420661261426e-06, 'epoch': 6.16}

 62%|██████▏   | 450/730 [39:47<24:53,  5.33s/it]
                                                 
{'debug/num_tok_total': 2423.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2423.0, 'debug/num_lat_loss': 1774.0, 'epoch': 6.16}

 62%|██████▏   | 450/730 [39:48<24:53,  5.33s/it]
                                                 
{'train/ce_loss': 2.15625, 'train/diffusion_loss': 0.41751629114151, 'epoch': 6.16}

 62%|██████▏   | 450/730 [39:48<24:53,  5.33s/it]
                                                 
{'train/learning_rate_real': 8.468878558643639e-06, 'epoch': 6.16}

 62%|██████▏   | 450/730 [39:48<24:53,  5.33s/it]
                                                 
{'debug/num_tok_total': 2431.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2431.0, 'debug/num_lat_loss': 1786.0, 'epoch': 6.16}

 62%|██████▏   | 450/730 [39:50<24:53,  5.33s/it]
                                                 
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.4401915371417999, 'epoch': 6.16}

 62%|██████▏   | 450/730 [39:50<24:53,  5.33s/it]
                                                 
{'train/learning_rate_real': 8.468878558643639e-06, 'epoch': 6.16}

 62%|██████▏   | 450/730 [39:50<24:53,  5.33s/it]
                                                 
{'debug/num_tok_total': 2407.0, 'debug/num_tok_loss': 1747.0, 'debug/num_lat_total': 2407.0, 'debug/num_lat_loss': 1747.0, 'epoch': 6.16}

 62%|██████▏   | 450/730 [39:51<24:53,  5.33s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.42721492052078247, 'epoch': 6.16}

 62%|██████▏   | 450/730 [39:51<24:53,  5.33s/it]
                                                 
{'train/learning_rate_real': 8.468878558643639e-06, 'epoch': 6.16}

 62%|██████▏   | 450/730 [39:51<24:53,  5.33s/it]
                                                 
{'debug/num_tok_total': 3316.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 3316.0, 'debug/num_lat_loss': 1794.0, 'epoch': 6.16}

 62%|██████▏   | 450/730 [39:52<24:53,  5.33s/it]
                                                 
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.2643775939941406, 'epoch': 6.16}

 62%|██████▏   | 450/730 [39:52<24:53,  5.33s/it]
                                                 
{'train/learning_rate_real': 8.468878558643639e-06, 'epoch': 6.16}

 62%|██████▏   | 450/730 [39:52<24:53,  5.33s/it]
 62%|██████▏   | 451/730 [39:53<24:52,  5.35s/it]
                                                 
{'debug/num_tok_total': 2876.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2876.0, 'debug/num_lat_loss': 1791.0, 'epoch': 6.18}

 62%|██████▏   | 451/730 [39:54<24:52,  5.35s/it]
                                                 
{'train/ce_loss': 1.9140625, 'train/diffusion_loss': 0.3673725426197052, 'epoch': 6.18}

 62%|██████▏   | 451/730 [39:54<24:52,  5.35s/it]
                                                 
{'train/learning_rate_real': 8.416415826394838e-06, 'epoch': 6.18}

 62%|██████▏   | 451/730 [39:54<24:52,  5.35s/it]
                                                 
{'debug/num_tok_total': 2374.0, 'debug/num_tok_loss': 1654.0, 'debug/num_lat_total': 2374.0, 'debug/num_lat_loss': 1654.0, 'epoch': 6.18}

 62%|██████▏   | 451/730 [39:55<24:52,  5.35s/it]
                                                 
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.42454299330711365, 'epoch': 6.18}

 62%|██████▏   | 451/730 [39:55<24:52,  5.35s/it]
                                                 
{'train/learning_rate_real': 8.416415826394838e-06, 'epoch': 6.18}

 62%|██████▏   | 451/730 [39:55<24:52,  5.35s/it]
                                                 
{'debug/num_tok_total': 2814.0, 'debug/num_tok_loss': 1759.0, 'debug/num_lat_total': 2814.0, 'debug/num_lat_loss': 1759.0, 'epoch': 6.18}

 62%|██████▏   | 451/730 [39:56<24:52,  5.35s/it]
                                                 
{'train/ce_loss': 1.8828125, 'train/diffusion_loss': 0.34102559089660645, 'epoch': 6.18}

 62%|██████▏   | 451/730 [39:56<24:52,  5.35s/it]
                                                 
{'train/learning_rate_real': 8.416415826394838e-06, 'epoch': 6.18}

 62%|██████▏   | 451/730 [39:56<24:52,  5.35s/it]
                                                 
{'debug/num_tok_total': 2851.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2851.0, 'debug/num_lat_loss': 1775.0, 'epoch': 6.18}

 62%|██████▏   | 451/730 [39:58<24:52,  5.35s/it]
                                                 
{'train/ce_loss': 1.6640625, 'train/diffusion_loss': 0.34808066487312317, 'epoch': 6.18}

 62%|██████▏   | 451/730 [39:58<24:52,  5.35s/it]
                                                 
{'train/learning_rate_real': 8.416415826394838e-06, 'epoch': 6.18}

 62%|██████▏   | 451/730 [39:58<24:52,  5.35s/it]
 62%|██████▏   | 452/730 [39:58<25:04,  5.41s/it]
                                                 
{'debug/num_tok_total': 2432.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2432.0, 'debug/num_lat_loss': 1794.0, 'epoch': 6.19}

 62%|██████▏   | 452/730 [39:59<25:04,  5.41s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.4507027566432953, 'epoch': 6.19}

 62%|██████▏   | 452/730 [39:59<25:04,  5.41s/it]
                                                 
{'train/learning_rate_real': 8.36403349747483e-06, 'epoch': 6.19}

 62%|██████▏   | 452/730 [39:59<25:04,  5.41s/it]
                                                 
{'debug/num_tok_total': 2347.0, 'debug/num_tok_loss': 1727.0, 'debug/num_lat_total': 2347.0, 'debug/num_lat_loss': 1727.0, 'epoch': 6.19}

 62%|██████▏   | 452/730 [40:00<25:04,  5.41s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.4538610875606537, 'epoch': 6.19}

 62%|██████▏   | 452/730 [40:00<25:04,  5.41s/it]
                                                 
{'train/learning_rate_real': 8.36403349747483e-06, 'epoch': 6.19}

 62%|██████▏   | 452/730 [40:00<25:04,  5.41s/it]
                                                 
{'debug/num_tok_total': 3058.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 3058.0, 'debug/num_lat_loss': 1774.0, 'epoch': 6.19}

 62%|██████▏   | 452/730 [40:02<25:04,  5.41s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.3334881067276001, 'epoch': 6.19}

 62%|██████▏   | 452/730 [40:02<25:04,  5.41s/it]
                                                 
{'train/learning_rate_real': 8.36403349747483e-06, 'epoch': 6.19}

 62%|██████▏   | 452/730 [40:02<25:04,  5.41s/it]
                                                 
{'debug/num_tok_total': 2661.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2661.0, 'debug/num_lat_loss': 1791.0, 'epoch': 6.19}

 62%|██████▏   | 452/730 [40:03<25:04,  5.41s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.4047246277332306, 'epoch': 6.19}

 62%|██████▏   | 452/730 [40:03<25:04,  5.41s/it]
                                                 
{'train/learning_rate_real': 8.36403349747483e-06, 'epoch': 6.19}

 62%|██████▏   | 452/730 [40:03<25:04,  5.41s/it]
 62%|██████▏   | 453/730 [40:03<24:44,  5.36s/it]
                                                 
{'debug/num_tok_total': 2305.0, 'debug/num_tok_loss': 1425.0, 'debug/num_lat_total': 2305.0, 'debug/num_lat_loss': 1425.0, 'epoch': 6.21}

 62%|██████▏   | 453/730 [40:05<24:44,  5.36s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.3473239839076996, 'epoch': 6.21}

 62%|██████▏   | 453/730 [40:05<24:44,  5.36s/it]
                                                 
{'train/learning_rate_real': 8.31173260326032e-06, 'epoch': 6.21}

 62%|██████▏   | 453/730 [40:05<24:44,  5.36s/it]
                                                 
{'debug/num_tok_total': 2586.0, 'debug/num_tok_loss': 1645.0, 'debug/num_lat_total': 2586.0, 'debug/num_lat_loss': 1645.0, 'epoch': 6.21}

 62%|██████▏   | 453/730 [40:06<24:44,  5.36s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.3578820824623108, 'epoch': 6.21}

 62%|██████▏   | 453/730 [40:06<24:44,  5.36s/it]
                                                 
{'train/learning_rate_real': 8.31173260326032e-06, 'epoch': 6.21}

 62%|██████▏   | 453/730 [40:06<24:44,  5.36s/it]
                                                 
{'debug/num_tok_total': 2921.0, 'debug/num_tok_loss': 1707.0, 'debug/num_lat_total': 2921.0, 'debug/num_lat_loss': 1707.0, 'epoch': 6.21}

 62%|██████▏   | 453/730 [40:07<24:44,  5.36s/it]
                                                 
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.3198740780353546, 'epoch': 6.21}

 62%|██████▏   | 453/730 [40:07<24:44,  5.36s/it]
                                                 
{'train/learning_rate_real': 8.31173260326032e-06, 'epoch': 6.21}

 62%|██████▏   | 453/730 [40:07<24:44,  5.36s/it]
                                                 
{'debug/num_tok_total': 2120.0, 'debug/num_tok_loss': 1688.0, 'debug/num_lat_total': 2120.0, 'debug/num_lat_loss': 1688.0, 'epoch': 6.21}

 62%|██████▏   | 453/730 [40:08<24:44,  5.36s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.490401953458786, 'epoch': 6.21}

 62%|██████▏   | 453/730 [40:08<24:44,  5.36s/it]
                                                 
{'train/learning_rate_real': 8.31173260326032e-06, 'epoch': 6.21}

 62%|██████▏   | 453/730 [40:08<24:44,  5.36s/it]
 62%|██████▏   | 454/730 [40:09<24:43,  5.38s/it]
                                                 
{'debug/num_tok_total': 2802.0, 'debug/num_tok_loss': 1750.0, 'debug/num_lat_total': 2802.0, 'debug/num_lat_loss': 1750.0, 'epoch': 6.22}

 62%|██████▏   | 454/730 [40:10<24:43,  5.38s/it]
                                                 
{'train/ce_loss': 1.875, 'train/diffusion_loss': 0.34608662128448486, 'epoch': 6.22}

 62%|██████▏   | 454/730 [40:10<24:43,  5.38s/it]
                                                 
{'train/learning_rate_real': 8.25951417352462e-06, 'epoch': 6.22}

 62%|██████▏   | 454/730 [40:10<24:43,  5.38s/it]
                                                 
{'debug/num_tok_total': 2443.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2443.0, 'debug/num_lat_loss': 1790.0, 'epoch': 6.22}

 62%|██████▏   | 454/730 [40:11<24:43,  5.38s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.43620598316192627, 'epoch': 6.22}

 62%|██████▏   | 454/730 [40:11<24:43,  5.38s/it]
                                                 
{'train/learning_rate_real': 8.25951417352462e-06, 'epoch': 6.22}

 62%|██████▏   | 454/730 [40:11<24:43,  5.38s/it]
                                                 
{'debug/num_tok_total': 2013.0, 'debug/num_tok_loss': 1798.0, 'debug/num_lat_total': 2013.0, 'debug/num_lat_loss': 1798.0, 'epoch': 6.22}

 62%|██████▏   | 454/730 [40:12<24:43,  5.38s/it]
                                                 
{'train/ce_loss': 1.8828125, 'train/diffusion_loss': 0.5517037510871887, 'epoch': 6.22}

 62%|██████▏   | 454/730 [40:12<24:43,  5.38s/it]
                                                 
{'train/learning_rate_real': 8.25951417352462e-06, 'epoch': 6.22}

 62%|██████▏   | 454/730 [40:12<24:43,  5.38s/it]
                                                 
{'debug/num_tok_total': 3277.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 3277.0, 'debug/num_lat_loss': 1777.0, 'epoch': 6.22}

 62%|██████▏   | 454/730 [40:14<24:43,  5.38s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.29576587677001953, 'epoch': 6.22}

 62%|██████▏   | 454/730 [40:14<24:43,  5.38s/it]
                                                 
{'train/learning_rate_real': 8.25951417352462e-06, 'epoch': 6.22}

 62%|██████▏   | 454/730 [40:14<24:43,  5.38s/it]
 62%|██████▏   | 455/730 [40:14<24:27,  5.34s/it]
                                                 
{'debug/num_tok_total': 2928.0, 'debug/num_tok_loss': 1820.0, 'debug/num_lat_total': 2928.0, 'debug/num_lat_loss': 1820.0, 'epoch': 6.23}

 62%|██████▏   | 455/730 [40:15<24:27,  5.34s/it]
                                                 
{'train/ce_loss': 2.09375, 'train/diffusion_loss': 0.3570694923400879, 'epoch': 6.23}

 62%|██████▏   | 455/730 [40:15<24:27,  5.34s/it]
                                                 
{'train/learning_rate_real': 8.207379236417373e-06, 'epoch': 6.23}

 62%|██████▏   | 455/730 [40:15<24:27,  5.34s/it]
                                                 
{'debug/num_tok_total': 2636.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2636.0, 'debug/num_lat_loss': 1781.0, 'epoch': 6.23}

 62%|██████▏   | 455/730 [40:17<24:27,  5.34s/it]
                                                 
{'train/ce_loss': 1.6640625, 'train/diffusion_loss': 0.4634343683719635, 'epoch': 6.23}

 62%|██████▏   | 455/730 [40:17<24:27,  5.34s/it]
                                                 
{'train/learning_rate_real': 8.207379236417373e-06, 'epoch': 6.23}

 62%|██████▏   | 455/730 [40:17<24:27,  5.34s/it]
                                                 
{'debug/num_tok_total': 2434.0, 'debug/num_tok_loss': 1674.0, 'debug/num_lat_total': 2434.0, 'debug/num_lat_loss': 1674.0, 'epoch': 6.23}

 62%|██████▏   | 455/730 [40:18<24:27,  5.34s/it]
                                                 
{'train/ce_loss': 2.03125, 'train/diffusion_loss': 0.42419081926345825, 'epoch': 6.23}

 62%|██████▏   | 455/730 [40:18<24:27,  5.34s/it]
                                                 
{'train/learning_rate_real': 8.207379236417373e-06, 'epoch': 6.23}

 62%|██████▏   | 455/730 [40:18<24:27,  5.34s/it]
                                                 
{'debug/num_tok_total': 2713.0, 'debug/num_tok_loss': 1717.0, 'debug/num_lat_total': 2713.0, 'debug/num_lat_loss': 1717.0, 'epoch': 6.23}

 62%|██████▏   | 455/730 [40:19<24:27,  5.34s/it]
                                                 
{'train/ce_loss': 1.671875, 'train/diffusion_loss': 0.3845445513725281, 'epoch': 6.23}

 62%|██████▏   | 455/730 [40:19<24:27,  5.34s/it]
                                                 
{'train/learning_rate_real': 8.207379236417373e-06, 'epoch': 6.23}

 62%|██████▏   | 455/730 [40:19<24:27,  5.34s/it]
 62%|██████▏   | 456/730 [40:20<24:34,  5.38s/it]
                                                 
{'debug/num_tok_total': 2632.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2632.0, 'debug/num_lat_loss': 1783.0, 'epoch': 6.25}

 62%|██████▏   | 456/730 [40:21<24:34,  5.38s/it]
                                                 
{'train/ce_loss': 1.9609375, 'train/diffusion_loss': 0.3944592773914337, 'epoch': 6.25}

 62%|██████▏   | 456/730 [40:21<24:34,  5.38s/it]
                                                 
{'train/learning_rate_real': 8.155328818444287e-06, 'epoch': 6.25}

 62%|██████▏   | 456/730 [40:21<24:34,  5.38s/it]
                                                 
{'debug/num_tok_total': 2625.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2625.0, 'debug/num_lat_loss': 1766.0, 'epoch': 6.25}

 62%|██████▏   | 456/730 [40:22<24:34,  5.38s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.3933887183666229, 'epoch': 6.25}

 62%|██████▏   | 456/730 [40:22<24:34,  5.38s/it]
                                                 
{'train/learning_rate_real': 8.155328818444287e-06, 'epoch': 6.25}

 62%|██████▏   | 456/730 [40:22<24:34,  5.38s/it]
                                                 
{'debug/num_tok_total': 2849.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2849.0, 'debug/num_lat_loss': 1775.0, 'epoch': 6.25}

 62%|██████▏   | 456/730 [40:23<24:34,  5.38s/it]
                                                 
{'train/ce_loss': 1.671875, 'train/diffusion_loss': 0.38806024193763733, 'epoch': 6.25}

 62%|██████▏   | 456/730 [40:23<24:34,  5.38s/it]
                                                 
{'train/learning_rate_real': 8.155328818444287e-06, 'epoch': 6.25}

 62%|██████▏   | 456/730 [40:23<24:34,  5.38s/it]
                                                 
{'debug/num_tok_total': 2239.0, 'debug/num_tok_loss': 1811.0, 'debug/num_lat_total': 2239.0, 'debug/num_lat_loss': 1811.0, 'epoch': 6.25}

 62%|██████▏   | 456/730 [40:24<24:34,  5.38s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.48635128140449524, 'epoch': 6.25}

 62%|██████▏   | 456/730 [40:24<24:34,  5.38s/it]
                                                 
{'train/learning_rate_real': 8.155328818444287e-06, 'epoch': 6.25}

 62%|██████▏   | 456/730 [40:24<24:34,  5.38s/it]
 63%|██████▎   | 457/730 [40:25<24:18,  5.34s/it]
                                                 
{'debug/num_tok_total': 2406.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2406.0, 'debug/num_lat_loss': 1770.0, 'epoch': 6.26}

 63%|██████▎   | 457/730 [40:26<24:18,  5.34s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.45152610540390015, 'epoch': 6.26}

 63%|██████▎   | 457/730 [40:26<24:18,  5.34s/it]
                                                 
{'train/learning_rate_real': 8.103363944446937e-06, 'epoch': 6.26}

 63%|██████▎   | 457/730 [40:26<24:18,  5.34s/it]
                                                 
{'debug/num_tok_total': 2665.0, 'debug/num_tok_loss': 1798.0, 'debug/num_lat_total': 2665.0, 'debug/num_lat_loss': 1798.0, 'epoch': 6.26}

 63%|██████▎   | 457/730 [40:27<24:18,  5.34s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.36739060282707214, 'epoch': 6.26}

 63%|██████▎   | 457/730 [40:27<24:18,  5.34s/it]
                                                 
{'train/learning_rate_real': 8.103363944446937e-06, 'epoch': 6.26}

 63%|██████▎   | 457/730 [40:27<24:18,  5.34s/it]
                                                 
{'debug/num_tok_total': 3089.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 3089.0, 'debug/num_lat_loss': 1791.0, 'epoch': 6.26}

 63%|██████▎   | 457/730 [40:29<24:18,  5.34s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.32328924536705017, 'epoch': 6.26}

 63%|██████▎   | 457/730 [40:29<24:18,  5.34s/it]
                                                 
{'train/learning_rate_real': 8.103363944446937e-06, 'epoch': 6.26}

 63%|██████▎   | 457/730 [40:29<24:18,  5.34s/it]
                                                 
{'debug/num_tok_total': 2849.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2849.0, 'debug/num_lat_loss': 1776.0, 'epoch': 6.26}

 63%|██████▎   | 457/730 [40:30<24:18,  5.34s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.36214885115623474, 'epoch': 6.26}

 63%|██████▎   | 457/730 [40:30<24:18,  5.34s/it]
                                                 
{'train/learning_rate_real': 8.103363944446937e-06, 'epoch': 6.26}

 63%|██████▎   | 457/730 [40:30<24:18,  5.34s/it]
 63%|██████▎   | 458/730 [40:30<24:30,  5.41s/it]
                                                 
{'debug/num_tok_total': 2713.0, 'debug/num_tok_loss': 1623.0, 'debug/num_lat_total': 2713.0, 'debug/num_lat_loss': 1623.0, 'epoch': 6.27}

 63%|██████▎   | 458/730 [40:32<24:30,  5.41s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.3258262574672699, 'epoch': 6.27}

 63%|██████▎   | 458/730 [40:32<24:30,  5.41s/it]
                                                 
{'train/learning_rate_real': 8.051485637582614e-06, 'epoch': 6.27}

 63%|██████▎   | 458/730 [40:32<24:30,  5.41s/it]
                                                 
{'debug/num_tok_total': 2085.0, 'debug/num_tok_loss': 1660.0, 'debug/num_lat_total': 2085.0, 'debug/num_lat_loss': 1660.0, 'epoch': 6.27}

 63%|██████▎   | 458/730 [40:33<24:30,  5.41s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.5024404525756836, 'epoch': 6.27}

 63%|██████▎   | 458/730 [40:33<24:30,  5.41s/it]
                                                 
{'train/learning_rate_real': 8.051485637582614e-06, 'epoch': 6.27}

 63%|██████▎   | 458/730 [40:33<24:30,  5.41s/it]
                                                 
{'debug/num_tok_total': 2040.0, 'debug/num_tok_loss': 1817.0, 'debug/num_lat_total': 2040.0, 'debug/num_lat_loss': 1817.0, 'epoch': 6.27}

 63%|██████▎   | 458/730 [40:34<24:30,  5.41s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.521512508392334, 'epoch': 6.27}

 63%|██████▎   | 458/730 [40:34<24:30,  5.41s/it]
                                                 
{'train/learning_rate_real': 8.051485637582614e-06, 'epoch': 6.27}

 63%|██████▎   | 458/730 [40:34<24:30,  5.41s/it]
                                                 
{'debug/num_tok_total': 2447.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2447.0, 'debug/num_lat_loss': 1781.0, 'epoch': 6.27}

 63%|██████▎   | 458/730 [40:35<24:30,  5.41s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.45959824323654175, 'epoch': 6.27}

 63%|██████▎   | 458/730 [40:35<24:30,  5.41s/it]
                                                 
{'train/learning_rate_real': 8.051485637582614e-06, 'epoch': 6.27}

 63%|██████▎   | 458/730 [40:35<24:30,  5.41s/it]
 63%|██████▎   | 459/730 [40:36<24:13,  5.36s/it]
                                                 
{'debug/num_tok_total': 2610.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2610.0, 'debug/num_lat_loss': 1763.0, 'epoch': 6.29}

 63%|██████▎   | 459/730 [40:37<24:13,  5.36s/it]
                                                 
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.39619550108909607, 'epoch': 6.29}

 63%|██████▎   | 459/730 [40:37<24:13,  5.36s/it]
                                                 
{'train/learning_rate_real': 7.999694919304127e-06, 'epoch': 6.29}

 63%|██████▎   | 459/730 [40:37<24:13,  5.36s/it]
                                                 
{'debug/num_tok_total': 2857.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2857.0, 'debug/num_lat_loss': 1784.0, 'epoch': 6.29}

 63%|██████▎   | 459/730 [40:38<24:13,  5.36s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.36867836117744446, 'epoch': 6.29}

 63%|██████▎   | 459/730 [40:38<24:13,  5.36s/it]
                                                 
{'train/learning_rate_real': 7.999694919304127e-06, 'epoch': 6.29}

 63%|██████▎   | 459/730 [40:38<24:13,  5.36s/it]
                                                 
{'debug/num_tok_total': 2638.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2638.0, 'debug/num_lat_loss': 1780.0, 'epoch': 6.29}

 63%|██████▎   | 459/730 [40:39<24:13,  5.36s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.38969263434410095, 'epoch': 6.29}

 63%|██████▎   | 459/730 [40:39<24:13,  5.36s/it]
                                                 
{'train/learning_rate_real': 7.999694919304127e-06, 'epoch': 6.29}

 63%|██████▎   | 459/730 [40:39<24:13,  5.36s/it]
                                                 
{'debug/num_tok_total': 2439.0, 'debug/num_tok_loss': 1800.0, 'debug/num_lat_total': 2439.0, 'debug/num_lat_loss': 1800.0, 'epoch': 6.29}

 63%|██████▎   | 459/730 [40:40<24:13,  5.36s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.4596479833126068, 'epoch': 6.29}

 63%|██████▎   | 459/730 [40:40<24:13,  5.36s/it]
                                                 
{'train/learning_rate_real': 7.999694919304127e-06, 'epoch': 6.29}

 63%|██████▎   | 459/730 [40:40<24:13,  5.36s/it]03/16/2026 07:28:13 - INFO - __main__ - LoRA debug step 460: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 63%|██████▎   | 460/730 [40:41<24:01,  5.34s/it]
                                                 
{'loss': 2.5348, 'grad_norm': 1.1804755926132202, 'learning_rate': 7.999694919304127e-06, 'epoch': 6.3}

 63%|██████▎   | 460/730 [40:41<24:01,  5.34s/it]
                                                 
{'debug/num_tok_total': 2841.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2841.0, 'debug/num_lat_loss': 1778.0, 'epoch': 6.3}

 63%|██████▎   | 460/730 [40:42<24:01,  5.34s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.34258580207824707, 'epoch': 6.3}

 63%|██████▎   | 460/730 [40:42<24:01,  5.34s/it]
                                                 
{'train/learning_rate_real': 7.947992809339723e-06, 'epoch': 6.3}

 63%|██████▎   | 460/730 [40:42<24:01,  5.34s/it]
                                                 
{'debug/num_tok_total': 2382.0, 'debug/num_tok_loss': 1755.0, 'debug/num_lat_total': 2382.0, 'debug/num_lat_loss': 1755.0, 'epoch': 6.3}

 63%|██████▎   | 460/730 [40:43<24:01,  5.34s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.434190958738327, 'epoch': 6.3}

 63%|██████▎   | 460/730 [40:43<24:01,  5.34s/it]
                                                 
{'train/learning_rate_real': 7.947992809339723e-06, 'epoch': 6.3}

 63%|██████▎   | 460/730 [40:43<24:01,  5.34s/it]
                                                 
{'debug/num_tok_total': 2493.0, 'debug/num_tok_loss': 1641.0, 'debug/num_lat_total': 2493.0, 'debug/num_lat_loss': 1641.0, 'epoch': 6.3}

 63%|██████▎   | 460/730 [40:44<24:01,  5.34s/it]
                                                 
{'train/ce_loss': 1.640625, 'train/diffusion_loss': 0.35652434825897217, 'epoch': 6.3}

 63%|██████▎   | 460/730 [40:44<24:01,  5.34s/it]
                                                 
{'train/learning_rate_real': 7.947992809339723e-06, 'epoch': 6.3}

 63%|██████▎   | 460/730 [40:44<24:01,  5.34s/it]
                                                 
{'debug/num_tok_total': 2411.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2411.0, 'debug/num_lat_loss': 1773.0, 'epoch': 6.3}

 63%|██████▎   | 460/730 [40:46<24:01,  5.34s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.45423412322998047, 'epoch': 6.3}

 63%|██████▎   | 460/730 [40:46<24:01,  5.34s/it]
                                                 
{'train/learning_rate_real': 7.947992809339723e-06, 'epoch': 6.3}

 63%|██████▎   | 460/730 [40:46<24:01,  5.34s/it]
 63%|██████▎   | 461/730 [40:46<23:41,  5.28s/it]
                                                 
{'debug/num_tok_total': 2658.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2658.0, 'debug/num_lat_loss': 1789.0, 'epoch': 6.32}

 63%|██████▎   | 461/730 [40:47<23:41,  5.28s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.3966357707977295, 'epoch': 6.32}

 63%|██████▎   | 461/730 [40:47<23:41,  5.28s/it]
                                                 
{'train/learning_rate_real': 7.896380325673018e-06, 'epoch': 6.32}

 63%|██████▎   | 461/730 [40:47<23:41,  5.28s/it]
                                                 
{'debug/num_tok_total': 2655.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2655.0, 'debug/num_lat_loss': 1787.0, 'epoch': 6.32}

 63%|██████▎   | 461/730 [40:48<23:41,  5.28s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.3964393436908722, 'epoch': 6.32}

 63%|██████▎   | 461/730 [40:48<23:41,  5.28s/it]
                                                 
{'train/learning_rate_real': 7.896380325673018e-06, 'epoch': 6.32}

 63%|██████▎   | 461/730 [40:48<23:41,  5.28s/it]
                                                 
{'debug/num_tok_total': 2392.0, 'debug/num_tok_loss': 1758.0, 'debug/num_lat_total': 2392.0, 'debug/num_lat_loss': 1758.0, 'epoch': 6.32}

 63%|██████▎   | 461/730 [40:50<23:41,  5.28s/it]
                                                 
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.44162872433662415, 'epoch': 6.32}

 63%|██████▎   | 461/730 [40:50<23:41,  5.28s/it]
                                                 
{'train/learning_rate_real': 7.896380325673018e-06, 'epoch': 6.32}

 63%|██████▎   | 461/730 [40:50<23:41,  5.28s/it]
                                                 
{'debug/num_tok_total': 2847.0, 'debug/num_tok_loss': 1575.0, 'debug/num_lat_total': 2847.0, 'debug/num_lat_loss': 1575.0, 'epoch': 6.32}

 63%|██████▎   | 461/730 [40:51<23:41,  5.28s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.307466983795166, 'epoch': 6.32}

 63%|██████▎   | 461/730 [40:51<23:41,  5.28s/it]
                                                 
{'train/learning_rate_real': 7.896380325673018e-06, 'epoch': 6.32}

 63%|██████▎   | 461/730 [40:51<23:41,  5.28s/it]
 63%|██████▎   | 462/730 [40:51<23:37,  5.29s/it]
                                                 
{'debug/num_tok_total': 2649.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2649.0, 'debug/num_lat_loss': 1782.0, 'epoch': 6.33}

 63%|██████▎   | 462/730 [40:53<23:37,  5.29s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.3796628713607788, 'epoch': 6.33}

 63%|██████▎   | 462/730 [40:53<23:37,  5.29s/it]
                                                 
{'train/learning_rate_real': 7.844858484522924e-06, 'epoch': 6.33}

 63%|██████▎   | 462/730 [40:53<23:37,  5.29s/it]
                                                 
{'debug/num_tok_total': 2612.0, 'debug/num_tok_loss': 1758.0, 'debug/num_lat_total': 2612.0, 'debug/num_lat_loss': 1758.0, 'epoch': 6.33}

 63%|██████▎   | 462/730 [40:54<23:37,  5.29s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.394775927066803, 'epoch': 6.33}

 63%|██████▎   | 462/730 [40:54<23:37,  5.29s/it]
                                                 
{'train/learning_rate_real': 7.844858484522924e-06, 'epoch': 6.33}

 63%|██████▎   | 462/730 [40:54<23:37,  5.29s/it]
                                                 
{'debug/num_tok_total': 2493.0, 'debug/num_tok_loss': 1620.0, 'debug/num_lat_total': 2493.0, 'debug/num_lat_loss': 1620.0, 'epoch': 6.33}

 63%|██████▎   | 462/730 [40:55<23:37,  5.29s/it]
                                                 
{'train/ce_loss': 1.90625, 'train/diffusion_loss': 0.37308046221733093, 'epoch': 6.33}

 63%|██████▎   | 462/730 [40:55<23:37,  5.29s/it]
                                                 
{'train/learning_rate_real': 7.844858484522924e-06, 'epoch': 6.33}

 63%|██████▎   | 462/730 [40:55<23:37,  5.29s/it]
                                                 
{'debug/num_tok_total': 2352.0, 'debug/num_tok_loss': 1714.0, 'debug/num_lat_total': 2352.0, 'debug/num_lat_loss': 1714.0, 'epoch': 6.33}

 63%|██████▎   | 462/730 [40:56<23:37,  5.29s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.4361802041530609, 'epoch': 6.33}

 63%|██████▎   | 462/730 [40:56<23:37,  5.29s/it]
                                                 
{'train/learning_rate_real': 7.844858484522924e-06, 'epoch': 6.33}

 63%|██████▎   | 462/730 [40:56<23:37,  5.29s/it]
 63%|██████▎   | 463/730 [40:57<23:31,  5.29s/it]
                                                 
{'debug/num_tok_total': 2814.0, 'debug/num_tok_loss': 1750.0, 'debug/num_lat_total': 2814.0, 'debug/num_lat_loss': 1750.0, 'epoch': 6.34}

 63%|██████▎   | 463/730 [40:58<23:31,  5.29s/it]
                                                 
{'train/ce_loss': 1.875, 'train/diffusion_loss': 0.3582160174846649, 'epoch': 6.34}

 63%|██████▎   | 463/730 [40:58<23:31,  5.29s/it]
                                                 
{'train/learning_rate_real': 7.793428300323672e-06, 'epoch': 6.34}

 63%|██████▎   | 463/730 [40:58<23:31,  5.29s/it]
                                                 
{'debug/num_tok_total': 2618.0, 'debug/num_tok_loss': 1569.0, 'debug/num_lat_total': 2618.0, 'debug/num_lat_loss': 1569.0, 'epoch': 6.34}

 63%|██████▎   | 463/730 [40:59<23:31,  5.29s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.34685248136520386, 'epoch': 6.34}

 63%|██████▎   | 463/730 [40:59<23:31,  5.29s/it]
                                                 
{'train/learning_rate_real': 7.793428300323672e-06, 'epoch': 6.34}

 63%|██████▎   | 463/730 [40:59<23:31,  5.29s/it]
                                                 
{'debug/num_tok_total': 2637.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2637.0, 'debug/num_lat_loss': 1786.0, 'epoch': 6.34}

 63%|██████▎   | 463/730 [41:00<23:31,  5.29s/it]
                                                 
{'train/ce_loss': 1.96875, 'train/diffusion_loss': 0.3918110728263855, 'epoch': 6.34}

 63%|██████▎   | 463/730 [41:00<23:31,  5.29s/it]
                                                 
{'train/learning_rate_real': 7.793428300323672e-06, 'epoch': 6.34}

 63%|██████▎   | 463/730 [41:00<23:31,  5.29s/it]
                                                 
{'debug/num_tok_total': 2520.0, 'debug/num_tok_loss': 1604.0, 'debug/num_lat_total': 2520.0, 'debug/num_lat_loss': 1604.0, 'epoch': 6.34}

 63%|██████▎   | 463/730 [41:02<23:31,  5.29s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.3743284046649933, 'epoch': 6.34}

 63%|██████▎   | 463/730 [41:02<23:31,  5.29s/it]
                                                 
{'train/learning_rate_real': 7.793428300323672e-06, 'epoch': 6.34}

 63%|██████▎   | 463/730 [41:02<23:31,  5.29s/it]
 64%|██████▎   | 464/730 [41:02<23:38,  5.33s/it]
                                                 
{'debug/num_tok_total': 2379.0, 'debug/num_tok_loss': 1754.0, 'debug/num_lat_total': 2379.0, 'debug/num_lat_loss': 1754.0, 'epoch': 6.36}

 64%|██████▎   | 464/730 [41:03<23:38,  5.33s/it]
                                                 
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.44991886615753174, 'epoch': 6.36}

 64%|██████▎   | 464/730 [41:03<23:38,  5.33s/it]
                                                 
{'train/learning_rate_real': 7.742090785704807e-06, 'epoch': 6.36}

 64%|██████▎   | 464/730 [41:03<23:38,  5.33s/it]
                                                 
{'debug/num_tok_total': 2404.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2404.0, 'debug/num_lat_loss': 1771.0, 'epoch': 6.36}

 64%|██████▎   | 464/730 [41:04<23:38,  5.33s/it]
                                                 
{'train/ce_loss': 2.09375, 'train/diffusion_loss': 0.42065078020095825, 'epoch': 6.36}

 64%|██████▎   | 464/730 [41:04<23:38,  5.33s/it]
                                                 
{'train/learning_rate_real': 7.742090785704807e-06, 'epoch': 6.36}

 64%|██████▎   | 464/730 [41:04<23:38,  5.33s/it]
                                                 
{'debug/num_tok_total': 2198.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2198.0, 'debug/num_lat_loss': 1764.0, 'epoch': 6.36}

 64%|██████▎   | 464/730 [41:06<23:38,  5.33s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.4649135172367096, 'epoch': 6.36}

 64%|██████▎   | 464/730 [41:06<23:38,  5.33s/it]
                                                 
{'train/learning_rate_real': 7.742090785704807e-06, 'epoch': 6.36}

 64%|██████▎   | 464/730 [41:06<23:38,  5.33s/it]
                                                 
{'debug/num_tok_total': 2425.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2425.0, 'debug/num_lat_loss': 1780.0, 'epoch': 6.36}

 64%|██████▎   | 464/730 [41:07<23:38,  5.33s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.4578315019607544, 'epoch': 6.36}

 64%|██████▎   | 464/730 [41:07<23:38,  5.33s/it]
                                                 
{'train/learning_rate_real': 7.742090785704807e-06, 'epoch': 6.36}

 64%|██████▎   | 464/730 [41:07<23:38,  5.33s/it]
 64%|██████▎   | 465/730 [41:07<23:10,  5.25s/it]
                                                 
{'debug/num_tok_total': 2834.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 2834.0, 'debug/num_lat_loss': 1768.0, 'epoch': 6.37}

 64%|██████▎   | 465/730 [41:08<23:10,  5.25s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.35330730676651, 'epoch': 6.37}

 64%|██████▎   | 465/730 [41:08<23:10,  5.25s/it]
                                                 
{'train/learning_rate_real': 7.690846951471268e-06, 'epoch': 6.37}

 64%|██████▎   | 465/730 [41:08<23:10,  5.25s/it]
                                                 
{'debug/num_tok_total': 2449.0, 'debug/num_tok_loss': 1813.0, 'debug/num_lat_total': 2449.0, 'debug/num_lat_loss': 1813.0, 'epoch': 6.37}

 64%|██████▎   | 465/730 [41:10<23:10,  5.25s/it]
                                                 
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.47506776452064514, 'epoch': 6.37}

 64%|██████▎   | 465/730 [41:10<23:10,  5.25s/it]
                                                 
{'train/learning_rate_real': 7.690846951471268e-06, 'epoch': 6.37}

 64%|██████▎   | 465/730 [41:10<23:10,  5.25s/it]
                                                 
{'debug/num_tok_total': 3080.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 3080.0, 'debug/num_lat_loss': 1785.0, 'epoch': 6.37}

 64%|██████▎   | 465/730 [41:11<23:10,  5.25s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.33262479305267334, 'epoch': 6.37}

 64%|██████▎   | 465/730 [41:11<23:10,  5.25s/it]
                                                 
{'train/learning_rate_real': 7.690846951471268e-06, 'epoch': 6.37}

 64%|██████▎   | 465/730 [41:11<23:10,  5.25s/it]
                                                 
{'debug/num_tok_total': 2112.0, 'debug/num_tok_loss': 1568.0, 'debug/num_lat_total': 2112.0, 'debug/num_lat_loss': 1568.0, 'epoch': 6.37}

 64%|██████▎   | 465/730 [41:12<23:10,  5.25s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.45155832171440125, 'epoch': 6.37}

 64%|██████▎   | 465/730 [41:12<23:10,  5.25s/it]
                                                 
{'train/learning_rate_real': 7.690846951471268e-06, 'epoch': 6.37}

 64%|██████▎   | 465/730 [41:12<23:10,  5.25s/it]
 64%|██████▍   | 466/730 [41:13<23:12,  5.28s/it]
                                                 
{'debug/num_tok_total': 2406.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2406.0, 'debug/num_lat_loss': 1763.0, 'epoch': 6.38}

 64%|██████▍   | 466/730 [41:14<23:12,  5.28s/it]
                                                 
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.47152021527290344, 'epoch': 6.38}

 64%|██████▍   | 466/730 [41:14<23:12,  5.28s/it]
                                                 
{'train/learning_rate_real': 7.639697806583493e-06, 'epoch': 6.38}

 64%|██████▍   | 466/730 [41:14<23:12,  5.28s/it]
                                                 
{'debug/num_tok_total': 2188.0, 'debug/num_tok_loss': 1762.0, 'debug/num_lat_total': 2188.0, 'debug/num_lat_loss': 1762.0, 'epoch': 6.38}

 64%|██████▍   | 466/730 [41:15<23:12,  5.28s/it]
                                                 
{'train/ce_loss': 2.03125, 'train/diffusion_loss': 0.4641861319541931, 'epoch': 6.38}

 64%|██████▍   | 466/730 [41:15<23:12,  5.28s/it]
                                                 
{'train/learning_rate_real': 7.639697806583493e-06, 'epoch': 6.38}

 64%|██████▍   | 466/730 [41:15<23:12,  5.28s/it]
                                                 
{'debug/num_tok_total': 2430.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2430.0, 'debug/num_lat_loss': 1787.0, 'epoch': 6.38}

 64%|██████▍   | 466/730 [41:16<23:12,  5.28s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.4269067347049713, 'epoch': 6.38}

 64%|██████▍   | 466/730 [41:16<23:12,  5.28s/it]
                                                 
{'train/learning_rate_real': 7.639697806583493e-06, 'epoch': 6.38}

 64%|██████▍   | 466/730 [41:16<23:12,  5.28s/it]
                                                 
{'debug/num_tok_total': 2855.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2855.0, 'debug/num_lat_loss': 1781.0, 'epoch': 6.38}

 64%|██████▍   | 466/730 [41:17<23:12,  5.28s/it]
                                                 
{'train/ce_loss': 2.03125, 'train/diffusion_loss': 0.3621404767036438, 'epoch': 6.38}

 64%|██████▍   | 466/730 [41:17<23:12,  5.28s/it]
                                                 
{'train/learning_rate_real': 7.639697806583493e-06, 'epoch': 6.38}

 64%|██████▍   | 466/730 [41:17<23:12,  5.28s/it]
 64%|██████▍   | 467/730 [41:18<22:58,  5.24s/it]
                                                 
{'debug/num_tok_total': 2879.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 2879.0, 'debug/num_lat_loss': 1801.0, 'epoch': 6.4}

 64%|██████▍   | 467/730 [41:19<22:58,  5.24s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.348606139421463, 'epoch': 6.4}

 64%|██████▍   | 467/730 [41:19<22:58,  5.24s/it]
                                                 
{'train/learning_rate_real': 7.588644358137539e-06, 'epoch': 6.4}

 64%|██████▍   | 467/730 [41:19<22:58,  5.24s/it]
                                                 
{'debug/num_tok_total': 3080.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 3080.0, 'debug/num_lat_loss': 1784.0, 'epoch': 6.4}

 64%|██████▍   | 467/730 [41:20<22:58,  5.24s/it]
                                                 
{'train/ce_loss': 1.9140625, 'train/diffusion_loss': 0.3045468032360077, 'epoch': 6.4}

 64%|██████▍   | 467/730 [41:20<22:58,  5.24s/it]
                                                 
{'train/learning_rate_real': 7.588644358137539e-06, 'epoch': 6.4}

 64%|██████▍   | 467/730 [41:20<22:58,  5.24s/it]
                                                 
{'debug/num_tok_total': 2317.0, 'debug/num_tok_loss': 1662.0, 'debug/num_lat_total': 2317.0, 'debug/num_lat_loss': 1662.0, 'epoch': 6.4}

 64%|██████▍   | 467/730 [41:21<22:58,  5.24s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.4378036558628082, 'epoch': 6.4}

 64%|██████▍   | 467/730 [41:21<22:58,  5.24s/it]
                                                 
{'train/learning_rate_real': 7.588644358137539e-06, 'epoch': 6.4}

 64%|██████▍   | 467/730 [41:21<22:58,  5.24s/it]
                                                 
{'debug/num_tok_total': 2831.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2831.0, 'debug/num_lat_loss': 1773.0, 'epoch': 6.4}

 64%|██████▍   | 467/730 [41:23<22:58,  5.24s/it]
                                                 
{'train/ce_loss': 1.8828125, 'train/diffusion_loss': 0.3627336919307709, 'epoch': 6.4}

 64%|██████▍   | 467/730 [41:23<22:58,  5.24s/it]
                                                 
{'train/learning_rate_real': 7.588644358137539e-06, 'epoch': 6.4}

 64%|██████▍   | 467/730 [41:23<22:58,  5.24s/it]
 64%|██████▍   | 468/730 [41:23<23:02,  5.28s/it]
                                                 
{'debug/num_tok_total': 2864.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2864.0, 'debug/num_lat_loss': 1784.0, 'epoch': 6.41}

 64%|██████▍   | 468/730 [41:24<23:02,  5.28s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.37544751167297363, 'epoch': 6.41}

 64%|██████▍   | 468/730 [41:24<23:02,  5.28s/it]
                                                 
{'train/learning_rate_real': 7.537687611345248e-06, 'epoch': 6.41}

 64%|██████▍   | 468/730 [41:24<23:02,  5.28s/it]
                                                 
{'debug/num_tok_total': 2979.0, 'debug/num_tok_loss': 1624.0, 'debug/num_lat_total': 2979.0, 'debug/num_lat_loss': 1624.0, 'epoch': 6.41}

 64%|██████▍   | 468/730 [41:26<23:02,  5.28s/it]
                                                 
{'train/ce_loss': 2.171875, 'train/diffusion_loss': 0.28546202182769775, 'epoch': 6.41}

 64%|██████▍   | 468/730 [41:26<23:02,  5.28s/it]
                                                 
{'train/learning_rate_real': 7.537687611345248e-06, 'epoch': 6.41}

 64%|██████▍   | 468/730 [41:26<23:02,  5.28s/it]
                                                 
{'debug/num_tok_total': 3312.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 3312.0, 'debug/num_lat_loss': 1799.0, 'epoch': 6.41}

 64%|██████▍   | 468/730 [41:27<23:02,  5.28s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.2472393810749054, 'epoch': 6.41}

 64%|██████▍   | 468/730 [41:27<23:02,  5.28s/it]
                                                 
{'train/learning_rate_real': 7.537687611345248e-06, 'epoch': 6.41}

 64%|██████▍   | 468/730 [41:27<23:02,  5.28s/it]
                                                 
{'debug/num_tok_total': 2874.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2874.0, 'debug/num_lat_loss': 1794.0, 'epoch': 6.41}

 64%|██████▍   | 468/730 [41:28<23:02,  5.28s/it]
                                                 
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.3382861018180847, 'epoch': 6.41}

 64%|██████▍   | 468/730 [41:28<23:02,  5.28s/it]
                                                 
{'train/learning_rate_real': 7.537687611345248e-06, 'epoch': 6.41}

 64%|██████▍   | 468/730 [41:28<23:02,  5.28s/it]
 64%|██████▍   | 469/730 [41:29<23:35,  5.42s/it]
                                                 
{'debug/num_tok_total': 3271.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 3271.0, 'debug/num_lat_loss': 1769.0, 'epoch': 6.42}

 64%|██████▍   | 469/730 [41:30<23:35,  5.42s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.3048734962940216, 'epoch': 6.42}

 64%|██████▍   | 469/730 [41:30<23:35,  5.42s/it]
                                                 
{'train/learning_rate_real': 7.486828569514482e-06, 'epoch': 6.42}

 64%|██████▍   | 469/730 [41:30<23:35,  5.42s/it]
                                                 
{'debug/num_tok_total': 3034.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 3034.0, 'debug/num_lat_loss': 1767.0, 'epoch': 6.42}

 64%|██████▍   | 469/730 [41:31<23:35,  5.42s/it]
                                                 
{'train/ce_loss': 1.8828125, 'train/diffusion_loss': 0.3138163685798645, 'epoch': 6.42}

 64%|██████▍   | 469/730 [41:31<23:35,  5.42s/it]
                                                 
{'train/learning_rate_real': 7.486828569514482e-06, 'epoch': 6.42}

 64%|██████▍   | 469/730 [41:31<23:35,  5.42s/it]
                                                 
{'debug/num_tok_total': 2430.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2430.0, 'debug/num_lat_loss': 1777.0, 'epoch': 6.42}

 64%|██████▍   | 469/730 [41:33<23:35,  5.42s/it]
                                                 
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.4463663399219513, 'epoch': 6.42}

 64%|██████▍   | 469/730 [41:33<23:35,  5.42s/it]
                                                 
{'train/learning_rate_real': 7.486828569514482e-06, 'epoch': 6.42}

 64%|██████▍   | 469/730 [41:33<23:35,  5.42s/it]
                                                 
{'debug/num_tok_total': 2894.0, 'debug/num_tok_loss': 1805.0, 'debug/num_lat_total': 2894.0, 'debug/num_lat_loss': 1805.0, 'epoch': 6.42}

 64%|██████▍   | 469/730 [41:34<23:35,  5.42s/it]
                                                 
{'train/ce_loss': 1.890625, 'train/diffusion_loss': 0.37263527512550354, 'epoch': 6.42}

 64%|██████▍   | 469/730 [41:34<23:35,  5.42s/it]
                                                 
{'train/learning_rate_real': 7.486828569514482e-06, 'epoch': 6.42}

 64%|██████▍   | 469/730 [41:34<23:35,  5.42s/it]03/16/2026 07:29:06 - INFO - __main__ - LoRA debug step 470: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 64%|██████▍   | 470/730 [41:34<23:43,  5.48s/it]
                                                 
{'loss': 2.4553, 'grad_norm': 1.241637110710144, 'learning_rate': 7.486828569514482e-06, 'epoch': 6.44}

 64%|██████▍   | 470/730 [41:34<23:43,  5.48s/it]
                                                 
{'debug/num_tok_total': 3264.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 3264.0, 'debug/num_lat_loss': 1773.0, 'epoch': 6.44}

 64%|██████▍   | 470/730 [41:36<23:43,  5.48s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.2671394646167755, 'epoch': 6.44}

 64%|██████▍   | 470/730 [41:36<23:43,  5.48s/it]
                                                 
{'train/learning_rate_real': 7.436068234029331e-06, 'epoch': 6.44}

 64%|██████▍   | 470/730 [41:36<23:43,  5.48s/it]
                                                 
{'debug/num_tok_total': 2414.0, 'debug/num_tok_loss': 1761.0, 'debug/num_lat_total': 2414.0, 'debug/num_lat_loss': 1761.0, 'epoch': 6.44}

 64%|██████▍   | 470/730 [41:37<23:43,  5.48s/it]
                                                 
{'train/ce_loss': 1.8515625, 'train/diffusion_loss': 0.4374586343765259, 'epoch': 6.44}

 64%|██████▍   | 470/730 [41:37<23:43,  5.48s/it]
                                                 
{'train/learning_rate_real': 7.436068234029331e-06, 'epoch': 6.44}

 64%|██████▍   | 470/730 [41:37<23:43,  5.48s/it]
                                                 
{'debug/num_tok_total': 2859.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2859.0, 'debug/num_lat_loss': 1780.0, 'epoch': 6.44}

 64%|██████▍   | 470/730 [41:38<23:43,  5.48s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.35167402029037476, 'epoch': 6.44}

 64%|██████▍   | 470/730 [41:38<23:43,  5.48s/it]
                                                 
{'train/learning_rate_real': 7.436068234029331e-06, 'epoch': 6.44}

 64%|██████▍   | 470/730 [41:38<23:43,  5.48s/it]
                                                 
{'debug/num_tok_total': 2861.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2861.0, 'debug/num_lat_loss': 1781.0, 'epoch': 6.44}

 64%|██████▍   | 470/730 [41:40<23:43,  5.48s/it]
                                                 
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.34927552938461304, 'epoch': 6.44}

 64%|██████▍   | 470/730 [41:40<23:43,  5.48s/it]
                                                 
{'train/learning_rate_real': 7.436068234029331e-06, 'epoch': 6.44}

 64%|██████▍   | 470/730 [41:40<23:43,  5.48s/it]
 65%|██████▍   | 471/730 [41:40<23:56,  5.55s/it]
                                                 
{'debug/num_tok_total': 3101.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 3101.0, 'debug/num_lat_loss': 1797.0, 'epoch': 6.45}

 65%|██████▍   | 471/730 [41:41<23:56,  5.55s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.2879214584827423, 'epoch': 6.45}

 65%|██████▍   | 471/730 [41:41<23:56,  5.55s/it]
                                                 
{'train/learning_rate_real': 7.385407604330436e-06, 'epoch': 6.45}

 65%|██████▍   | 471/730 [41:41<23:56,  5.55s/it]
                                                 
{'debug/num_tok_total': 2651.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2651.0, 'debug/num_lat_loss': 1783.0, 'epoch': 6.45}

 65%|██████▍   | 471/730 [41:43<23:56,  5.55s/it]
                                                 
{'train/ce_loss': 1.6015625, 'train/diffusion_loss': 0.4041873514652252, 'epoch': 6.45}

 65%|██████▍   | 471/730 [41:43<23:56,  5.55s/it]
                                                 
{'train/learning_rate_real': 7.385407604330436e-06, 'epoch': 6.45}

 65%|██████▍   | 471/730 [41:43<23:56,  5.55s/it]
                                                 
{'debug/num_tok_total': 3087.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 3087.0, 'debug/num_lat_loss': 1788.0, 'epoch': 6.45}

 65%|██████▍   | 471/730 [41:44<23:56,  5.55s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.34933826327323914, 'epoch': 6.45}

 65%|██████▍   | 471/730 [41:44<23:56,  5.55s/it]
                                                 
{'train/learning_rate_real': 7.385407604330436e-06, 'epoch': 6.45}

 65%|██████▍   | 471/730 [41:44<23:56,  5.55s/it]
                                                 
{'debug/num_tok_total': 2837.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2837.0, 'debug/num_lat_loss': 1770.0, 'epoch': 6.45}

 65%|██████▍   | 471/730 [41:45<23:56,  5.55s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.34352102875709534, 'epoch': 6.45}

 65%|██████▍   | 471/730 [41:45<23:56,  5.55s/it]
                                                 
{'train/learning_rate_real': 7.385407604330436e-06, 'epoch': 6.45}

 65%|██████▍   | 471/730 [41:45<23:56,  5.55s/it]
 65%|██████▍   | 472/730 [41:46<24:08,  5.61s/it]
                                                 
{'debug/num_tok_total': 2379.0, 'debug/num_tok_loss': 1739.0, 'debug/num_lat_total': 2379.0, 'debug/num_lat_loss': 1739.0, 'epoch': 6.47}

 65%|██████▍   | 472/730 [41:47<24:08,  5.61s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.4246566593647003, 'epoch': 6.47}

 65%|██████▍   | 472/730 [41:47<24:08,  5.61s/it]
                                                 
{'train/learning_rate_real': 7.334847677895276e-06, 'epoch': 6.47}

 65%|██████▍   | 472/730 [41:47<24:08,  5.61s/it]
                                                 
{'debug/num_tok_total': 2655.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2655.0, 'debug/num_lat_loss': 1792.0, 'epoch': 6.47}

 65%|██████▍   | 472/730 [41:48<24:08,  5.61s/it]
                                                 
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.4245871305465698, 'epoch': 6.47}

 65%|██████▍   | 472/730 [41:48<24:08,  5.61s/it]
                                                 
{'train/learning_rate_real': 7.334847677895276e-06, 'epoch': 6.47}

 65%|██████▍   | 472/730 [41:48<24:08,  5.61s/it]
                                                 
{'debug/num_tok_total': 2411.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2411.0, 'debug/num_lat_loss': 1773.0, 'epoch': 6.47}

 65%|██████▍   | 472/730 [41:49<24:08,  5.61s/it]
                                                 
{'train/ce_loss': 2.078125, 'train/diffusion_loss': 0.45312103629112244, 'epoch': 6.47}

 65%|██████▍   | 472/730 [41:49<24:08,  5.61s/it]
                                                 
{'train/learning_rate_real': 7.334847677895276e-06, 'epoch': 6.47}

 65%|██████▍   | 472/730 [41:49<24:08,  5.61s/it]
                                                 
{'debug/num_tok_total': 2201.0, 'debug/num_tok_loss': 1762.0, 'debug/num_lat_total': 2201.0, 'debug/num_lat_loss': 1762.0, 'epoch': 6.47}

 65%|██████▍   | 472/730 [41:51<24:08,  5.61s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.4615404009819031, 'epoch': 6.47}

 65%|██████▍   | 472/730 [41:51<24:08,  5.61s/it]
                                                 
{'train/learning_rate_real': 7.334847677895276e-06, 'epoch': 6.47}

 65%|██████▍   | 472/730 [41:51<24:08,  5.61s/it]
 65%|██████▍   | 473/730 [41:51<23:31,  5.49s/it]
                                                 
{'debug/num_tok_total': 1973.0, 'debug/num_tok_loss': 1760.0, 'debug/num_lat_total': 1973.0, 'debug/num_lat_loss': 1760.0, 'epoch': 6.48}

 65%|██████▍   | 473/730 [41:52<23:31,  5.49s/it]
                                                 
{'train/ce_loss': 1.9453125, 'train/diffusion_loss': 0.5112565755844116, 'epoch': 6.48}

 65%|██████▍   | 473/730 [41:52<23:31,  5.49s/it]
                                                 
{'train/learning_rate_real': 7.284389450218544e-06, 'epoch': 6.48}

 65%|██████▍   | 473/730 [41:52<23:31,  5.49s/it]
                                                 
{'debug/num_tok_total': 3050.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 3050.0, 'debug/num_lat_loss': 1766.0, 'epoch': 6.48}

 65%|██████▍   | 473/730 [41:53<23:31,  5.49s/it]
                                                 
{'train/ce_loss': 1.9375, 'train/diffusion_loss': 0.3069973886013031, 'epoch': 6.48}

 65%|██████▍   | 473/730 [41:53<23:31,  5.49s/it]
                                                 
{'train/learning_rate_real': 7.284389450218544e-06, 'epoch': 6.48}

 65%|██████▍   | 473/730 [41:53<23:31,  5.49s/it]
                                                 
{'debug/num_tok_total': 3007.0, 'debug/num_tok_loss': 1644.0, 'debug/num_lat_total': 3007.0, 'debug/num_lat_loss': 1644.0, 'epoch': 6.48}

 65%|██████▍   | 473/730 [41:55<23:31,  5.49s/it]
                                                 
{'train/ce_loss': 1.671875, 'train/diffusion_loss': 0.2497856318950653, 'epoch': 6.48}

 65%|██████▍   | 473/730 [41:55<23:31,  5.49s/it]
                                                 
{'train/learning_rate_real': 7.284389450218544e-06, 'epoch': 6.48}

 65%|██████▍   | 473/730 [41:55<23:31,  5.49s/it]
                                                 
{'debug/num_tok_total': 2382.0, 'debug/num_tok_loss': 1751.0, 'debug/num_lat_total': 2382.0, 'debug/num_lat_loss': 1751.0, 'epoch': 6.48}

 65%|██████▍   | 473/730 [41:56<23:31,  5.49s/it]
                                                 
{'train/ce_loss': 1.875, 'train/diffusion_loss': 0.4508194327354431, 'epoch': 6.48}

 65%|██████▍   | 473/730 [41:56<23:31,  5.49s/it]
                                                 
{'train/learning_rate_real': 7.284389450218544e-06, 'epoch': 6.48}

 65%|██████▍   | 473/730 [41:56<23:31,  5.49s/it]
 65%|██████▍   | 474/730 [41:56<23:16,  5.45s/it]
                                                 
{'debug/num_tok_total': 2387.0, 'debug/num_tok_loss': 1754.0, 'debug/num_lat_total': 2387.0, 'debug/num_lat_loss': 1754.0, 'epoch': 6.49}

 65%|██████▍   | 474/730 [41:58<23:16,  5.45s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.4856170415878296, 'epoch': 6.49}

 65%|██████▍   | 474/730 [41:58<23:16,  5.45s/it]
                                                 
{'train/learning_rate_real': 7.234033914792554e-06, 'epoch': 6.49}

 65%|██████▍   | 474/730 [41:58<23:16,  5.45s/it]
                                                 
{'debug/num_tok_total': 2641.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2641.0, 'debug/num_lat_loss': 1771.0, 'epoch': 6.49}

 65%|██████▍   | 474/730 [41:59<23:16,  5.45s/it]
                                                 
{'train/ce_loss': 1.9140625, 'train/diffusion_loss': 0.4076163172721863, 'epoch': 6.49}

 65%|██████▍   | 474/730 [41:59<23:16,  5.45s/it]
                                                 
{'train/learning_rate_real': 7.234033914792554e-06, 'epoch': 6.49}

 65%|██████▍   | 474/730 [41:59<23:16,  5.45s/it]
                                                 
{'debug/num_tok_total': 2451.0, 'debug/num_tok_loss': 1800.0, 'debug/num_lat_total': 2451.0, 'debug/num_lat_loss': 1800.0, 'epoch': 6.49}

 65%|██████▍   | 474/730 [42:00<23:16,  5.45s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.443136602640152, 'epoch': 6.49}

 65%|██████▍   | 474/730 [42:00<23:16,  5.45s/it]
                                                 
{'train/learning_rate_real': 7.234033914792554e-06, 'epoch': 6.49}

 65%|██████▍   | 474/730 [42:00<23:16,  5.45s/it]
                                                 
{'debug/num_tok_total': 2655.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2655.0, 'debug/num_lat_loss': 1784.0, 'epoch': 6.49}

 65%|██████▍   | 474/730 [42:01<23:16,  5.45s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.3735961318016052, 'epoch': 6.49}

 65%|██████▍   | 474/730 [42:01<23:16,  5.45s/it]
                                                 
{'train/learning_rate_real': 7.234033914792554e-06, 'epoch': 6.49}

 65%|██████▍   | 474/730 [42:01<23:16,  5.45s/it]
 65%|██████▌   | 475/730 [42:02<22:46,  5.36s/it]
                                                 
{'debug/num_tok_total': 2241.0, 'debug/num_tok_loss': 1486.0, 'debug/num_lat_total': 2241.0, 'debug/num_lat_loss': 1486.0, 'epoch': 6.51}

 65%|██████▌   | 475/730 [42:03<22:46,  5.36s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.4047999382019043, 'epoch': 6.51}

 65%|██████▌   | 475/730 [42:03<22:46,  5.36s/it]
                                                 
{'train/learning_rate_real': 7.183782063087668e-06, 'epoch': 6.51}

 65%|██████▌   | 475/730 [42:03<22:46,  5.36s/it]
                                                 
{'debug/num_tok_total': 2424.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2424.0, 'debug/num_lat_loss': 1791.0, 'epoch': 6.51}

 65%|██████▌   | 475/730 [42:04<22:46,  5.36s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.43483036756515503, 'epoch': 6.51}

 65%|██████▌   | 475/730 [42:04<22:46,  5.36s/it]
                                                 
{'train/learning_rate_real': 7.183782063087668e-06, 'epoch': 6.51}

 65%|██████▌   | 475/730 [42:04<22:46,  5.36s/it]
                                                 
{'debug/num_tok_total': 3093.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 3093.0, 'debug/num_lat_loss': 1795.0, 'epoch': 6.51}

 65%|██████▌   | 475/730 [42:05<22:46,  5.36s/it]
                                                 
{'train/ce_loss': 2.046875, 'train/diffusion_loss': 0.3262596130371094, 'epoch': 6.51}

 65%|██████▌   | 475/730 [42:05<22:46,  5.36s/it]
                                                 
{'train/learning_rate_real': 7.183782063087668e-06, 'epoch': 6.51}

 65%|██████▌   | 475/730 [42:05<22:46,  5.36s/it]
                                                 
{'debug/num_tok_total': 3122.0, 'debug/num_tok_loss': 1816.0, 'debug/num_lat_total': 3122.0, 'debug/num_lat_loss': 1816.0, 'epoch': 6.51}

 65%|██████▌   | 475/730 [42:07<22:46,  5.36s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.31287816166877747, 'epoch': 6.51}

 65%|██████▌   | 475/730 [42:07<22:46,  5.36s/it]
                                                 
{'train/learning_rate_real': 7.183782063087668e-06, 'epoch': 6.51}

 65%|██████▌   | 475/730 [42:07<22:46,  5.36s/it]
 65%|██████▌   | 476/730 [42:07<22:48,  5.39s/it]
                                                 
{'debug/num_tok_total': 2856.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2856.0, 'debug/num_lat_loss': 1775.0, 'epoch': 6.52}

 65%|██████▌   | 476/730 [42:08<22:48,  5.39s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.35940900444984436, 'epoch': 6.52}

 65%|██████▌   | 476/730 [42:08<22:48,  5.39s/it]
                                                 
{'train/learning_rate_real': 7.133634884532769e-06, 'epoch': 6.52}

 65%|██████▌   | 476/730 [42:08<22:48,  5.39s/it]
                                                 
{'debug/num_tok_total': 3072.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 3072.0, 'debug/num_lat_loss': 1787.0, 'epoch': 6.52}

 65%|██████▌   | 476/730 [42:10<22:48,  5.39s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.3121586740016937, 'epoch': 6.52}

 65%|██████▌   | 476/730 [42:10<22:48,  5.39s/it]
                                                 
{'train/learning_rate_real': 7.133634884532769e-06, 'epoch': 6.52}

 65%|██████▌   | 476/730 [42:10<22:48,  5.39s/it]
                                                 
{'debug/num_tok_total': 2642.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2642.0, 'debug/num_lat_loss': 1781.0, 'epoch': 6.52}

 65%|██████▌   | 476/730 [42:11<22:48,  5.39s/it]
                                                 
{'train/ce_loss': 1.921875, 'train/diffusion_loss': 0.40311864018440247, 'epoch': 6.52}

 65%|██████▌   | 476/730 [42:11<22:48,  5.39s/it]
                                                 
{'train/learning_rate_real': 7.133634884532769e-06, 'epoch': 6.52}

 65%|██████▌   | 476/730 [42:11<22:48,  5.39s/it]
                                                 
{'debug/num_tok_total': 2756.0, 'debug/num_tok_loss': 1670.0, 'debug/num_lat_total': 2756.0, 'debug/num_lat_loss': 1670.0, 'epoch': 6.52}

 65%|██████▌   | 476/730 [42:12<22:48,  5.39s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.33772632479667664, 'epoch': 6.52}

 65%|██████▌   | 476/730 [42:12<22:48,  5.39s/it]
                                                 
{'train/learning_rate_real': 7.133634884532769e-06, 'epoch': 6.52}

 65%|██████▌   | 476/730 [42:12<22:48,  5.39s/it]
 65%|██████▌   | 477/730 [42:13<22:56,  5.44s/it]
                                                 
{'debug/num_tok_total': 2457.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2457.0, 'debug/num_lat_loss': 1802.0, 'epoch': 6.53}

 65%|██████▌   | 477/730 [42:14<22:56,  5.44s/it]
                                                 
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.4384009540081024, 'epoch': 6.53}

 65%|██████▌   | 477/730 [42:14<22:56,  5.44s/it]
                                                 
{'train/learning_rate_real': 7.083593366495804e-06, 'epoch': 6.53}

 65%|██████▌   | 477/730 [42:14<22:56,  5.44s/it]
                                                 
{'debug/num_tok_total': 2459.0, 'debug/num_tok_loss': 1617.0, 'debug/num_lat_total': 2459.0, 'debug/num_lat_loss': 1617.0, 'epoch': 6.53}

 65%|██████▌   | 477/730 [42:15<22:56,  5.44s/it]
                                                 
{'train/ce_loss': 1.8984375, 'train/diffusion_loss': 0.382244735956192, 'epoch': 6.53}

 65%|██████▌   | 477/730 [42:15<22:56,  5.44s/it]
                                                 
{'train/learning_rate_real': 7.083593366495804e-06, 'epoch': 6.53}

 65%|██████▌   | 477/730 [42:15<22:56,  5.44s/it]
                                                 
{'debug/num_tok_total': 2610.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2610.0, 'debug/num_lat_loss': 1770.0, 'epoch': 6.53}

 65%|██████▌   | 477/730 [42:16<22:56,  5.44s/it]
                                                 
{'train/ce_loss': 1.6875, 'train/diffusion_loss': 0.39042559266090393, 'epoch': 6.53}

 65%|██████▌   | 477/730 [42:16<22:56,  5.44s/it]
                                                 
{'train/learning_rate_real': 7.083593366495804e-06, 'epoch': 6.53}

 65%|██████▌   | 477/730 [42:16<22:56,  5.44s/it]
                                                 
{'debug/num_tok_total': 2445.0, 'debug/num_tok_loss': 1800.0, 'debug/num_lat_total': 2445.0, 'debug/num_lat_loss': 1800.0, 'epoch': 6.53}

 65%|██████▌   | 477/730 [42:17<22:56,  5.44s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.4671259820461273, 'epoch': 6.53}

 65%|██████▌   | 477/730 [42:17<22:56,  5.44s/it]
                                                 
{'train/learning_rate_real': 7.083593366495804e-06, 'epoch': 6.53}

 65%|██████▌   | 477/730 [42:17<22:56,  5.44s/it]
 65%|██████▌   | 478/730 [42:18<22:32,  5.37s/it]
                                                 
{'debug/num_tok_total': 2621.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2621.0, 'debug/num_lat_loss': 1764.0, 'epoch': 6.55}

 65%|██████▌   | 478/730 [42:19<22:32,  5.37s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.40727078914642334, 'epoch': 6.55}

 65%|██████▌   | 478/730 [42:19<22:32,  5.37s/it]
                                                 
{'train/learning_rate_real': 7.033658494264309e-06, 'epoch': 6.55}

 65%|██████▌   | 478/730 [42:19<22:32,  5.37s/it]
                                                 
{'debug/num_tok_total': 2446.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2446.0, 'debug/num_lat_loss': 1794.0, 'epoch': 6.55}

 65%|██████▌   | 478/730 [42:20<22:32,  5.37s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.46277564764022827, 'epoch': 6.55}

 65%|██████▌   | 478/730 [42:20<22:32,  5.37s/it]
                                                 
{'train/learning_rate_real': 7.033658494264309e-06, 'epoch': 6.55}

 65%|██████▌   | 478/730 [42:20<22:32,  5.37s/it]
                                                 
{'debug/num_tok_total': 3282.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 3282.0, 'debug/num_lat_loss': 1783.0, 'epoch': 6.55}

 65%|██████▌   | 478/730 [42:22<22:32,  5.37s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.2462470382452011, 'epoch': 6.55}

 65%|██████▌   | 478/730 [42:22<22:32,  5.37s/it]
                                                 
{'train/learning_rate_real': 7.033658494264309e-06, 'epoch': 6.55}

 65%|██████▌   | 478/730 [42:22<22:32,  5.37s/it]
                                                 
{'debug/num_tok_total': 2755.0, 'debug/num_tok_loss': 1686.0, 'debug/num_lat_total': 2755.0, 'debug/num_lat_loss': 1686.0, 'epoch': 6.55}

 65%|██████▌   | 478/730 [42:23<22:32,  5.37s/it]
                                                 
{'train/ce_loss': 1.6640625, 'train/diffusion_loss': 0.3402400016784668, 'epoch': 6.55}

 65%|██████▌   | 478/730 [42:23<22:32,  5.37s/it]
                                                 
{'train/learning_rate_real': 7.033658494264309e-06, 'epoch': 6.55}

 65%|██████▌   | 478/730 [42:23<22:32,  5.37s/it]
 66%|██████▌   | 479/730 [42:24<23:01,  5.50s/it]
                                                 
{'debug/num_tok_total': 2625.0, 'debug/num_tok_loss': 1762.0, 'debug/num_lat_total': 2625.0, 'debug/num_lat_loss': 1762.0, 'epoch': 6.56}

 66%|██████▌   | 479/730 [42:25<23:01,  5.50s/it]
                                                 
{'train/ce_loss': 1.6875, 'train/diffusion_loss': 0.37732428312301636, 'epoch': 6.56}

 66%|██████▌   | 479/730 [42:25<23:01,  5.50s/it]
                                                 
{'train/learning_rate_real': 6.98383125102604e-06, 'epoch': 6.56}

 66%|██████▌   | 479/730 [42:25<23:01,  5.50s/it]
                                                 
{'debug/num_tok_total': 2450.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2450.0, 'debug/num_lat_loss': 1790.0, 'epoch': 6.56}

 66%|██████▌   | 479/730 [42:26<23:01,  5.50s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.44480493664741516, 'epoch': 6.56}

 66%|██████▌   | 479/730 [42:26<23:01,  5.50s/it]
                                                 
{'train/learning_rate_real': 6.98383125102604e-06, 'epoch': 6.56}

 66%|██████▌   | 479/730 [42:26<23:01,  5.50s/it]
                                                 
{'debug/num_tok_total': 2645.0, 'debug/num_tok_loss': 1554.0, 'debug/num_lat_total': 2645.0, 'debug/num_lat_loss': 1554.0, 'epoch': 6.56}

 66%|██████▌   | 479/730 [42:27<23:01,  5.50s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.3186272084712982, 'epoch': 6.56}

 66%|██████▌   | 479/730 [42:27<23:01,  5.50s/it]
                                                 
{'train/learning_rate_real': 6.98383125102604e-06, 'epoch': 6.56}

 66%|██████▌   | 479/730 [42:27<23:01,  5.50s/it]
                                                 
{'debug/num_tok_total': 2859.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2859.0, 'debug/num_lat_loss': 1791.0, 'epoch': 6.56}

 66%|██████▌   | 479/730 [42:29<23:01,  5.50s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.3586609959602356, 'epoch': 6.56}

 66%|██████▌   | 479/730 [42:29<23:01,  5.50s/it]
                                                 
{'train/learning_rate_real': 6.98383125102604e-06, 'epoch': 6.56}

 66%|██████▌   | 479/730 [42:29<23:01,  5.50s/it]03/16/2026 07:30:01 - INFO - __main__ - LoRA debug step 480: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 66%|██████▌   | 480/730 [42:30<23:23,  5.62s/it]
                                                 
{'loss': 2.4317, 'grad_norm': 1.1788009405136108, 'learning_rate': 6.98383125102604e-06, 'epoch': 6.58}

 66%|██████▌   | 480/730 [42:30<23:23,  5.62s/it]
                                                 
{'debug/num_tok_total': 3062.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 3062.0, 'debug/num_lat_loss': 1779.0, 'epoch': 6.58}

 66%|██████▌   | 480/730 [42:31<23:23,  5.62s/it]
                                                 
{'train/ce_loss': 1.9609375, 'train/diffusion_loss': 0.3285578191280365, 'epoch': 6.58}

 66%|██████▌   | 480/730 [42:31<23:23,  5.62s/it]
                                                 
{'train/learning_rate_real': 6.9341126178496085e-06, 'epoch': 6.58}

 66%|██████▌   | 480/730 [42:31<23:23,  5.62s/it]
                                                 
{'debug/num_tok_total': 3072.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 3072.0, 'debug/num_lat_loss': 1782.0, 'epoch': 6.58}

 66%|██████▌   | 480/730 [42:32<23:23,  5.62s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.3409457802772522, 'epoch': 6.58}

 66%|██████▌   | 480/730 [42:32<23:23,  5.62s/it]
                                                 
{'train/learning_rate_real': 6.9341126178496085e-06, 'epoch': 6.58}

 66%|██████▌   | 480/730 [42:32<23:23,  5.62s/it]
                                                 
{'debug/num_tok_total': 2466.0, 'debug/num_tok_loss': 1691.0, 'debug/num_lat_total': 2466.0, 'debug/num_lat_loss': 1691.0, 'epoch': 6.58}

 66%|██████▌   | 480/730 [42:33<23:23,  5.62s/it]
                                                 
{'train/ce_loss': 1.9296875, 'train/diffusion_loss': 0.4096618592739105, 'epoch': 6.58}

 66%|██████▌   | 480/730 [42:33<23:23,  5.62s/it]
                                                 
{'train/learning_rate_real': 6.9341126178496085e-06, 'epoch': 6.58}

 66%|██████▌   | 480/730 [42:33<23:23,  5.62s/it]
                                                 
{'debug/num_tok_total': 2623.0, 'debug/num_tok_loss': 1758.0, 'debug/num_lat_total': 2623.0, 'debug/num_lat_loss': 1758.0, 'epoch': 6.58}

 66%|██████▌   | 480/730 [42:35<23:23,  5.62s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.38009077310562134, 'epoch': 6.58}

 66%|██████▌   | 480/730 [42:35<23:23,  5.62s/it]
                                                 
{'train/learning_rate_real': 6.9341126178496085e-06, 'epoch': 6.58}

 66%|██████▌   | 480/730 [42:35<23:23,  5.62s/it]
 66%|██████▌   | 481/730 [42:35<23:20,  5.63s/it]
                                                 
{'debug/num_tok_total': 2050.0, 'debug/num_tok_loss': 1594.0, 'debug/num_lat_total': 2050.0, 'debug/num_lat_loss': 1594.0, 'epoch': 6.59}

 66%|██████▌   | 481/730 [42:36<23:20,  5.63s/it]
                                                 
{'train/ce_loss': 1.890625, 'train/diffusion_loss': 0.4798370897769928, 'epoch': 6.59}

 66%|██████▌   | 481/730 [42:36<23:20,  5.63s/it]
                                                 
{'train/learning_rate_real': 6.884503573665141e-06, 'epoch': 6.59}

 66%|██████▌   | 481/730 [42:36<23:20,  5.63s/it]
                                                 
{'debug/num_tok_total': 3270.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 3270.0, 'debug/num_lat_loss': 1769.0, 'epoch': 6.59}

 66%|██████▌   | 481/730 [42:38<23:20,  5.63s/it]
                                                 
{'train/ce_loss': 1.90625, 'train/diffusion_loss': 0.29756802320480347, 'epoch': 6.59}

 66%|██████▌   | 481/730 [42:38<23:20,  5.63s/it]
                                                 
{'train/learning_rate_real': 6.884503573665141e-06, 'epoch': 6.59}

 66%|██████▌   | 481/730 [42:38<23:20,  5.63s/it]
                                                 
{'debug/num_tok_total': 2693.0, 'debug/num_tok_loss': 1810.0, 'debug/num_lat_total': 2693.0, 'debug/num_lat_loss': 1810.0, 'epoch': 6.59}

 66%|██████▌   | 481/730 [42:39<23:20,  5.63s/it]
                                                 
{'train/ce_loss': 1.890625, 'train/diffusion_loss': 0.38403022289276123, 'epoch': 6.59}

 66%|██████▌   | 481/730 [42:39<23:20,  5.63s/it]
                                                 
{'train/learning_rate_real': 6.884503573665141e-06, 'epoch': 6.59}

 66%|██████▌   | 481/730 [42:39<23:20,  5.63s/it]
                                                 
{'debug/num_tok_total': 2858.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2858.0, 'debug/num_lat_loss': 1787.0, 'epoch': 6.59}

 66%|██████▌   | 481/730 [42:40<23:20,  5.63s/it]
                                                 
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.3850100040435791, 'epoch': 6.59}

 66%|██████▌   | 481/730 [42:40<23:20,  5.63s/it]
                                                 
{'train/learning_rate_real': 6.884503573665141e-06, 'epoch': 6.59}

 66%|██████▌   | 481/730 [42:40<23:20,  5.63s/it]
 66%|██████▌   | 482/730 [42:41<23:07,  5.60s/it]
                                                 
{'debug/num_tok_total': 2858.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2858.0, 'debug/num_lat_loss': 1779.0, 'epoch': 6.6}

 66%|██████▌   | 482/730 [42:42<23:07,  5.60s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.3516102433204651, 'epoch': 6.6}

 66%|██████▌   | 482/730 [42:42<23:07,  5.60s/it]
                                                 
{'train/learning_rate_real': 6.835005095245037e-06, 'epoch': 6.6}

 66%|██████▌   | 482/730 [42:42<23:07,  5.60s/it]
                                                 
{'debug/num_tok_total': 2849.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2849.0, 'debug/num_lat_loss': 1772.0, 'epoch': 6.6}

 66%|██████▌   | 482/730 [42:43<23:07,  5.60s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.34323450922966003, 'epoch': 6.6}

 66%|██████▌   | 482/730 [42:43<23:07,  5.60s/it]
                                                 
{'train/learning_rate_real': 6.835005095245037e-06, 'epoch': 6.6}

 66%|██████▌   | 482/730 [42:43<23:07,  5.60s/it]
                                                 
{'debug/num_tok_total': 2628.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2628.0, 'debug/num_lat_loss': 1771.0, 'epoch': 6.6}

 66%|██████▌   | 482/730 [42:44<23:07,  5.60s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.39498472213745117, 'epoch': 6.6}

 66%|██████▌   | 482/730 [42:44<23:07,  5.60s/it]
                                                 
{'train/learning_rate_real': 6.835005095245037e-06, 'epoch': 6.6}

 66%|██████▌   | 482/730 [42:44<23:07,  5.60s/it]
                                                 
{'debug/num_tok_total': 2856.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2856.0, 'debug/num_lat_loss': 1792.0, 'epoch': 6.6}

 66%|██████▌   | 482/730 [42:46<23:07,  5.60s/it]
                                                 
{'train/ce_loss': 1.875, 'train/diffusion_loss': 0.39164021611213684, 'epoch': 6.6}

 66%|██████▌   | 482/730 [42:46<23:07,  5.60s/it]
                                                 
{'train/learning_rate_real': 6.835005095245037e-06, 'epoch': 6.6}

 66%|██████▌   | 482/730 [42:46<23:07,  5.60s/it]
 66%|██████▌   | 483/730 [42:46<23:16,  5.65s/it]
                                                 
{'debug/num_tok_total': 2388.0, 'debug/num_tok_loss': 1754.0, 'debug/num_lat_total': 2388.0, 'debug/num_lat_loss': 1754.0, 'epoch': 6.62}

 66%|██████▌   | 483/730 [42:48<23:16,  5.65s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.45020803809165955, 'epoch': 6.62}

 66%|██████▌   | 483/730 [42:48<23:16,  5.65s/it]
                                                 
{'train/learning_rate_real': 6.785618157184711e-06, 'epoch': 6.62}

 66%|██████▌   | 483/730 [42:48<23:16,  5.65s/it]
                                                 
{'debug/num_tok_total': 1991.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 1991.0, 'debug/num_lat_loss': 1783.0, 'epoch': 6.62}

 66%|██████▌   | 483/730 [42:49<23:16,  5.65s/it]
                                                 
{'train/ce_loss': 1.96875, 'train/diffusion_loss': 0.5166428685188293, 'epoch': 6.62}

 66%|██████▌   | 483/730 [42:49<23:16,  5.65s/it]
                                                 
{'train/learning_rate_real': 6.785618157184711e-06, 'epoch': 6.62}

 66%|██████▌   | 483/730 [42:49<23:16,  5.65s/it]
                                                 
{'debug/num_tok_total': 2866.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2866.0, 'debug/num_lat_loss': 1792.0, 'epoch': 6.62}

 66%|██████▌   | 483/730 [42:50<23:16,  5.65s/it]
                                                 
{'train/ce_loss': 2.046875, 'train/diffusion_loss': 0.36153414845466614, 'epoch': 6.62}

 66%|██████▌   | 483/730 [42:50<23:16,  5.65s/it]
                                                 
{'train/learning_rate_real': 6.785618157184711e-06, 'epoch': 6.62}

 66%|██████▌   | 483/730 [42:50<23:16,  5.65s/it]
                                                 
{'debug/num_tok_total': 2897.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2897.0, 'debug/num_lat_loss': 1799.0, 'epoch': 6.62}

 66%|██████▌   | 483/730 [42:51<23:16,  5.65s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.3449109196662903, 'epoch': 6.62}

 66%|██████▌   | 483/730 [42:51<23:16,  5.65s/it]
                                                 
{'train/learning_rate_real': 6.785618157184711e-06, 'epoch': 6.62}

 66%|██████▌   | 483/730 [42:51<23:16,  5.65s/it]
 66%|██████▋   | 484/730 [42:52<22:33,  5.50s/it]
                                                 
{'debug/num_tok_total': 2638.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2638.0, 'debug/num_lat_loss': 1778.0, 'epoch': 6.63}

 66%|██████▋   | 484/730 [42:53<22:33,  5.50s/it]
                                                 
{'train/ce_loss': 1.8828125, 'train/diffusion_loss': 0.4143303334712982, 'epoch': 6.63}

 66%|██████▋   | 484/730 [42:53<22:33,  5.50s/it]
                                                 
{'train/learning_rate_real': 6.736343731883425e-06, 'epoch': 6.63}

 66%|██████▋   | 484/730 [42:53<22:33,  5.50s/it]
                                                 
{'debug/num_tok_total': 2243.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 2243.0, 'debug/num_lat_loss': 1801.0, 'epoch': 6.63}

 66%|██████▋   | 484/730 [42:54<22:33,  5.50s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.5016627907752991, 'epoch': 6.63}

 66%|██████▋   | 484/730 [42:54<22:33,  5.50s/it]
                                                 
{'train/learning_rate_real': 6.736343731883425e-06, 'epoch': 6.63}

 66%|██████▋   | 484/730 [42:54<22:33,  5.50s/it]
                                                 
{'debug/num_tok_total': 2435.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2435.0, 'debug/num_lat_loss': 1785.0, 'epoch': 6.63}

 66%|██████▋   | 484/730 [42:55<22:33,  5.50s/it]
                                                 
{'train/ce_loss': 1.90625, 'train/diffusion_loss': 0.45818042755126953, 'epoch': 6.63}

 66%|██████▋   | 484/730 [42:55<22:33,  5.50s/it]
                                                 
{'train/learning_rate_real': 6.736343731883425e-06, 'epoch': 6.63}

 66%|██████▋   | 484/730 [42:55<22:33,  5.50s/it]
                                                 
{'debug/num_tok_total': 2849.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2849.0, 'debug/num_lat_loss': 1784.0, 'epoch': 6.63}

 66%|██████▋   | 484/730 [42:56<22:33,  5.50s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.35906147956848145, 'epoch': 6.63}

 66%|██████▋   | 484/730 [42:56<22:33,  5.50s/it]
                                                 
{'train/learning_rate_real': 6.736343731883425e-06, 'epoch': 6.63}

 66%|██████▋   | 484/730 [42:56<22:33,  5.50s/it]
 66%|██████▋   | 485/730 [42:57<22:13,  5.44s/it]
                                                 
{'debug/num_tok_total': 2005.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2005.0, 'debug/num_lat_loss': 1786.0, 'epoch': 6.64}

 66%|██████▋   | 485/730 [42:58<22:13,  5.44s/it]
                                                 
{'train/ce_loss': 1.90625, 'train/diffusion_loss': 0.5256776809692383, 'epoch': 6.64}

 66%|██████▋   | 485/730 [42:58<22:13,  5.44s/it]
                                                 
{'train/learning_rate_real': 6.687182789525132e-06, 'epoch': 6.64}

 66%|██████▋   | 485/730 [42:58<22:13,  5.44s/it]
                                                 
{'debug/num_tok_total': 3072.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 3072.0, 'debug/num_lat_loss': 1788.0, 'epoch': 6.64}

 66%|██████▋   | 485/730 [42:59<22:13,  5.44s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.2845451235771179, 'epoch': 6.64}

 66%|██████▋   | 485/730 [42:59<22:13,  5.44s/it]
                                                 
{'train/learning_rate_real': 6.687182789525132e-06, 'epoch': 6.64}

 66%|██████▋   | 485/730 [42:59<22:13,  5.44s/it]
                                                 
{'debug/num_tok_total': 2723.0, 'debug/num_tok_loss': 1716.0, 'debug/num_lat_total': 2723.0, 'debug/num_lat_loss': 1716.0, 'epoch': 6.64}

 66%|██████▋   | 485/730 [43:01<22:13,  5.44s/it]
                                                 
{'train/ce_loss': 1.921875, 'train/diffusion_loss': 0.36981528997421265, 'epoch': 6.64}

 66%|██████▋   | 485/730 [43:01<22:13,  5.44s/it]
                                                 
{'train/learning_rate_real': 6.687182789525132e-06, 'epoch': 6.64}

 66%|██████▋   | 485/730 [43:01<22:13,  5.44s/it]
                                                 
{'debug/num_tok_total': 2446.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2446.0, 'debug/num_lat_loss': 1792.0, 'epoch': 6.64}

 66%|██████▋   | 485/730 [43:02<22:13,  5.44s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.4601265788078308, 'epoch': 6.64}

 66%|██████▋   | 485/730 [43:02<22:13,  5.44s/it]
                                                 
{'train/learning_rate_real': 6.687182789525132e-06, 'epoch': 6.64}

 66%|██████▋   | 485/730 [43:02<22:13,  5.44s/it]
 67%|██████▋   | 486/730 [43:02<22:14,  5.47s/it]
                                                 
{'debug/num_tok_total': 3077.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 3077.0, 'debug/num_lat_loss': 1781.0, 'epoch': 6.66}

 67%|██████▋   | 486/730 [43:04<22:14,  5.47s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.3013792037963867, 'epoch': 6.66}

 67%|██████▋   | 486/730 [43:04<22:14,  5.47s/it]
                                                 
{'train/learning_rate_real': 6.638136298059365e-06, 'epoch': 6.66}

 67%|██████▋   | 486/730 [43:04<22:14,  5.47s/it]
                                                 
{'debug/num_tok_total': 2396.0, 'debug/num_tok_loss': 1762.0, 'debug/num_lat_total': 2396.0, 'debug/num_lat_loss': 1762.0, 'epoch': 6.66}

 67%|██████▋   | 486/730 [43:05<22:14,  5.47s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.4467102885246277, 'epoch': 6.66}

 67%|██████▋   | 486/730 [43:05<22:14,  5.47s/it]
                                                 
{'train/learning_rate_real': 6.638136298059365e-06, 'epoch': 6.66}

 67%|██████▋   | 486/730 [43:05<22:14,  5.47s/it]
                                                 
{'debug/num_tok_total': 2600.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 2600.0, 'debug/num_lat_loss': 1765.0, 'epoch': 6.66}

 67%|██████▋   | 486/730 [43:06<22:14,  5.47s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.39547091722488403, 'epoch': 6.66}

 67%|██████▋   | 486/730 [43:06<22:14,  5.47s/it]
                                                 
{'train/learning_rate_real': 6.638136298059365e-06, 'epoch': 6.66}

 67%|██████▋   | 486/730 [43:06<22:14,  5.47s/it]
                                                 
{'debug/num_tok_total': 2425.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2425.0, 'debug/num_lat_loss': 1779.0, 'epoch': 6.66}

 67%|██████▋   | 486/730 [43:07<22:14,  5.47s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.4325716197490692, 'epoch': 6.66}

 67%|██████▋   | 486/730 [43:07<22:14,  5.47s/it]
                                                 
{'train/learning_rate_real': 6.638136298059365e-06, 'epoch': 6.66}

 67%|██████▋   | 486/730 [43:07<22:14,  5.47s/it]
 67%|██████▋   | 487/730 [43:08<21:47,  5.38s/it]
                                                 
{'debug/num_tok_total': 2514.0, 'debug/num_tok_loss': 1664.0, 'debug/num_lat_total': 2514.0, 'debug/num_lat_loss': 1664.0, 'epoch': 6.67}

 67%|██████▋   | 487/730 [43:09<21:47,  5.38s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.4225166440010071, 'epoch': 6.67}

 67%|██████▋   | 487/730 [43:09<21:47,  5.38s/it]
                                                 
{'train/learning_rate_real': 6.589205223182188e-06, 'epoch': 6.67}

 67%|██████▋   | 487/730 [43:09<21:47,  5.38s/it]
                                                 
{'debug/num_tok_total': 2899.0, 'debug/num_tok_loss': 1804.0, 'debug/num_lat_total': 2899.0, 'debug/num_lat_loss': 1804.0, 'epoch': 6.67}

 67%|██████▋   | 487/730 [43:10<21:47,  5.38s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.35381177067756653, 'epoch': 6.67}

 67%|██████▋   | 487/730 [43:10<21:47,  5.38s/it]
                                                 
{'train/learning_rate_real': 6.589205223182188e-06, 'epoch': 6.67}

 67%|██████▋   | 487/730 [43:10<21:47,  5.38s/it]
                                                 
{'debug/num_tok_total': 2242.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2242.0, 'debug/num_lat_loss': 1802.0, 'epoch': 6.67}

 67%|██████▋   | 487/730 [43:11<21:47,  5.38s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.48267003893852234, 'epoch': 6.67}

 67%|██████▋   | 487/730 [43:11<21:47,  5.38s/it]
                                                 
{'train/learning_rate_real': 6.589205223182188e-06, 'epoch': 6.67}

 67%|██████▋   | 487/730 [43:11<21:47,  5.38s/it]
                                                 
{'debug/num_tok_total': 2008.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2008.0, 'debug/num_lat_loss': 1796.0, 'epoch': 6.67}

 67%|██████▋   | 487/730 [43:12<21:47,  5.38s/it]
                                                 
{'train/ce_loss': 2.046875, 'train/diffusion_loss': 0.5555636882781982, 'epoch': 6.67}

 67%|██████▋   | 487/730 [43:12<21:47,  5.38s/it]
                                                 
{'train/learning_rate_real': 6.589205223182188e-06, 'epoch': 6.67}

 67%|██████▋   | 487/730 [43:12<21:47,  5.38s/it]
 67%|██████▋   | 488/730 [43:13<21:22,  5.30s/it]
                                                 
{'debug/num_tok_total': 2235.0, 'debug/num_tok_loss': 1803.0, 'debug/num_lat_total': 2235.0, 'debug/num_lat_loss': 1803.0, 'epoch': 6.68}

 67%|██████▋   | 488/730 [43:14<21:22,  5.30s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.486251562833786, 'epoch': 6.68}

 67%|██████▋   | 488/730 [43:14<21:22,  5.30s/it]
                                                 
{'train/learning_rate_real': 6.540390528317186e-06, 'epoch': 6.68}

 67%|██████▋   | 488/730 [43:14<21:22,  5.30s/it]
                                                 
{'debug/num_tok_total': 2426.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2426.0, 'debug/num_lat_loss': 1775.0, 'epoch': 6.68}

 67%|██████▋   | 488/730 [43:15<21:22,  5.30s/it]
                                                 
{'train/ce_loss': 1.9375, 'train/diffusion_loss': 0.4475606381893158, 'epoch': 6.68}

 67%|██████▋   | 488/730 [43:15<21:22,  5.30s/it]
                                                 
{'train/learning_rate_real': 6.540390528317186e-06, 'epoch': 6.68}

 67%|██████▋   | 488/730 [43:15<21:22,  5.30s/it]
                                                 
{'debug/num_tok_total': 3104.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 3104.0, 'debug/num_lat_loss': 1801.0, 'epoch': 6.68}

 67%|██████▋   | 488/730 [43:16<21:22,  5.30s/it]
                                                 
{'train/ce_loss': 1.6875, 'train/diffusion_loss': 0.32157978415489197, 'epoch': 6.68}

 67%|██████▋   | 488/730 [43:16<21:22,  5.30s/it]
                                                 
{'train/learning_rate_real': 6.540390528317186e-06, 'epoch': 6.68}

 67%|██████▋   | 488/730 [43:16<21:22,  5.30s/it]
                                                 
{'debug/num_tok_total': 2641.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2641.0, 'debug/num_lat_loss': 1786.0, 'epoch': 6.68}

 67%|██████▋   | 488/730 [43:18<21:22,  5.30s/it]
                                                 
{'train/ce_loss': 2.015625, 'train/diffusion_loss': 0.41339778900146484, 'epoch': 6.68}

 67%|██████▋   | 488/730 [43:18<21:22,  5.30s/it]
                                                 
{'train/learning_rate_real': 6.540390528317186e-06, 'epoch': 6.68}

 67%|██████▋   | 488/730 [43:18<21:22,  5.30s/it]
 67%|██████▋   | 489/730 [43:18<21:19,  5.31s/it]
                                                 
{'debug/num_tok_total': 2876.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2876.0, 'debug/num_lat_loss': 1793.0, 'epoch': 6.7}

 67%|██████▋   | 489/730 [43:19<21:19,  5.31s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.3517734408378601, 'epoch': 6.7}

 67%|██████▋   | 489/730 [43:19<21:19,  5.31s/it]
                                                 
{'train/learning_rate_real': 6.4916931745965e-06, 'epoch': 6.7}

 67%|██████▋   | 489/730 [43:19<21:19,  5.31s/it]
                                                 
{'debug/num_tok_total': 3059.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 3059.0, 'debug/num_lat_loss': 1782.0, 'epoch': 6.7}

 67%|██████▋   | 489/730 [43:21<21:19,  5.31s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.32052215933799744, 'epoch': 6.7}

 67%|██████▋   | 489/730 [43:21<21:19,  5.31s/it]
                                                 
{'train/learning_rate_real': 6.4916931745965e-06, 'epoch': 6.7}

 67%|██████▋   | 489/730 [43:21<21:19,  5.31s/it]
                                                 
{'debug/num_tok_total': 3543.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 3543.0, 'debug/num_lat_loss': 1801.0, 'epoch': 6.7}

 67%|██████▋   | 489/730 [43:22<21:19,  5.31s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.19956395030021667, 'epoch': 6.7}

 67%|██████▋   | 489/730 [43:22<21:19,  5.31s/it]
                                                 
{'train/learning_rate_real': 6.4916931745965e-06, 'epoch': 6.7}

 67%|██████▋   | 489/730 [43:22<21:19,  5.31s/it]
                                                 
{'debug/num_tok_total': 2621.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2621.0, 'debug/num_lat_loss': 1777.0, 'epoch': 6.7}

 67%|██████▋   | 489/730 [43:24<21:19,  5.31s/it]
                                                 
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.4163946211338043, 'epoch': 6.7}

 67%|██████▋   | 489/730 [43:24<21:19,  5.31s/it]
                                                 
{'train/learning_rate_real': 6.4916931745965e-06, 'epoch': 6.7}

 67%|██████▋   | 489/730 [43:24<21:19,  5.31s/it]03/16/2026 07:30:56 - INFO - __main__ - LoRA debug step 490: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 67%|██████▋   | 490/730 [43:24<22:02,  5.51s/it]
                                                 
{'loss': 2.517, 'grad_norm': 1.2326087951660156, 'learning_rate': 6.4916931745965e-06, 'epoch': 6.71}

 67%|██████▋   | 490/730 [43:24<22:02,  5.51s/it]
                                                 
{'debug/num_tok_total': 2551.0, 'debug/num_tok_loss': 1699.0, 'debug/num_lat_total': 2551.0, 'debug/num_lat_loss': 1699.0, 'epoch': 6.71}

 67%|██████▋   | 490/730 [43:25<22:02,  5.51s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.4079773724079132, 'epoch': 6.71}

 67%|██████▋   | 490/730 [43:25<22:02,  5.51s/it]
                                                 
{'train/learning_rate_real': 6.443114120841874e-06, 'epoch': 6.71}

 67%|██████▋   | 490/730 [43:25<22:02,  5.51s/it]
                                                 
{'debug/num_tok_total': 2604.0, 'debug/num_tok_loss': 1756.0, 'debug/num_lat_total': 2604.0, 'debug/num_lat_loss': 1756.0, 'epoch': 6.71}

 67%|██████▋   | 490/730 [43:26<22:02,  5.51s/it]
                                                 
{'train/ce_loss': 2.03125, 'train/diffusion_loss': 0.38808706402778625, 'epoch': 6.71}

 67%|██████▋   | 490/730 [43:26<22:02,  5.51s/it]
                                                 
{'train/learning_rate_real': 6.443114120841874e-06, 'epoch': 6.71}

 67%|██████▋   | 490/730 [43:26<22:02,  5.51s/it]
                                                 
{'debug/num_tok_total': 3095.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 3095.0, 'debug/num_lat_loss': 1790.0, 'epoch': 6.71}

 67%|██████▋   | 490/730 [43:28<22:02,  5.51s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.2910838723182678, 'epoch': 6.71}

 67%|██████▋   | 490/730 [43:28<22:02,  5.51s/it]
                                                 
{'train/learning_rate_real': 6.443114120841874e-06, 'epoch': 6.71}

 67%|██████▋   | 490/730 [43:28<22:02,  5.51s/it]
                                                 
{'debug/num_tok_total': 1805.0, 'debug/num_tok_loss': 1585.0, 'debug/num_lat_total': 1805.0, 'debug/num_lat_loss': 1585.0, 'epoch': 6.71}

 67%|██████▋   | 490/730 [43:29<22:02,  5.51s/it]
                                                 
{'train/ce_loss': 1.671875, 'train/diffusion_loss': 0.5143107771873474, 'epoch': 6.71}

 67%|██████▋   | 490/730 [43:29<22:02,  5.51s/it]
                                                 
{'train/learning_rate_real': 6.443114120841874e-06, 'epoch': 6.71}

 67%|██████▋   | 490/730 [43:29<22:02,  5.51s/it]
 67%|██████▋   | 491/730 [43:29<21:33,  5.41s/it]
                                                 
{'debug/num_tok_total': 2610.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 2610.0, 'debug/num_lat_loss': 1765.0, 'epoch': 6.73}

 67%|██████▋   | 491/730 [43:30<21:33,  5.41s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.3970773220062256, 'epoch': 6.73}

 67%|██████▋   | 491/730 [43:30<21:33,  5.41s/it]
                                                 
{'train/learning_rate_real': 6.394654323545815e-06, 'epoch': 6.73}

 67%|██████▋   | 491/730 [43:30<21:33,  5.41s/it]
                                                 
{'debug/num_tok_total': 2657.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2657.0, 'debug/num_lat_loss': 1792.0, 'epoch': 6.73}

 67%|██████▋   | 491/730 [43:32<21:33,  5.41s/it]
                                                 
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.39846810698509216, 'epoch': 6.73}

 67%|██████▋   | 491/730 [43:32<21:33,  5.41s/it]
                                                 
{'train/learning_rate_real': 6.394654323545815e-06, 'epoch': 6.73}

 67%|██████▋   | 491/730 [43:32<21:33,  5.41s/it]
                                                 
{'debug/num_tok_total': 2931.0, 'debug/num_tok_loss': 1710.0, 'debug/num_lat_total': 2931.0, 'debug/num_lat_loss': 1710.0, 'epoch': 6.73}

 67%|██████▋   | 491/730 [43:33<21:33,  5.41s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.3249343931674957, 'epoch': 6.73}

 67%|██████▋   | 491/730 [43:33<21:33,  5.41s/it]
                                                 
{'train/learning_rate_real': 6.394654323545815e-06, 'epoch': 6.73}

 67%|██████▋   | 491/730 [43:33<21:33,  5.41s/it]
                                                 
{'debug/num_tok_total': 3048.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 3048.0, 'debug/num_lat_loss': 1763.0, 'epoch': 6.73}

 67%|██████▋   | 491/730 [43:34<21:33,  5.41s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.3127351999282837, 'epoch': 6.73}

 67%|██████▋   | 491/730 [43:34<21:33,  5.41s/it]
                                                 
{'train/learning_rate_real': 6.394654323545815e-06, 'epoch': 6.73}

 67%|██████▋   | 491/730 [43:34<21:33,  5.41s/it]
 67%|██████▋   | 492/730 [43:35<21:36,  5.45s/it]
                                                 
{'debug/num_tok_total': 2645.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2645.0, 'debug/num_lat_loss': 1786.0, 'epoch': 6.74}

 67%|██████▋   | 492/730 [43:36<21:36,  5.45s/it]
                                                 
{'train/ce_loss': 1.921875, 'train/diffusion_loss': 0.39847704768180847, 'epoch': 6.74}

 67%|██████▋   | 492/730 [43:36<21:36,  5.45s/it]
                                                 
{'train/learning_rate_real': 6.3463147368527265e-06, 'epoch': 6.74}

 67%|██████▋   | 492/730 [43:36<21:36,  5.45s/it]
                                                 
{'debug/num_tok_total': 2839.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2839.0, 'debug/num_lat_loss': 1775.0, 'epoch': 6.74}

 67%|██████▋   | 492/730 [43:37<21:36,  5.45s/it]
                                                 
{'train/ce_loss': 1.8515625, 'train/diffusion_loss': 0.3508879244327545, 'epoch': 6.74}

 67%|██████▋   | 492/730 [43:37<21:36,  5.45s/it]
                                                 
{'train/learning_rate_real': 6.3463147368527265e-06, 'epoch': 6.74}

 67%|██████▋   | 492/730 [43:37<21:36,  5.45s/it]
                                                 
{'debug/num_tok_total': 2657.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2657.0, 'debug/num_lat_loss': 1784.0, 'epoch': 6.74}

 67%|██████▋   | 492/730 [43:39<21:36,  5.45s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.41999784111976624, 'epoch': 6.74}

 67%|██████▋   | 492/730 [43:39<21:36,  5.45s/it]
                                                 
{'train/learning_rate_real': 6.3463147368527265e-06, 'epoch': 6.74}

 67%|██████▋   | 492/730 [43:39<21:36,  5.45s/it]
                                                 
{'debug/num_tok_total': 3056.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 3056.0, 'debug/num_lat_loss': 1766.0, 'epoch': 6.74}

 67%|██████▋   | 492/730 [43:40<21:36,  5.45s/it]
                                                 
{'train/ce_loss': 1.9375, 'train/diffusion_loss': 0.31177952885627747, 'epoch': 6.74}

 67%|██████▋   | 492/730 [43:40<21:36,  5.45s/it]
                                                 
{'train/learning_rate_real': 6.3463147368527265e-06, 'epoch': 6.74}

 67%|██████▋   | 492/730 [43:40<21:36,  5.45s/it]
 68%|██████▊   | 493/730 [43:40<21:38,  5.48s/it]
                                                 
{'debug/num_tok_total': 2257.0, 'debug/num_tok_loss': 1620.0, 'debug/num_lat_total': 2257.0, 'debug/num_lat_loss': 1620.0, 'epoch': 6.75}

 68%|██████▊   | 493/730 [43:41<21:38,  5.48s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.42162320017814636, 'epoch': 6.75}

 68%|██████▊   | 493/730 [43:41<21:38,  5.48s/it]
                                                 
{'train/learning_rate_real': 6.298096312540147e-06, 'epoch': 6.75}

 68%|██████▊   | 493/730 [43:41<21:38,  5.48s/it]
                                                 
{'debug/num_tok_total': 2183.0, 'debug/num_tok_loss': 1758.0, 'debug/num_lat_total': 2183.0, 'debug/num_lat_loss': 1758.0, 'epoch': 6.75}

 68%|██████▊   | 493/730 [43:43<21:38,  5.48s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.484187513589859, 'epoch': 6.75}

 68%|██████▊   | 493/730 [43:43<21:38,  5.48s/it]
                                                 
{'train/learning_rate_real': 6.298096312540147e-06, 'epoch': 6.75}

 68%|██████▊   | 493/730 [43:43<21:38,  5.48s/it]
                                                 
{'debug/num_tok_total': 2203.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2203.0, 'debug/num_lat_loss': 1775.0, 'epoch': 6.75}

 68%|██████▊   | 493/730 [43:44<21:38,  5.48s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.4836789071559906, 'epoch': 6.75}

 68%|██████▊   | 493/730 [43:44<21:38,  5.48s/it]
                                                 
{'train/learning_rate_real': 6.298096312540147e-06, 'epoch': 6.75}

 68%|██████▊   | 493/730 [43:44<21:38,  5.48s/it]
                                                 
{'debug/num_tok_total': 2628.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2628.0, 'debug/num_lat_loss': 1780.0, 'epoch': 6.75}

 68%|██████▊   | 493/730 [43:45<21:38,  5.48s/it]
                                                 
{'train/ce_loss': 2.078125, 'train/diffusion_loss': 0.3942737281322479, 'epoch': 6.75}

 68%|██████▊   | 493/730 [43:45<21:38,  5.48s/it]
                                                 
{'train/learning_rate_real': 6.298096312540147e-06, 'epoch': 6.75}

 68%|██████▊   | 493/730 [43:45<21:38,  5.48s/it]
 68%|██████▊   | 494/730 [43:45<21:00,  5.34s/it]
                                                 
{'debug/num_tok_total': 2232.0, 'debug/num_tok_loss': 1800.0, 'debug/num_lat_total': 2232.0, 'debug/num_lat_loss': 1800.0, 'epoch': 6.77}

 68%|██████▊   | 494/730 [43:46<21:00,  5.34s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.5258052945137024, 'epoch': 6.77}

 68%|██████▊   | 494/730 [43:46<21:00,  5.34s/it]
                                                 
{'train/learning_rate_real': 6.250000000000003e-06, 'epoch': 6.77}

 68%|██████▊   | 494/730 [43:46<21:00,  5.34s/it]
                                                 
{'debug/num_tok_total': 2775.0, 'debug/num_tok_loss': 1632.0, 'debug/num_lat_total': 2775.0, 'debug/num_lat_loss': 1632.0, 'epoch': 6.77}

 68%|██████▊   | 494/730 [43:48<21:00,  5.34s/it]
                                                 
{'train/ce_loss': 1.9921875, 'train/diffusion_loss': 0.3061950206756592, 'epoch': 6.77}

 68%|██████▊   | 494/730 [43:48<21:00,  5.34s/it]
                                                 
{'train/learning_rate_real': 6.250000000000003e-06, 'epoch': 6.77}

 68%|██████▊   | 494/730 [43:48<21:00,  5.34s/it]
                                                 
{'debug/num_tok_total': 2886.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 2886.0, 'debug/num_lat_loss': 1797.0, 'epoch': 6.77}

 68%|██████▊   | 494/730 [43:49<21:00,  5.34s/it]
                                                 
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.35783007740974426, 'epoch': 6.77}

 68%|██████▊   | 494/730 [43:49<21:00,  5.34s/it]
                                                 
{'train/learning_rate_real': 6.250000000000003e-06, 'epoch': 6.77}

 68%|██████▊   | 494/730 [43:49<21:00,  5.34s/it]
                                                 
{'debug/num_tok_total': 2663.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 2663.0, 'debug/num_lat_loss': 1797.0, 'epoch': 6.77}

 68%|██████▊   | 494/730 [43:50<21:00,  5.34s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.3804702162742615, 'epoch': 6.77}

 68%|██████▊   | 494/730 [43:50<21:00,  5.34s/it]
                                                 
{'train/learning_rate_real': 6.250000000000003e-06, 'epoch': 6.77}

 68%|██████▊   | 494/730 [43:50<21:00,  5.34s/it]
 68%|██████▊   | 495/730 [43:51<20:55,  5.34s/it]
                                                 
{'debug/num_tok_total': 2225.0, 'debug/num_tok_loss': 1588.0, 'debug/num_lat_total': 2225.0, 'debug/num_lat_loss': 1588.0, 'epoch': 6.78}

 68%|██████▊   | 495/730 [43:52<20:55,  5.34s/it]
                                                 
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.4286140501499176, 'epoch': 6.78}

 68%|██████▊   | 495/730 [43:52<20:55,  5.34s/it]
                                                 
{'train/learning_rate_real': 6.202026746219899e-06, 'epoch': 6.78}

 68%|██████▊   | 495/730 [43:52<20:55,  5.34s/it]
                                                 
{'debug/num_tok_total': 2687.0, 'debug/num_tok_loss': 1595.0, 'debug/num_lat_total': 2687.0, 'debug/num_lat_loss': 1595.0, 'epoch': 6.78}

 68%|██████▊   | 495/730 [43:53<20:55,  5.34s/it]
                                                 
{'train/ce_loss': 1.671875, 'train/diffusion_loss': 0.3461858332157135, 'epoch': 6.78}

 68%|██████▊   | 495/730 [43:53<20:55,  5.34s/it]
                                                 
{'train/learning_rate_real': 6.202026746219899e-06, 'epoch': 6.78}

 68%|██████▊   | 495/730 [43:53<20:55,  5.34s/it]
                                                 
{'debug/num_tok_total': 2883.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2883.0, 'debug/num_lat_loss': 1796.0, 'epoch': 6.78}

 68%|██████▊   | 495/730 [43:54<20:55,  5.34s/it]
                                                 
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.358326256275177, 'epoch': 6.78}

 68%|██████▊   | 495/730 [43:54<20:55,  5.34s/it]
                                                 
{'train/learning_rate_real': 6.202026746219899e-06, 'epoch': 6.78}

 68%|██████▊   | 495/730 [43:54<20:55,  5.34s/it]
                                                 
{'debug/num_tok_total': 2243.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2243.0, 'debug/num_lat_loss': 1799.0, 'epoch': 6.78}

 68%|██████▊   | 495/730 [43:55<20:55,  5.34s/it]
                                                 
{'train/ce_loss': 2.109375, 'train/diffusion_loss': 0.48621073365211487, 'epoch': 6.78}

 68%|██████▊   | 495/730 [43:56<20:55,  5.34s/it]
                                                 
{'train/learning_rate_real': 6.202026746219899e-06, 'epoch': 6.78}

 68%|██████▊   | 495/730 [43:56<20:55,  5.34s/it]
 68%|██████▊   | 496/730 [43:56<20:45,  5.32s/it]
                                                 
{'debug/num_tok_total': 2212.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2212.0, 'debug/num_lat_loss': 1781.0, 'epoch': 6.79}

 68%|██████▊   | 496/730 [43:57<20:45,  5.32s/it]
                                                 
{'train/ce_loss': 1.9765625, 'train/diffusion_loss': 0.4894958734512329, 'epoch': 6.79}

 68%|██████▊   | 496/730 [43:57<20:45,  5.32s/it]
                                                 
{'train/learning_rate_real': 6.1541774957645045e-06, 'epoch': 6.79}

 68%|██████▊   | 496/730 [43:57<20:45,  5.32s/it]
                                                 
{'debug/num_tok_total': 2658.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2658.0, 'debug/num_lat_loss': 1792.0, 'epoch': 6.79}

 68%|██████▊   | 496/730 [43:58<20:45,  5.32s/it]
                                                 
{'train/ce_loss': 1.6640625, 'train/diffusion_loss': 0.3915718197822571, 'epoch': 6.79}

 68%|██████▊   | 496/730 [43:58<20:45,  5.32s/it]
                                                 
{'train/learning_rate_real': 6.1541774957645045e-06, 'epoch': 6.79}

 68%|██████▊   | 496/730 [43:58<20:45,  5.32s/it]
                                                 
{'debug/num_tok_total': 2406.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 2406.0, 'debug/num_lat_loss': 1769.0, 'epoch': 6.79}

 68%|██████▊   | 496/730 [43:59<20:45,  5.32s/it]
                                                 
{'train/ce_loss': 1.6484375, 'train/diffusion_loss': 0.4248881936073303, 'epoch': 6.79}

 68%|██████▊   | 496/730 [43:59<20:45,  5.32s/it]
                                                 
{'train/learning_rate_real': 6.1541774957645045e-06, 'epoch': 6.79}

 68%|██████▊   | 496/730 [43:59<20:45,  5.32s/it]
                                                 
{'debug/num_tok_total': 2430.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2430.0, 'debug/num_lat_loss': 1778.0, 'epoch': 6.79}

 68%|██████▊   | 496/730 [44:01<20:45,  5.32s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.43933916091918945, 'epoch': 6.79}

 68%|██████▊   | 496/730 [44:01<20:45,  5.32s/it]
                                                 
{'train/learning_rate_real': 6.1541774957645045e-06, 'epoch': 6.79}

 68%|██████▊   | 496/730 [44:01<20:45,  5.32s/it]
 68%|██████▊   | 497/730 [44:01<20:28,  5.27s/it]
                                                 
{'debug/num_tok_total': 2756.0, 'debug/num_tok_loss': 1670.0, 'debug/num_lat_total': 2756.0, 'debug/num_lat_loss': 1670.0, 'epoch': 6.81}

 68%|██████▊   | 497/730 [44:02<20:28,  5.27s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.3271120488643646, 'epoch': 6.81}

 68%|██████▊   | 497/730 [44:02<20:28,  5.27s/it]
                                                 
{'train/learning_rate_real': 6.106453190756918e-06, 'epoch': 6.81}

 68%|██████▊   | 497/730 [44:02<20:28,  5.27s/it]
                                                 
{'debug/num_tok_total': 2663.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 2663.0, 'debug/num_lat_loss': 1801.0, 'epoch': 6.81}

 68%|██████▊   | 497/730 [44:04<20:28,  5.27s/it]
                                                 
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.4082147479057312, 'epoch': 6.81}

 68%|██████▊   | 497/730 [44:04<20:28,  5.27s/it]
                                                 
{'train/learning_rate_real': 6.106453190756918e-06, 'epoch': 6.81}

 68%|██████▊   | 497/730 [44:04<20:28,  5.27s/it]
                                                 
{'debug/num_tok_total': 2445.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2445.0, 'debug/num_lat_loss': 1802.0, 'epoch': 6.81}

 68%|██████▊   | 497/730 [44:05<20:28,  5.27s/it]
                                                 
{'train/ce_loss': 1.9296875, 'train/diffusion_loss': 0.4544348120689392, 'epoch': 6.81}

 68%|██████▊   | 497/730 [44:05<20:28,  5.27s/it]
                                                 
{'train/learning_rate_real': 6.106453190756918e-06, 'epoch': 6.81}

 68%|██████▊   | 497/730 [44:05<20:28,  5.27s/it]
                                                 
{'debug/num_tok_total': 2019.0, 'debug/num_tok_loss': 1798.0, 'debug/num_lat_total': 2019.0, 'debug/num_lat_loss': 1798.0, 'epoch': 6.81}

 68%|██████▊   | 497/730 [44:06<20:28,  5.27s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.5195409059524536, 'epoch': 6.81}

 68%|██████▊   | 497/730 [44:06<20:28,  5.27s/it]
                                                 
{'train/learning_rate_real': 6.106453190756918e-06, 'epoch': 6.81}

 68%|██████▊   | 497/730 [44:06<20:28,  5.27s/it]
 68%|██████▊   | 498/730 [44:06<20:12,  5.23s/it]
                                                 
{'debug/num_tok_total': 2876.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 2876.0, 'debug/num_lat_loss': 1797.0, 'epoch': 6.82}

 68%|██████▊   | 498/730 [44:07<20:12,  5.23s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.3601064085960388, 'epoch': 6.82}

 68%|██████▊   | 498/730 [44:07<20:12,  5.23s/it]
                                                 
{'train/learning_rate_real': 6.058854770860153e-06, 'epoch': 6.82}

 68%|██████▊   | 498/730 [44:07<20:12,  5.23s/it]
                                                 
{'debug/num_tok_total': 2679.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2679.0, 'debug/num_lat_loss': 1799.0, 'epoch': 6.82}

 68%|██████▊   | 498/730 [44:09<20:12,  5.23s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.406594455242157, 'epoch': 6.82}

 68%|██████▊   | 498/730 [44:09<20:12,  5.23s/it]
                                                 
{'train/learning_rate_real': 6.058854770860153e-06, 'epoch': 6.82}

 68%|██████▊   | 498/730 [44:09<20:12,  5.23s/it]
                                                 
{'debug/num_tok_total': 2846.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2846.0, 'debug/num_lat_loss': 1780.0, 'epoch': 6.82}

 68%|██████▊   | 498/730 [44:10<20:12,  5.23s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.37573036551475525, 'epoch': 6.82}

 68%|██████▊   | 498/730 [44:10<20:12,  5.23s/it]
                                                 
{'train/learning_rate_real': 6.058854770860153e-06, 'epoch': 6.82}

 68%|██████▊   | 498/730 [44:10<20:12,  5.23s/it]
                                                 
{'debug/num_tok_total': 2237.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 2237.0, 'debug/num_lat_loss': 1797.0, 'epoch': 6.82}

 68%|██████▊   | 498/730 [44:11<20:12,  5.23s/it]
                                                 
{'train/ce_loss': 1.890625, 'train/diffusion_loss': 0.47708219289779663, 'epoch': 6.82}

 68%|██████▊   | 498/730 [44:11<20:12,  5.23s/it]
                                                 
{'train/learning_rate_real': 6.058854770860153e-06, 'epoch': 6.82}

 68%|██████▊   | 498/730 [44:11<20:12,  5.23s/it]
 68%|██████▊   | 499/730 [44:12<20:10,  5.24s/it]
                                                 
{'debug/num_tok_total': 2207.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2207.0, 'debug/num_lat_loss': 1773.0, 'epoch': 6.84}

 68%|██████▊   | 499/730 [44:13<20:10,  5.24s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.4737388491630554, 'epoch': 6.84}

 68%|██████▊   | 499/730 [44:13<20:10,  5.24s/it]
                                                 
{'train/learning_rate_real': 6.01138317325862e-06, 'epoch': 6.84}

 68%|██████▊   | 499/730 [44:13<20:10,  5.24s/it]
                                                 
{'debug/num_tok_total': 2840.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2840.0, 'debug/num_lat_loss': 1776.0, 'epoch': 6.84}

 68%|██████▊   | 499/730 [44:14<20:10,  5.24s/it]
                                                 
{'train/ce_loss': 1.9375, 'train/diffusion_loss': 0.37716373801231384, 'epoch': 6.84}

 68%|██████▊   | 499/730 [44:14<20:10,  5.24s/it]
                                                 
{'train/learning_rate_real': 6.01138317325862e-06, 'epoch': 6.84}

 68%|██████▊   | 499/730 [44:14<20:10,  5.24s/it]
                                                 
{'debug/num_tok_total': 1775.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 1775.0, 'debug/num_lat_loss': 1775.0, 'epoch': 6.84}

 68%|██████▊   | 499/730 [44:15<20:10,  5.24s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.5851710438728333, 'epoch': 6.84}

 68%|██████▊   | 499/730 [44:15<20:10,  5.24s/it]
                                                 
{'train/learning_rate_real': 6.01138317325862e-06, 'epoch': 6.84}

 68%|██████▊   | 499/730 [44:15<20:10,  5.24s/it]
                                                 
{'debug/num_tok_total': 2635.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2635.0, 'debug/num_lat_loss': 1785.0, 'epoch': 6.84}

 68%|██████▊   | 499/730 [44:16<20:10,  5.24s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.4042584002017975, 'epoch': 6.84}

 68%|██████▊   | 499/730 [44:16<20:10,  5.24s/it]
                                                 
{'train/learning_rate_real': 6.01138317325862e-06, 'epoch': 6.84}

 68%|██████▊   | 499/730 [44:16<20:10,  5.24s/it]03/16/2026 07:31:48 - INFO - __main__ - LoRA debug step 500: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 68%|██████▊   | 500/730 [44:16<19:37,  5.12s/it]
                                                 
{'loss': 2.5893, 'grad_norm': 1.2495994567871094, 'learning_rate': 6.01138317325862e-06, 'epoch': 6.85}

 68%|██████▊   | 500/730 [44:16<19:37,  5.12s/it]
                                                 
{'debug/num_tok_total': 2815.0, 'debug/num_tok_loss': 1756.0, 'debug/num_lat_total': 2815.0, 'debug/num_lat_loss': 1756.0, 'epoch': 6.85}

 68%|██████▊   | 500/730 [44:18<19:37,  5.12s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.35664883255958557, 'epoch': 6.85}

 68%|██████▊   | 500/730 [44:18<19:37,  5.12s/it]
                                                 
{'train/learning_rate_real': 5.964039332639656e-06, 'epoch': 6.85}

 68%|██████▊   | 500/730 [44:18<19:37,  5.12s/it]
                                                 
{'debug/num_tok_total': 2233.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 2233.0, 'debug/num_lat_loss': 1797.0, 'epoch': 6.85}

 68%|██████▊   | 500/730 [44:19<19:37,  5.12s/it]
                                                 
{'train/ce_loss': 1.6328125, 'train/diffusion_loss': 0.467814564704895, 'epoch': 6.85}

 68%|██████▊   | 500/730 [44:19<19:37,  5.12s/it]
                                                 
{'train/learning_rate_real': 5.964039332639656e-06, 'epoch': 6.85}

 68%|██████▊   | 500/730 [44:19<19:37,  5.12s/it]
                                                 
{'debug/num_tok_total': 2625.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2625.0, 'debug/num_lat_loss': 1771.0, 'epoch': 6.85}

 68%|██████▊   | 500/730 [44:20<19:37,  5.12s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.4203612506389618, 'epoch': 6.85}

 68%|██████▊   | 500/730 [44:20<19:37,  5.12s/it]
                                                 
{'train/learning_rate_real': 5.964039332639656e-06, 'epoch': 6.85}

 68%|██████▊   | 500/730 [44:20<19:37,  5.12s/it]
                                                 
{'debug/num_tok_total': 3262.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 3262.0, 'debug/num_lat_loss': 1773.0, 'epoch': 6.85}

 68%|██████▊   | 500/730 [44:21<19:37,  5.12s/it]
                                                 
{'train/ce_loss': 1.890625, 'train/diffusion_loss': 0.26534196734428406, 'epoch': 6.85}

 68%|██████▊   | 500/730 [44:21<19:37,  5.12s/it]
                                                 
{'train/learning_rate_real': 5.964039332639656e-06, 'epoch': 6.85}

 68%|██████▊   | 500/730 [44:21<19:37,  5.12s/it]
 69%|██████▊   | 501/730 [44:22<19:49,  5.19s/it]
                                                 
{'debug/num_tok_total': 2419.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2419.0, 'debug/num_lat_loss': 1772.0, 'epoch': 6.86}

 69%|██████▊   | 501/730 [44:23<19:49,  5.19s/it]
                                                 
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.4430917799472809, 'epoch': 6.86}

 69%|██████▊   | 501/730 [44:23<19:49,  5.19s/it]
                                                 
{'train/learning_rate_real': 5.916824181175162e-06, 'epoch': 6.86}

 69%|██████▊   | 501/730 [44:23<19:49,  5.19s/it]
                                                 
{'debug/num_tok_total': 3041.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 3041.0, 'debug/num_lat_loss': 1768.0, 'epoch': 6.86}

 69%|██████▊   | 501/730 [44:24<19:49,  5.19s/it]
                                                 
{'train/ce_loss': 1.96875, 'train/diffusion_loss': 0.30250975489616394, 'epoch': 6.86}

 69%|██████▊   | 501/730 [44:24<19:49,  5.19s/it]
                                                 
{'train/learning_rate_real': 5.916824181175162e-06, 'epoch': 6.86}

 69%|██████▊   | 501/730 [44:24<19:49,  5.19s/it]
                                                 
{'debug/num_tok_total': 3298.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 3298.0, 'debug/num_lat_loss': 1791.0, 'epoch': 6.86}

 69%|██████▊   | 501/730 [44:25<19:49,  5.19s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.2596778869628906, 'epoch': 6.86}

 69%|██████▊   | 501/730 [44:25<19:49,  5.19s/it]
                                                 
{'train/learning_rate_real': 5.916824181175162e-06, 'epoch': 6.86}

 69%|██████▊   | 501/730 [44:25<19:49,  5.19s/it]
                                                 
{'debug/num_tok_total': 1983.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 1983.0, 'debug/num_lat_loss': 1771.0, 'epoch': 6.86}

 69%|██████▊   | 501/730 [44:27<19:49,  5.19s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.5229732990264893, 'epoch': 6.86}

 69%|██████▊   | 501/730 [44:27<19:49,  5.19s/it]
                                                 
{'train/learning_rate_real': 5.916824181175162e-06, 'epoch': 6.86}

 69%|██████▊   | 501/730 [44:27<19:49,  5.19s/it]
 69%|██████▉   | 502/730 [44:27<19:48,  5.21s/it]
                                                 
{'debug/num_tok_total': 1980.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 1980.0, 'debug/num_lat_loss': 1774.0, 'epoch': 6.88}

 69%|██████▉   | 502/730 [44:28<19:48,  5.21s/it]
                                                 
{'train/ce_loss': 1.8515625, 'train/diffusion_loss': 0.5443798899650574, 'epoch': 6.88}

 69%|██████▉   | 502/730 [44:28<19:48,  5.21s/it]
                                                 
{'train/learning_rate_real': 5.86973864850322e-06, 'epoch': 6.88}

 69%|██████▉   | 502/730 [44:28<19:48,  5.21s/it]
                                                 
{'debug/num_tok_total': 3018.0, 'debug/num_tok_loss': 1751.0, 'debug/num_lat_total': 3018.0, 'debug/num_lat_loss': 1751.0, 'epoch': 6.88}

 69%|██████▉   | 502/730 [44:29<19:48,  5.21s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.30692437291145325, 'epoch': 6.88}

 69%|██████▉   | 502/730 [44:29<19:48,  5.21s/it]
                                                 
{'train/learning_rate_real': 5.86973864850322e-06, 'epoch': 6.88}

 69%|██████▉   | 502/730 [44:29<19:48,  5.21s/it]
                                                 
{'debug/num_tok_total': 2645.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2645.0, 'debug/num_lat_loss': 1785.0, 'epoch': 6.88}

 69%|██████▉   | 502/730 [44:30<19:48,  5.21s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.40486401319503784, 'epoch': 6.88}

 69%|██████▉   | 502/730 [44:30<19:48,  5.21s/it]
                                                 
{'train/learning_rate_real': 5.86973864850322e-06, 'epoch': 6.88}

 69%|██████▉   | 502/730 [44:30<19:48,  5.21s/it]
                                                 
{'debug/num_tok_total': 2502.0, 'debug/num_tok_loss': 1704.0, 'debug/num_lat_total': 2502.0, 'debug/num_lat_loss': 1704.0, 'epoch': 6.88}

 69%|██████▉   | 502/730 [44:32<19:48,  5.21s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.416448712348938, 'epoch': 6.88}

 69%|██████▉   | 502/730 [44:32<19:48,  5.21s/it]
                                                 
{'train/learning_rate_real': 5.86973864850322e-06, 'epoch': 6.88}

 69%|██████▉   | 502/730 [44:32<19:48,  5.21s/it]
 69%|██████▉   | 503/730 [44:32<19:41,  5.21s/it]
                                                 
{'debug/num_tok_total': 2834.0, 'debug/num_tok_loss': 1629.0, 'debug/num_lat_total': 2834.0, 'debug/num_lat_loss': 1629.0, 'epoch': 6.89}

 69%|██████▉   | 503/730 [44:33<19:41,  5.21s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.28291934728622437, 'epoch': 6.89}

 69%|██████▉   | 503/730 [44:33<19:41,  5.21s/it]
                                                 
{'train/learning_rate_real': 5.822783661709783e-06, 'epoch': 6.89}

 69%|██████▉   | 503/730 [44:33<19:41,  5.21s/it]
                                                 
{'debug/num_tok_total': 3040.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 3040.0, 'debug/num_lat_loss': 1765.0, 'epoch': 6.89}

 69%|██████▉   | 503/730 [44:35<19:41,  5.21s/it]
                                                 
{'train/ce_loss': 1.9140625, 'train/diffusion_loss': 0.3073432445526123, 'epoch': 6.89}

 69%|██████▉   | 503/730 [44:35<19:41,  5.21s/it]
                                                 
{'train/learning_rate_real': 5.822783661709783e-06, 'epoch': 6.89}

 69%|██████▉   | 503/730 [44:35<19:41,  5.21s/it]
                                                 
{'debug/num_tok_total': 3080.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 3080.0, 'debug/num_lat_loss': 1792.0, 'epoch': 6.89}

 69%|██████▉   | 503/730 [44:36<19:41,  5.21s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.3200998902320862, 'epoch': 6.89}

 69%|██████▉   | 503/730 [44:36<19:41,  5.21s/it]
                                                 
{'train/learning_rate_real': 5.822783661709783e-06, 'epoch': 6.89}

 69%|██████▉   | 503/730 [44:36<19:41,  5.21s/it]
                                                 
{'debug/num_tok_total': 3263.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 3263.0, 'debug/num_lat_loss': 1766.0, 'epoch': 6.89}

 69%|██████▉   | 503/730 [44:38<19:41,  5.21s/it]
                                                 
{'train/ce_loss': 1.6015625, 'train/diffusion_loss': 0.2872559726238251, 'epoch': 6.89}

 69%|██████▉   | 503/730 [44:38<19:41,  5.21s/it]
                                                 
{'train/learning_rate_real': 5.822783661709783e-06, 'epoch': 6.89}

 69%|██████▉   | 503/730 [44:38<19:41,  5.21s/it]
 69%|██████▉   | 504/730 [44:38<20:25,  5.42s/it]
                                                 
{'debug/num_tok_total': 2804.0, 'debug/num_tok_loss': 1751.0, 'debug/num_lat_total': 2804.0, 'debug/num_lat_loss': 1751.0, 'epoch': 6.9}

 69%|██████▉   | 504/730 [44:39<20:25,  5.42s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.34600263833999634, 'epoch': 6.9}

 69%|██████▉   | 504/730 [44:39<20:25,  5.42s/it]
                                                 
{'train/learning_rate_real': 5.775960145310452e-06, 'epoch': 6.9}

 69%|██████▉   | 504/730 [44:39<20:25,  5.42s/it]
                                                 
{'debug/num_tok_total': 2152.0, 'debug/num_tok_loss': 1707.0, 'debug/num_lat_total': 2152.0, 'debug/num_lat_loss': 1707.0, 'epoch': 6.9}

 69%|██████▉   | 504/730 [44:41<20:25,  5.42s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.5055350661277771, 'epoch': 6.9}

 69%|██████▉   | 504/730 [44:41<20:25,  5.42s/it]
                                                 
{'train/learning_rate_real': 5.775960145310452e-06, 'epoch': 6.9}

 69%|██████▉   | 504/730 [44:41<20:25,  5.42s/it]
                                                 
{'debug/num_tok_total': 2411.0, 'debug/num_tok_loss': 1759.0, 'debug/num_lat_total': 2411.0, 'debug/num_lat_loss': 1759.0, 'epoch': 6.9}

 69%|██████▉   | 504/730 [44:42<20:25,  5.42s/it]
                                                 
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.4250890910625458, 'epoch': 6.9}

 69%|██████▉   | 504/730 [44:42<20:25,  5.42s/it]
                                                 
{'train/learning_rate_real': 5.775960145310452e-06, 'epoch': 6.9}

 69%|██████▉   | 504/730 [44:42<20:25,  5.42s/it]
                                                 
{'debug/num_tok_total': 2835.0, 'debug/num_tok_loss': 1761.0, 'debug/num_lat_total': 2835.0, 'debug/num_lat_loss': 1761.0, 'epoch': 6.9}

 69%|██████▉   | 504/730 [44:43<20:25,  5.42s/it]
                                                 
{'train/ce_loss': 1.890625, 'train/diffusion_loss': 0.35115864872932434, 'epoch': 6.9}

 69%|██████▉   | 504/730 [44:43<20:25,  5.42s/it]
                                                 
{'train/learning_rate_real': 5.775960145310452e-06, 'epoch': 6.9}

 69%|██████▉   | 504/730 [44:43<20:25,  5.42s/it]
 69%|██████▉   | 505/730 [44:43<20:15,  5.40s/it]
                                                 
{'debug/num_tok_total': 2875.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2875.0, 'debug/num_lat_loss': 1786.0, 'epoch': 6.92}

 69%|██████▉   | 505/730 [44:45<20:15,  5.40s/it]
                                                 
{'train/ce_loss': 1.9296875, 'train/diffusion_loss': 0.383247971534729, 'epoch': 6.92}

 69%|██████▉   | 505/730 [44:45<20:15,  5.40s/it]
                                                 
{'train/learning_rate_real': 5.7292690212322375e-06, 'epoch': 6.92}

 69%|██████▉   | 505/730 [44:45<20:15,  5.40s/it]
                                                 
{'debug/num_tok_total': 2630.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2630.0, 'debug/num_lat_loss': 1778.0, 'epoch': 6.92}

 69%|██████▉   | 505/730 [44:46<20:15,  5.40s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.38524457812309265, 'epoch': 6.92}

 69%|██████▉   | 505/730 [44:46<20:15,  5.40s/it]
                                                 
{'train/learning_rate_real': 5.7292690212322375e-06, 'epoch': 6.92}

 69%|██████▉   | 505/730 [44:46<20:15,  5.40s/it]
                                                 
{'debug/num_tok_total': 2457.0, 'debug/num_tok_loss': 1800.0, 'debug/num_lat_total': 2457.0, 'debug/num_lat_loss': 1800.0, 'epoch': 6.92}

 69%|██████▉   | 505/730 [44:47<20:15,  5.40s/it]
                                                 
{'train/ce_loss': 1.6875, 'train/diffusion_loss': 0.45821723341941833, 'epoch': 6.92}

 69%|██████▉   | 505/730 [44:47<20:15,  5.40s/it]
                                                 
{'train/learning_rate_real': 5.7292690212322375e-06, 'epoch': 6.92}

 69%|██████▉   | 505/730 [44:47<20:15,  5.40s/it]
                                                 
{'debug/num_tok_total': 2750.0, 'debug/num_tok_loss': 1680.0, 'debug/num_lat_total': 2750.0, 'debug/num_lat_loss': 1680.0, 'epoch': 6.92}

 69%|██████▉   | 505/730 [44:48<20:15,  5.40s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.3501158356666565, 'epoch': 6.92}

 69%|██████▉   | 505/730 [44:48<20:15,  5.40s/it]
                                                 
{'train/learning_rate_real': 5.7292690212322375e-06, 'epoch': 6.92}

 69%|██████▉   | 505/730 [44:48<20:15,  5.40s/it]
 69%|██████▉   | 506/730 [44:49<20:11,  5.41s/it]
                                                 
{'debug/num_tok_total': 3269.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 3269.0, 'debug/num_lat_loss': 1769.0, 'epoch': 6.93}

 69%|██████▉   | 506/730 [44:50<20:11,  5.41s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.2662365436553955, 'epoch': 6.93}

 69%|██████▉   | 506/730 [44:50<20:11,  5.41s/it]
                                                 
{'train/learning_rate_real': 5.682711208795439e-06, 'epoch': 6.93}

 69%|██████▉   | 506/730 [44:50<20:11,  5.41s/it]
                                                 
{'debug/num_tok_total': 2255.0, 'debug/num_tok_loss': 1815.0, 'debug/num_lat_total': 2255.0, 'debug/num_lat_loss': 1815.0, 'epoch': 6.93}

 69%|██████▉   | 506/730 [44:51<20:11,  5.41s/it]
                                                 
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.48993542790412903, 'epoch': 6.93}

 69%|██████▉   | 506/730 [44:51<20:11,  5.41s/it]
                                                 
{'train/learning_rate_real': 5.682711208795439e-06, 'epoch': 6.93}

 69%|██████▉   | 506/730 [44:51<20:11,  5.41s/it]
                                                 
{'debug/num_tok_total': 2132.0, 'debug/num_tok_loss': 1721.0, 'debug/num_lat_total': 2132.0, 'debug/num_lat_loss': 1721.0, 'epoch': 6.93}

 69%|██████▉   | 506/730 [44:52<20:11,  5.41s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.4791540503501892, 'epoch': 6.93}

 69%|██████▉   | 506/730 [44:52<20:11,  5.41s/it]
                                                 
{'train/learning_rate_real': 5.682711208795439e-06, 'epoch': 6.93}

 69%|██████▉   | 506/730 [44:52<20:11,  5.41s/it]
                                                 
{'debug/num_tok_total': 3105.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 3105.0, 'debug/num_lat_loss': 1799.0, 'epoch': 6.93}

 69%|██████▉   | 506/730 [44:54<20:11,  5.41s/it]
                                                 
{'train/ce_loss': 1.90625, 'train/diffusion_loss': 0.3023548722267151, 'epoch': 6.93}

 69%|██████▉   | 506/730 [44:54<20:11,  5.41s/it]
                                                 
{'train/learning_rate_real': 5.682711208795439e-06, 'epoch': 6.93}

 69%|██████▉   | 506/730 [44:54<20:11,  5.41s/it]
 69%|██████▉   | 507/730 [44:54<19:55,  5.36s/it]
                                                 
{'debug/num_tok_total': 2899.0, 'debug/num_tok_loss': 1803.0, 'debug/num_lat_total': 2899.0, 'debug/num_lat_loss': 1803.0, 'epoch': 6.95}

 69%|██████▉   | 507/730 [44:55<19:55,  5.36s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.3727547228336334, 'epoch': 6.95}

 69%|██████▉   | 507/730 [44:55<19:55,  5.36s/it]
                                                 
{'train/learning_rate_real': 5.636287624695526e-06, 'epoch': 6.95}

 69%|██████▉   | 507/730 [44:55<19:55,  5.36s/it]
                                                 
{'debug/num_tok_total': 2626.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2626.0, 'debug/num_lat_loss': 1772.0, 'epoch': 6.95}

 69%|██████▉   | 507/730 [44:57<19:55,  5.36s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.41778868436813354, 'epoch': 6.95}

 69%|██████▉   | 507/730 [44:57<19:55,  5.36s/it]
                                                 
{'train/learning_rate_real': 5.636287624695526e-06, 'epoch': 6.95}

 69%|██████▉   | 507/730 [44:57<19:55,  5.36s/it]
                                                 
{'debug/num_tok_total': 2879.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 2879.0, 'debug/num_lat_loss': 1797.0, 'epoch': 6.95}

 69%|██████▉   | 507/730 [44:58<19:55,  5.36s/it]
                                                 
{'train/ce_loss': 1.9765625, 'train/diffusion_loss': 0.37614157795906067, 'epoch': 6.95}

 69%|██████▉   | 507/730 [44:58<19:55,  5.36s/it]
                                                 
{'train/learning_rate_real': 5.636287624695526e-06, 'epoch': 6.95}

 69%|██████▉   | 507/730 [44:58<19:55,  5.36s/it]
                                                 
{'debug/num_tok_total': 3044.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 3044.0, 'debug/num_lat_loss': 1768.0, 'epoch': 6.95}

 69%|██████▉   | 507/730 [44:59<19:55,  5.36s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.3197665512561798, 'epoch': 6.95}

 69%|██████▉   | 507/730 [44:59<19:55,  5.36s/it]
                                                 
{'train/learning_rate_real': 5.636287624695526e-06, 'epoch': 6.95}

 69%|██████▉   | 507/730 [44:59<19:55,  5.36s/it]
 70%|██████▉   | 508/730 [45:00<20:01,  5.41s/it]
                                                 
{'debug/num_tok_total': 2468.0, 'debug/num_tok_loss': 1814.0, 'debug/num_lat_total': 2468.0, 'debug/num_lat_loss': 1814.0, 'epoch': 6.96}

 70%|██████▉   | 508/730 [45:01<20:01,  5.41s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.4490810036659241, 'epoch': 6.96}

 70%|██████▉   | 508/730 [45:01<20:01,  5.41s/it]
                                                 
{'train/learning_rate_real': 5.589999182985087e-06, 'epoch': 6.96}

 70%|██████▉   | 508/730 [45:01<20:01,  5.41s/it]
                                                 
{'debug/num_tok_total': 2155.0, 'debug/num_tok_loss': 1707.0, 'debug/num_lat_total': 2155.0, 'debug/num_lat_loss': 1707.0, 'epoch': 6.96}

 70%|██████▉   | 508/730 [45:02<20:01,  5.41s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.48848262429237366, 'epoch': 6.96}

 70%|██████▉   | 508/730 [45:02<20:01,  5.41s/it]
                                                 
{'train/learning_rate_real': 5.589999182985087e-06, 'epoch': 6.96}

 70%|██████▉   | 508/730 [45:02<20:01,  5.41s/it]
                                                 
{'debug/num_tok_total': 2009.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2009.0, 'debug/num_lat_loss': 1788.0, 'epoch': 6.96}

 70%|██████▉   | 508/730 [45:03<20:01,  5.41s/it]
                                                 
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.5553423166275024, 'epoch': 6.96}

 70%|██████▉   | 508/730 [45:03<20:01,  5.41s/it]
                                                 
{'train/learning_rate_real': 5.589999182985087e-06, 'epoch': 6.96}

 70%|██████▉   | 508/730 [45:03<20:01,  5.41s/it]
                                                 
{'debug/num_tok_total': 2374.0, 'debug/num_tok_loss': 1494.0, 'debug/num_lat_total': 2374.0, 'debug/num_lat_loss': 1494.0, 'epoch': 6.96}

 70%|██████▉   | 508/730 [45:04<20:01,  5.41s/it]
                                                 
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.38217484951019287, 'epoch': 6.96}

 70%|██████▉   | 508/730 [45:04<20:01,  5.41s/it]
                                                 
{'train/learning_rate_real': 5.589999182985087e-06, 'epoch': 6.96}

 70%|██████▉   | 508/730 [45:04<20:01,  5.41s/it]
 70%|██████▉   | 509/730 [45:05<19:25,  5.28s/it]
                                                 
{'debug/num_tok_total': 3324.0, 'debug/num_tok_loss': 1800.0, 'debug/num_lat_total': 3324.0, 'debug/num_lat_loss': 1800.0, 'epoch': 6.97}

 70%|██████▉   | 509/730 [45:06<19:25,  5.28s/it]
                                                 
{'train/ce_loss': 1.6875, 'train/diffusion_loss': 0.2931516170501709, 'epoch': 6.97}

 70%|██████▉   | 509/730 [45:06<19:25,  5.28s/it]
                                                 
{'train/learning_rate_real': 5.543846795055834e-06, 'epoch': 6.97}

 70%|██████▉   | 509/730 [45:06<19:25,  5.28s/it]
                                                 
{'debug/num_tok_total': 2830.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2830.0, 'debug/num_lat_loss': 1773.0, 'epoch': 6.97}

 70%|██████▉   | 509/730 [45:07<19:25,  5.28s/it]
                                                 
{'train/ce_loss': 1.578125, 'train/diffusion_loss': 0.32618090510368347, 'epoch': 6.97}

 70%|██████▉   | 509/730 [45:07<19:25,  5.28s/it]
                                                 
{'train/learning_rate_real': 5.543846795055834e-06, 'epoch': 6.97}

 70%|██████▉   | 509/730 [45:07<19:25,  5.28s/it]
                                                 
{'debug/num_tok_total': 3421.0, 'debug/num_tok_loss': 1740.0, 'debug/num_lat_total': 3421.0, 'debug/num_lat_loss': 1740.0, 'epoch': 6.97}

 70%|██████▉   | 509/730 [45:09<19:25,  5.28s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.23644323647022247, 'epoch': 6.97}

 70%|██████▉   | 509/730 [45:09<19:25,  5.28s/it]
                                                 
{'train/learning_rate_real': 5.543846795055834e-06, 'epoch': 6.97}

 70%|██████▉   | 509/730 [45:09<19:25,  5.28s/it]
                                                 
{'debug/num_tok_total': 2437.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2437.0, 'debug/num_lat_loss': 1785.0, 'epoch': 6.97}

 70%|██████▉   | 509/730 [45:10<19:25,  5.28s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.47299614548683167, 'epoch': 6.97}

 70%|██████▉   | 509/730 [45:10<19:25,  5.28s/it]
                                                 
{'train/learning_rate_real': 5.543846795055834e-06, 'epoch': 6.97}

 70%|██████▉   | 509/730 [45:10<19:25,  5.28s/it]03/16/2026 07:32:42 - INFO - __main__ - LoRA debug step 510: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 70%|██████▉   | 510/730 [45:10<19:51,  5.42s/it]
                                                 
{'loss': 2.4339, 'grad_norm': 1.2261403799057007, 'learning_rate': 5.543846795055834e-06, 'epoch': 6.99}

 70%|██████▉   | 510/730 [45:10<19:51,  5.42s/it]
                                                 
{'debug/num_tok_total': 2451.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2451.0, 'debug/num_lat_loss': 1796.0, 'epoch': 6.99}

 70%|██████▉   | 510/730 [45:11<19:51,  5.42s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.41865476965904236, 'epoch': 6.99}

 70%|██████▉   | 510/730 [45:11<19:51,  5.42s/it]
                                                 
{'train/learning_rate_real': 5.4978313696206734e-06, 'epoch': 6.99}

 70%|██████▉   | 510/730 [45:11<19:51,  5.42s/it]
                                                 
{'debug/num_tok_total': 2655.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2655.0, 'debug/num_lat_loss': 1792.0, 'epoch': 6.99}

 70%|██████▉   | 510/730 [45:13<19:51,  5.42s/it]
                                                 
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.3996216952800751, 'epoch': 6.99}

 70%|██████▉   | 510/730 [45:13<19:51,  5.42s/it]
                                                 
{'train/learning_rate_real': 5.4978313696206734e-06, 'epoch': 6.99}

 70%|██████▉   | 510/730 [45:13<19:51,  5.42s/it]
                                                 
{'debug/num_tok_total': 2622.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2622.0, 'debug/num_lat_loss': 1771.0, 'epoch': 6.99}

 70%|██████▉   | 510/730 [45:14<19:51,  5.42s/it]
                                                 
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.38985252380371094, 'epoch': 6.99}

 70%|██████▉   | 510/730 [45:14<19:51,  5.42s/it]
                                                 
{'train/learning_rate_real': 5.4978313696206734e-06, 'epoch': 6.99}

 70%|██████▉   | 510/730 [45:14<19:51,  5.42s/it]
                                                 
{'debug/num_tok_total': 869.0, 'debug/num_tok_loss': 442.0, 'debug/num_lat_total': 869.0, 'debug/num_lat_loss': 442.0, 'epoch': 6.99}

 70%|██████▉   | 510/730 [45:14<19:51,  5.42s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.20271852612495422, 'epoch': 6.99}

 70%|██████▉   | 510/730 [45:14<19:51,  5.42s/it]
                                                 
{'train/learning_rate_real': 5.4978313696206734e-06, 'epoch': 6.99}

 70%|██████▉   | 510/730 [45:14<19:51,  5.42s/it]
 70%|███████   | 511/730 [45:15<18:32,  5.08s/it]
                                                 
{'debug/num_tok_total': 2196.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 2196.0, 'debug/num_lat_loss': 1769.0, 'epoch': 7.0}

 70%|███████   | 511/730 [45:16<18:32,  5.08s/it]
                                                 
{'train/ce_loss': 2.078125, 'train/diffusion_loss': 0.4979832172393799, 'epoch': 7.0}

 70%|███████   | 511/730 [45:16<18:32,  5.08s/it]
                                                 
{'train/learning_rate_real': 5.451953812695798e-06, 'epoch': 7.0}

 70%|███████   | 511/730 [45:16<18:32,  5.08s/it]
                                                 
{'debug/num_tok_total': 2448.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2448.0, 'debug/num_lat_loss': 1787.0, 'epoch': 7.0}

 70%|███████   | 511/730 [45:17<18:32,  5.08s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.4312325716018677, 'epoch': 7.0}

 70%|███████   | 511/730 [45:17<18:32,  5.08s/it]
                                                 
{'train/learning_rate_real': 5.451953812695798e-06, 'epoch': 7.0}

 70%|███████   | 511/730 [45:17<18:32,  5.08s/it]
                                                 
{'debug/num_tok_total': 2665.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2665.0, 'debug/num_lat_loss': 1795.0, 'epoch': 7.0}

 70%|███████   | 511/730 [45:18<18:32,  5.08s/it]
                                                 
{'train/ce_loss': 1.9375, 'train/diffusion_loss': 0.3989337384700775, 'epoch': 7.0}

 70%|███████   | 511/730 [45:18<18:32,  5.08s/it]
                                                 
{'train/learning_rate_real': 5.451953812695798e-06, 'epoch': 7.0}

 70%|███████   | 511/730 [45:18<18:32,  5.08s/it]
                                                 
{'debug/num_tok_total': 2468.0, 'debug/num_tok_loss': 1810.0, 'debug/num_lat_total': 2468.0, 'debug/num_lat_loss': 1810.0, 'epoch': 7.0}

 70%|███████   | 511/730 [45:19<18:32,  5.08s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.45066267251968384, 'epoch': 7.0}

 70%|███████   | 511/730 [45:19<18:32,  5.08s/it]
                                                 
{'train/learning_rate_real': 5.451953812695798e-06, 'epoch': 7.0}

 70%|███████   | 511/730 [45:19<18:32,  5.08s/it]
 70%|███████   | 512/730 [45:20<18:32,  5.10s/it]
                                                 
{'debug/num_tok_total': 2226.0, 'debug/num_tok_loss': 1527.0, 'debug/num_lat_total': 2226.0, 'debug/num_lat_loss': 1527.0, 'epoch': 7.01}

 70%|███████   | 512/730 [45:21<18:32,  5.10s/it]
                                                 
{'train/ce_loss': 1.8515625, 'train/diffusion_loss': 0.42891430854797363, 'epoch': 7.01}

 70%|███████   | 512/730 [45:21<18:32,  5.10s/it]
                                                 
{'train/learning_rate_real': 5.406215027582855e-06, 'epoch': 7.01}

 70%|███████   | 512/730 [45:21<18:32,  5.10s/it]
                                                 
{'debug/num_tok_total': 2455.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2455.0, 'debug/num_lat_loss': 1802.0, 'epoch': 7.01}

 70%|███████   | 512/730 [45:22<18:32,  5.10s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.4596862494945526, 'epoch': 7.01}

 70%|███████   | 512/730 [45:22<18:32,  5.10s/it]
                                                 
{'train/learning_rate_real': 5.406215027582855e-06, 'epoch': 7.01}

 70%|███████   | 512/730 [45:22<18:32,  5.10s/it]
                                                 
{'debug/num_tok_total': 2857.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2857.0, 'debug/num_lat_loss': 1787.0, 'epoch': 7.01}

 70%|███████   | 512/730 [45:23<18:32,  5.10s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.3518066704273224, 'epoch': 7.01}

 70%|███████   | 512/730 [45:23<18:32,  5.10s/it]
                                                 
{'train/learning_rate_real': 5.406215027582855e-06, 'epoch': 7.01}

 70%|███████   | 512/730 [45:23<18:32,  5.10s/it]
                                                 
{'debug/num_tok_total': 2442.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2442.0, 'debug/num_lat_loss': 1796.0, 'epoch': 7.01}

 70%|███████   | 512/730 [45:25<18:32,  5.10s/it]
                                                 
{'train/ce_loss': 2.078125, 'train/diffusion_loss': 0.42538321018218994, 'epoch': 7.01}

 70%|███████   | 512/730 [45:25<18:32,  5.10s/it]
                                                 
{'train/learning_rate_real': 5.406215027582855e-06, 'epoch': 7.01}

 70%|███████   | 512/730 [45:25<18:32,  5.10s/it]
 70%|███████   | 513/730 [45:25<18:40,  5.16s/it]
                                                 
{'debug/num_tok_total': 2359.0, 'debug/num_tok_loss': 1715.0, 'debug/num_lat_total': 2359.0, 'debug/num_lat_loss': 1715.0, 'epoch': 7.03}

 70%|███████   | 513/730 [45:26<18:40,  5.16s/it]
                                                 
{'train/ce_loss': 2.296875, 'train/diffusion_loss': 0.455152690410614, 'epoch': 7.03}

 70%|███████   | 513/730 [45:26<18:40,  5.16s/it]
                                                 
{'train/learning_rate_real': 5.360615914851147e-06, 'epoch': 7.03}

 70%|███████   | 513/730 [45:26<18:40,  5.16s/it]
                                                 
{'debug/num_tok_total': 2438.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2438.0, 'debug/num_lat_loss': 1791.0, 'epoch': 7.03}

 70%|███████   | 513/730 [45:27<18:40,  5.16s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.46227774024009705, 'epoch': 7.03}

 70%|███████   | 513/730 [45:27<18:40,  5.16s/it]
                                                 
{'train/learning_rate_real': 5.360615914851147e-06, 'epoch': 7.03}

 70%|███████   | 513/730 [45:27<18:40,  5.16s/it]
                                                 
{'debug/num_tok_total': 3058.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 3058.0, 'debug/num_lat_loss': 1770.0, 'epoch': 7.03}

 70%|███████   | 513/730 [45:29<18:40,  5.16s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.3015802204608917, 'epoch': 7.03}

 70%|███████   | 513/730 [45:29<18:40,  5.16s/it]
                                                 
{'train/learning_rate_real': 5.360615914851147e-06, 'epoch': 7.03}

 70%|███████   | 513/730 [45:29<18:40,  5.16s/it]
                                                 
{'debug/num_tok_total': 2462.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 2462.0, 'debug/num_lat_loss': 1797.0, 'epoch': 7.03}

 70%|███████   | 513/730 [45:30<18:40,  5.16s/it]
                                                 
{'train/ce_loss': 1.9296875, 'train/diffusion_loss': 0.44757771492004395, 'epoch': 7.03}

 70%|███████   | 513/730 [45:30<18:40,  5.16s/it]
                                                 
{'train/learning_rate_real': 5.360615914851147e-06, 'epoch': 7.03}

 70%|███████   | 513/730 [45:30<18:40,  5.16s/it]
 70%|███████   | 514/730 [45:30<18:39,  5.18s/it]
                                                 
{'debug/num_tok_total': 2819.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2819.0, 'debug/num_lat_loss': 1771.0, 'epoch': 7.04}

 70%|███████   | 514/730 [45:32<18:39,  5.18s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.3520268499851227, 'epoch': 7.04}

 70%|███████   | 514/730 [45:32<18:39,  5.18s/it]
                                                 
{'train/learning_rate_real': 5.315157372319915e-06, 'epoch': 7.04}

 70%|███████   | 514/730 [45:32<18:39,  5.18s/it]
                                                 
{'debug/num_tok_total': 2279.0, 'debug/num_tok_loss': 1705.0, 'debug/num_lat_total': 2279.0, 'debug/num_lat_loss': 1705.0, 'epoch': 7.04}

 70%|███████   | 514/730 [45:33<18:39,  5.18s/it]
                                                 
{'train/ce_loss': 1.8984375, 'train/diffusion_loss': 0.4434811770915985, 'epoch': 7.04}

 70%|███████   | 514/730 [45:33<18:39,  5.18s/it]
                                                 
{'train/learning_rate_real': 5.315157372319915e-06, 'epoch': 7.04}

 70%|███████   | 514/730 [45:33<18:39,  5.18s/it]
                                                 
{'debug/num_tok_total': 2864.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2864.0, 'debug/num_lat_loss': 1779.0, 'epoch': 7.04}

 70%|███████   | 514/730 [45:34<18:39,  5.18s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.36410924792289734, 'epoch': 7.04}

 70%|███████   | 514/730 [45:34<18:39,  5.18s/it]
                                                 
{'train/learning_rate_real': 5.315157372319915e-06, 'epoch': 7.04}

 70%|███████   | 514/730 [45:34<18:39,  5.18s/it]
                                                 
{'debug/num_tok_total': 2463.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2463.0, 'debug/num_lat_loss': 1802.0, 'epoch': 7.04}

 70%|███████   | 514/730 [45:35<18:39,  5.18s/it]
                                                 
{'train/ce_loss': 1.9140625, 'train/diffusion_loss': 0.4653569161891937, 'epoch': 7.04}

 70%|███████   | 514/730 [45:35<18:39,  5.18s/it]
                                                 
{'train/learning_rate_real': 5.315157372319915e-06, 'epoch': 7.04}

 70%|███████   | 514/730 [45:35<18:39,  5.18s/it]
 71%|███████   | 515/730 [45:36<18:50,  5.26s/it]
                                                 
{'debug/num_tok_total': 2487.0, 'debug/num_tok_loss': 1703.0, 'debug/num_lat_total': 2487.0, 'debug/num_lat_loss': 1703.0, 'epoch': 7.05}

 71%|███████   | 515/730 [45:37<18:50,  5.26s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.4114701747894287, 'epoch': 7.05}

 71%|███████   | 515/730 [45:37<18:50,  5.26s/it]
                                                 
{'train/learning_rate_real': 5.269840295040664e-06, 'epoch': 7.05}

 71%|███████   | 515/730 [45:37<18:50,  5.26s/it]
                                                 
{'debug/num_tok_total': 3038.0, 'debug/num_tok_loss': 1758.0, 'debug/num_lat_total': 3038.0, 'debug/num_lat_loss': 1758.0, 'epoch': 7.05}

 71%|███████   | 515/730 [45:38<18:50,  5.26s/it]
                                                 
{'train/ce_loss': 1.625, 'train/diffusion_loss': 0.3299405574798584, 'epoch': 7.05}

 71%|███████   | 515/730 [45:38<18:50,  5.26s/it]
                                                 
{'train/learning_rate_real': 5.269840295040664e-06, 'epoch': 7.05}

 71%|███████   | 515/730 [45:38<18:50,  5.26s/it]
                                                 
{'debug/num_tok_total': 2228.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2228.0, 'debug/num_lat_loss': 1790.0, 'epoch': 7.05}

 71%|███████   | 515/730 [45:39<18:50,  5.26s/it]
                                                 
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.4894968271255493, 'epoch': 7.05}

 71%|███████   | 515/730 [45:39<18:50,  5.26s/it]
                                                 
{'train/learning_rate_real': 5.269840295040664e-06, 'epoch': 7.05}

 71%|███████   | 515/730 [45:39<18:50,  5.26s/it]
                                                 
{'debug/num_tok_total': 2846.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2846.0, 'debug/num_lat_loss': 1782.0, 'epoch': 7.05}

 71%|███████   | 515/730 [45:41<18:50,  5.26s/it]
                                                 
{'train/ce_loss': 1.65625, 'train/diffusion_loss': 0.33853039145469666, 'epoch': 7.05}

 71%|███████   | 515/730 [45:41<18:50,  5.26s/it]
                                                 
{'train/learning_rate_real': 5.269840295040664e-06, 'epoch': 7.05}

 71%|███████   | 515/730 [45:41<18:50,  5.26s/it]
 71%|███████   | 516/730 [45:41<18:50,  5.28s/it]
                                                 
{'debug/num_tok_total': 3053.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 3053.0, 'debug/num_lat_loss': 1779.0, 'epoch': 7.07}

 71%|███████   | 516/730 [45:42<18:50,  5.28s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.3660178482532501, 'epoch': 7.07}

 71%|███████   | 516/730 [45:42<18:50,  5.28s/it]
                                                 
{'train/learning_rate_real': 5.224665575279532e-06, 'epoch': 7.07}

 71%|███████   | 516/730 [45:42<18:50,  5.28s/it]
                                                 
{'debug/num_tok_total': 2650.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2650.0, 'debug/num_lat_loss': 1786.0, 'epoch': 7.07}

 71%|███████   | 516/730 [45:44<18:50,  5.28s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.4022609293460846, 'epoch': 7.07}

 71%|███████   | 516/730 [45:44<18:50,  5.28s/it]
                                                 
{'train/learning_rate_real': 5.224665575279532e-06, 'epoch': 7.07}

 71%|███████   | 516/730 [45:44<18:50,  5.28s/it]
                                                 
{'debug/num_tok_total': 2454.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2454.0, 'debug/num_lat_loss': 1802.0, 'epoch': 7.07}

 71%|███████   | 516/730 [45:45<18:50,  5.28s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.4582630693912506, 'epoch': 7.07}

 71%|███████   | 516/730 [45:45<18:50,  5.28s/it]
                                                 
{'train/learning_rate_real': 5.224665575279532e-06, 'epoch': 7.07}

 71%|███████   | 516/730 [45:45<18:50,  5.28s/it]
                                                 
{'debug/num_tok_total': 2263.0, 'debug/num_tok_loss': 1610.0, 'debug/num_lat_total': 2263.0, 'debug/num_lat_loss': 1610.0, 'epoch': 7.07}

 71%|███████   | 516/730 [45:46<18:50,  5.28s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.4195789396762848, 'epoch': 7.07}

 71%|███████   | 516/730 [45:46<18:50,  5.28s/it]
                                                 
{'train/learning_rate_real': 5.224665575279532e-06, 'epoch': 7.07}

 71%|███████   | 516/730 [45:46<18:50,  5.28s/it]
 71%|███████   | 517/730 [45:46<18:43,  5.27s/it]
                                                 
{'debug/num_tok_total': 2754.0, 'debug/num_tok_loss': 1680.0, 'debug/num_lat_total': 2754.0, 'debug/num_lat_loss': 1680.0, 'epoch': 7.08}

 71%|███████   | 517/730 [45:48<18:43,  5.27s/it]
                                                 
{'train/ce_loss': 1.9140625, 'train/diffusion_loss': 0.3254970610141754, 'epoch': 7.08}

 71%|███████   | 517/730 [45:48<18:43,  5.27s/it]
                                                 
{'train/learning_rate_real': 5.1796341024997085e-06, 'epoch': 7.08}

 71%|███████   | 517/730 [45:48<18:43,  5.27s/it]
                                                 
{'debug/num_tok_total': 2977.0, 'debug/num_tok_loss': 1621.0, 'debug/num_lat_total': 2977.0, 'debug/num_lat_loss': 1621.0, 'epoch': 7.08}

 71%|███████   | 517/730 [45:49<18:43,  5.27s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.2587592601776123, 'epoch': 7.08}

 71%|███████   | 517/730 [45:49<18:43,  5.27s/it]
                                                 
{'train/learning_rate_real': 5.1796341024997085e-06, 'epoch': 7.08}

 71%|███████   | 517/730 [45:49<18:43,  5.27s/it]
                                                 
{'debug/num_tok_total': 2148.0, 'debug/num_tok_loss': 1711.0, 'debug/num_lat_total': 2148.0, 'debug/num_lat_loss': 1711.0, 'epoch': 7.08}

 71%|███████   | 517/730 [45:50<18:43,  5.27s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.44445499777793884, 'epoch': 7.08}

 71%|███████   | 517/730 [45:50<18:43,  5.27s/it]
                                                 
{'train/learning_rate_real': 5.1796341024997085e-06, 'epoch': 7.08}

 71%|███████   | 517/730 [45:50<18:43,  5.27s/it]
                                                 
{'debug/num_tok_total': 2329.0, 'debug/num_tok_loss': 1623.0, 'debug/num_lat_total': 2329.0, 'debug/num_lat_loss': 1623.0, 'epoch': 7.08}

 71%|███████   | 517/730 [45:51<18:43,  5.27s/it]
                                                 
{'train/ce_loss': 1.6875, 'train/diffusion_loss': 0.4028322100639343, 'epoch': 7.08}

 71%|███████   | 517/730 [45:51<18:43,  5.27s/it]
                                                 
{'train/learning_rate_real': 5.1796341024997085e-06, 'epoch': 7.08}

 71%|███████   | 517/730 [45:51<18:43,  5.27s/it]
 71%|███████   | 518/730 [45:52<18:51,  5.34s/it]
                                                 
{'debug/num_tok_total': 2865.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2865.0, 'debug/num_lat_loss': 1776.0, 'epoch': 7.1}

 71%|███████   | 518/730 [45:53<18:51,  5.34s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.3416700065135956, 'epoch': 7.1}

 71%|███████   | 518/730 [45:53<18:51,  5.34s/it]
                                                 
{'train/learning_rate_real': 5.134746763343958e-06, 'epoch': 7.1}

 71%|███████   | 518/730 [45:53<18:51,  5.34s/it]
                                                 
{'debug/num_tok_total': 2614.0, 'debug/num_tok_loss': 1527.0, 'debug/num_lat_total': 2614.0, 'debug/num_lat_loss': 1527.0, 'epoch': 7.1}

 71%|███████   | 518/730 [45:54<18:51,  5.34s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.3446652889251709, 'epoch': 7.1}

 71%|███████   | 518/730 [45:54<18:51,  5.34s/it]
                                                 
{'train/learning_rate_real': 5.134746763343958e-06, 'epoch': 7.1}

 71%|███████   | 518/730 [45:54<18:51,  5.34s/it]
                                                 
{'debug/num_tok_total': 3077.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 3077.0, 'debug/num_lat_loss': 1778.0, 'epoch': 7.1}

 71%|███████   | 518/730 [45:56<18:51,  5.34s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.3180380165576935, 'epoch': 7.1}

 71%|███████   | 518/730 [45:56<18:51,  5.34s/it]
                                                 
{'train/learning_rate_real': 5.134746763343958e-06, 'epoch': 7.1}

 71%|███████   | 518/730 [45:56<18:51,  5.34s/it]
                                                 
{'debug/num_tok_total': 2646.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2646.0, 'debug/num_lat_loss': 1788.0, 'epoch': 7.1}

 71%|███████   | 518/730 [45:57<18:51,  5.34s/it]
                                                 
{'train/ce_loss': 1.6640625, 'train/diffusion_loss': 0.4019910395145416, 'epoch': 7.1}

 71%|███████   | 518/730 [45:57<18:51,  5.34s/it]
                                                 
{'train/learning_rate_real': 5.134746763343958e-06, 'epoch': 7.1}

 71%|███████   | 518/730 [45:57<18:51,  5.34s/it]
 71%|███████   | 519/730 [45:57<18:59,  5.40s/it]
                                                 
{'debug/num_tok_total': 2082.0, 'debug/num_tok_loss': 1650.0, 'debug/num_lat_total': 2082.0, 'debug/num_lat_loss': 1650.0, 'epoch': 7.11}

 71%|███████   | 519/730 [45:58<18:59,  5.40s/it]
                                                 
{'train/ce_loss': 1.6875, 'train/diffusion_loss': 0.48401737213134766, 'epoch': 7.11}

 71%|███████   | 519/730 [45:58<18:59,  5.40s/it]
                                                 
{'train/learning_rate_real': 5.090004441617116e-06, 'epoch': 7.11}

 71%|███████   | 519/730 [45:58<18:59,  5.40s/it]
                                                 
{'debug/num_tok_total': 2837.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2837.0, 'debug/num_lat_loss': 1773.0, 'epoch': 7.11}

 71%|███████   | 519/730 [46:00<18:59,  5.40s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.36429744958877563, 'epoch': 7.11}

 71%|███████   | 519/730 [46:00<18:59,  5.40s/it]
                                                 
{'train/learning_rate_real': 5.090004441617116e-06, 'epoch': 7.11}

 71%|███████   | 519/730 [46:00<18:59,  5.40s/it]
                                                 
{'debug/num_tok_total': 2582.0, 'debug/num_tok_loss': 1740.0, 'debug/num_lat_total': 2582.0, 'debug/num_lat_loss': 1740.0, 'epoch': 7.11}

 71%|███████   | 519/730 [46:01<18:59,  5.40s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.39463311433792114, 'epoch': 7.11}

 71%|███████   | 519/730 [46:01<18:59,  5.40s/it]
                                                 
{'train/learning_rate_real': 5.090004441617116e-06, 'epoch': 7.11}

 71%|███████   | 519/730 [46:01<18:59,  5.40s/it]
                                                 
{'debug/num_tok_total': 3077.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 3077.0, 'debug/num_lat_loss': 1781.0, 'epoch': 7.11}

 71%|███████   | 519/730 [46:02<18:59,  5.40s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.3135707676410675, 'epoch': 7.11}

 71%|███████   | 519/730 [46:02<18:59,  5.40s/it]
                                                 
{'train/learning_rate_real': 5.090004441617116e-06, 'epoch': 7.11}

 71%|███████   | 519/730 [46:02<18:59,  5.40s/it]03/16/2026 07:33:35 - INFO - __main__ - LoRA debug step 520: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 71%|███████   | 520/730 [46:03<19:00,  5.43s/it]
                                                 
{'loss': 2.4948, 'grad_norm': 1.2527204751968384, 'learning_rate': 5.090004441617116e-06, 'epoch': 7.12}

 71%|███████   | 520/730 [46:03<19:00,  5.43s/it]
                                                 
{'debug/num_tok_total': 2628.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2628.0, 'debug/num_lat_loss': 1776.0, 'epoch': 7.12}

 71%|███████   | 520/730 [46:04<19:00,  5.43s/it]
                                                 
{'train/ce_loss': 1.65625, 'train/diffusion_loss': 0.40028801560401917, 'epoch': 7.12}

 71%|███████   | 520/730 [46:04<19:00,  5.43s/it]
                                                 
{'train/learning_rate_real': 5.0454080182687314e-06, 'epoch': 7.12}

 71%|███████   | 520/730 [46:04<19:00,  5.43s/it]
                                                 
{'debug/num_tok_total': 2634.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2634.0, 'debug/num_lat_loss': 1780.0, 'epoch': 7.12}

 71%|███████   | 520/730 [46:05<19:00,  5.43s/it]
                                                 
{'train/ce_loss': 1.59375, 'train/diffusion_loss': 0.394565224647522, 'epoch': 7.12}

 71%|███████   | 520/730 [46:05<19:00,  5.43s/it]
                                                 
{'train/learning_rate_real': 5.0454080182687314e-06, 'epoch': 7.12}

 71%|███████   | 520/730 [46:05<19:00,  5.43s/it]
                                                 
{'debug/num_tok_total': 2399.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 2399.0, 'debug/num_lat_loss': 1769.0, 'epoch': 7.12}

 71%|███████   | 520/730 [46:06<19:00,  5.43s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.4412299394607544, 'epoch': 7.12}

 71%|███████   | 520/730 [46:06<19:00,  5.43s/it]
                                                 
{'train/learning_rate_real': 5.0454080182687314e-06, 'epoch': 7.12}

 71%|███████   | 520/730 [46:06<19:00,  5.43s/it]
                                                 
{'debug/num_tok_total': 2829.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2829.0, 'debug/num_lat_loss': 1774.0, 'epoch': 7.12}

 71%|███████   | 520/730 [46:08<19:00,  5.43s/it]
                                                 
{'train/ce_loss': 1.671875, 'train/diffusion_loss': 0.3732450604438782, 'epoch': 7.12}

 71%|███████   | 520/730 [46:08<19:00,  5.43s/it]
                                                 
{'train/learning_rate_real': 5.0454080182687314e-06, 'epoch': 7.12}

 71%|███████   | 520/730 [46:08<19:00,  5.43s/it]
 71%|███████▏  | 521/730 [46:08<18:49,  5.40s/it]
                                                 
{'debug/num_tok_total': 2624.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2624.0, 'debug/num_lat_loss': 1775.0, 'epoch': 7.14}

 71%|███████▏  | 521/730 [46:09<18:49,  5.40s/it]
                                                 
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.39827144145965576, 'epoch': 7.14}

 71%|███████▏  | 521/730 [46:09<18:49,  5.40s/it]
                                                 
{'train/learning_rate_real': 5.000958371375697e-06, 'epoch': 7.14}

 71%|███████▏  | 521/730 [46:09<18:49,  5.40s/it]
                                                 
{'debug/num_tok_total': 2847.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2847.0, 'debug/num_lat_loss': 1779.0, 'epoch': 7.14}

 71%|███████▏  | 521/730 [46:11<18:49,  5.40s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.35442906618118286, 'epoch': 7.14}

 71%|███████▏  | 521/730 [46:11<18:49,  5.40s/it]
                                                 
{'train/learning_rate_real': 5.000958371375697e-06, 'epoch': 7.14}

 71%|███████▏  | 521/730 [46:11<18:49,  5.40s/it]
                                                 
{'debug/num_tok_total': 2444.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2444.0, 'debug/num_lat_loss': 1789.0, 'epoch': 7.14}

 71%|███████▏  | 521/730 [46:12<18:49,  5.40s/it]
                                                 
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.44314831495285034, 'epoch': 7.14}

 71%|███████▏  | 521/730 [46:12<18:49,  5.40s/it]
                                                 
{'train/learning_rate_real': 5.000958371375697e-06, 'epoch': 7.14}

 71%|███████▏  | 521/730 [46:12<18:49,  5.40s/it]
                                                 
{'debug/num_tok_total': 2632.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2632.0, 'debug/num_lat_loss': 1780.0, 'epoch': 7.14}

 71%|███████▏  | 521/730 [46:13<18:49,  5.40s/it]
                                                 
{'train/ce_loss': 1.6875, 'train/diffusion_loss': 0.38872817158699036, 'epoch': 7.14}

 71%|███████▏  | 521/730 [46:13<18:49,  5.40s/it]
                                                 
{'train/learning_rate_real': 5.000958371375697e-06, 'epoch': 7.14}

 71%|███████▏  | 521/730 [46:13<18:49,  5.40s/it]
 72%|███████▏  | 522/730 [46:14<18:35,  5.36s/it]
                                                 
{'debug/num_tok_total': 2616.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2616.0, 'debug/num_lat_loss': 1763.0, 'epoch': 7.15}

 72%|███████▏  | 522/730 [46:15<18:35,  5.36s/it]
                                                 
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.3981302082538605, 'epoch': 7.15}

 72%|███████▏  | 522/730 [46:15<18:35,  5.36s/it]
                                                 
{'train/learning_rate_real': 4.956656376124951e-06, 'epoch': 7.15}

 72%|███████▏  | 522/730 [46:15<18:35,  5.36s/it]
                                                 
{'debug/num_tok_total': 2462.0, 'debug/num_tok_loss': 1608.0, 'debug/num_lat_total': 2462.0, 'debug/num_lat_loss': 1608.0, 'epoch': 7.15}

 72%|███████▏  | 522/730 [46:16<18:35,  5.36s/it]
                                                 
{'train/ce_loss': 1.65625, 'train/diffusion_loss': 0.36556708812713623, 'epoch': 7.15}

 72%|███████▏  | 522/730 [46:16<18:35,  5.36s/it]
                                                 
{'train/learning_rate_real': 4.956656376124951e-06, 'epoch': 7.15}

 72%|███████▏  | 522/730 [46:16<18:35,  5.36s/it]
                                                 
{'debug/num_tok_total': 2827.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 2827.0, 'debug/num_lat_loss': 1765.0, 'epoch': 7.15}

 72%|███████▏  | 522/730 [46:17<18:35,  5.36s/it]
                                                 
{'train/ce_loss': 1.6328125, 'train/diffusion_loss': 0.35889357328414917, 'epoch': 7.15}

 72%|███████▏  | 522/730 [46:17<18:35,  5.36s/it]
                                                 
{'train/learning_rate_real': 4.956656376124951e-06, 'epoch': 7.15}

 72%|███████▏  | 522/730 [46:17<18:35,  5.36s/it]
                                                 
{'debug/num_tok_total': 2648.0, 'debug/num_tok_loss': 1667.0, 'debug/num_lat_total': 2648.0, 'debug/num_lat_loss': 1667.0, 'epoch': 7.15}

 72%|███████▏  | 522/730 [46:18<18:35,  5.36s/it]
                                                 
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.33070722222328186, 'epoch': 7.15}

 72%|███████▏  | 522/730 [46:18<18:35,  5.36s/it]
                                                 
{'train/learning_rate_real': 4.956656376124951e-06, 'epoch': 7.15}

 72%|███████▏  | 522/730 [46:18<18:35,  5.36s/it]
 72%|███████▏  | 523/730 [46:19<18:30,  5.36s/it]
                                                 
{'debug/num_tok_total': 3306.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 3306.0, 'debug/num_lat_loss': 1791.0, 'epoch': 7.16}

 72%|███████▏  | 523/730 [46:20<18:30,  5.36s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.2841149568557739, 'epoch': 7.16}

 72%|███████▏  | 523/730 [46:20<18:30,  5.36s/it]
                                                 
{'train/learning_rate_real': 4.912502904796282e-06, 'epoch': 7.16}

 72%|███████▏  | 523/730 [46:20<18:30,  5.36s/it]
                                                 
{'debug/num_tok_total': 2836.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2836.0, 'debug/num_lat_loss': 1775.0, 'epoch': 7.16}

 72%|███████▏  | 523/730 [46:21<18:30,  5.36s/it]
                                                 
{'train/ce_loss': 1.8515625, 'train/diffusion_loss': 0.37643498182296753, 'epoch': 7.16}

 72%|███████▏  | 523/730 [46:21<18:30,  5.36s/it]
                                                 
{'train/learning_rate_real': 4.912502904796282e-06, 'epoch': 7.16}

 72%|███████▏  | 523/730 [46:21<18:30,  5.36s/it]
                                                 
{'debug/num_tok_total': 2912.0, 'debug/num_tok_loss': 1809.0, 'debug/num_lat_total': 2912.0, 'debug/num_lat_loss': 1809.0, 'epoch': 7.16}

 72%|███████▏  | 523/730 [46:23<18:30,  5.36s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.3642564117908478, 'epoch': 7.16}

 72%|███████▏  | 523/730 [46:23<18:30,  5.36s/it]
                                                 
{'train/learning_rate_real': 4.912502904796282e-06, 'epoch': 7.16}

 72%|███████▏  | 523/730 [46:23<18:30,  5.36s/it]
                                                 
{'debug/num_tok_total': 2840.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2840.0, 'debug/num_lat_loss': 1772.0, 'epoch': 7.16}

 72%|███████▏  | 523/730 [46:24<18:30,  5.36s/it]
                                                 
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.355696439743042, 'epoch': 7.16}

 72%|███████▏  | 523/730 [46:24<18:30,  5.36s/it]
                                                 
{'train/learning_rate_real': 4.912502904796282e-06, 'epoch': 7.16}

 72%|███████▏  | 523/730 [46:24<18:30,  5.36s/it]
 72%|███████▏  | 524/730 [46:25<18:43,  5.46s/it]
                                                 
{'debug/num_tok_total': 2825.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2825.0, 'debug/num_lat_loss': 1770.0, 'epoch': 7.18}

 72%|███████▏  | 524/730 [46:26<18:43,  5.46s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.3547402024269104, 'epoch': 7.18}

 72%|███████▏  | 524/730 [46:26<18:43,  5.46s/it]
                                                 
{'train/learning_rate_real': 4.868498826745108e-06, 'epoch': 7.18}

 72%|███████▏  | 524/730 [46:26<18:43,  5.46s/it]
                                                 
{'debug/num_tok_total': 2201.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2201.0, 'debug/num_lat_loss': 1776.0, 'epoch': 7.18}

 72%|███████▏  | 524/730 [46:27<18:43,  5.46s/it]
                                                 
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.5162613987922668, 'epoch': 7.18}

 72%|███████▏  | 524/730 [46:27<18:43,  5.46s/it]
                                                 
{'train/learning_rate_real': 4.868498826745108e-06, 'epoch': 7.18}

 72%|███████▏  | 524/730 [46:27<18:43,  5.46s/it]
                                                 
{'debug/num_tok_total': 2437.0, 'debug/num_tok_loss': 1592.0, 'debug/num_lat_total': 2437.0, 'debug/num_lat_loss': 1592.0, 'epoch': 7.18}

 72%|███████▏  | 524/730 [46:28<18:43,  5.46s/it]
                                                 
{'train/ce_loss': 1.6171875, 'train/diffusion_loss': 0.3750528395175934, 'epoch': 7.18}

 72%|███████▏  | 524/730 [46:28<18:43,  5.46s/it]
                                                 
{'train/learning_rate_real': 4.868498826745108e-06, 'epoch': 7.18}

 72%|███████▏  | 524/730 [46:28<18:43,  5.46s/it]
                                                 
{'debug/num_tok_total': 3070.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 3070.0, 'debug/num_lat_loss': 1775.0, 'epoch': 7.18}

 72%|███████▏  | 524/730 [46:29<18:43,  5.46s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.32973960041999817, 'epoch': 7.18}

 72%|███████▏  | 524/730 [46:29<18:43,  5.46s/it]
                                                 
{'train/learning_rate_real': 4.868498826745108e-06, 'epoch': 7.18}

 72%|███████▏  | 524/730 [46:29<18:43,  5.46s/it]
 72%|███████▏  | 525/730 [46:30<18:34,  5.44s/it]
                                                 
{'debug/num_tok_total': 2277.0, 'debug/num_tok_loss': 1634.0, 'debug/num_lat_total': 2277.0, 'debug/num_lat_loss': 1634.0, 'epoch': 7.19}

 72%|███████▏  | 525/730 [46:31<18:34,  5.44s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.446077436208725, 'epoch': 7.19}

 72%|███████▏  | 525/730 [46:31<18:34,  5.44s/it]
                                                 
{'train/learning_rate_real': 4.824645008385399e-06, 'epoch': 7.19}

 72%|███████▏  | 525/730 [46:31<18:34,  5.44s/it]
                                                 
{'debug/num_tok_total': 2638.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2638.0, 'debug/num_lat_loss': 1776.0, 'epoch': 7.19}

 72%|███████▏  | 525/730 [46:32<18:34,  5.44s/it]
                                                 
{'train/ce_loss': 1.9609375, 'train/diffusion_loss': 0.3760990798473358, 'epoch': 7.19}

 72%|███████▏  | 525/730 [46:32<18:34,  5.44s/it]
                                                 
{'train/learning_rate_real': 4.824645008385399e-06, 'epoch': 7.19}

 72%|███████▏  | 525/730 [46:32<18:34,  5.44s/it]
                                                 
{'debug/num_tok_total': 2400.0, 'debug/num_tok_loss': 1761.0, 'debug/num_lat_total': 2400.0, 'debug/num_lat_loss': 1761.0, 'epoch': 7.19}

 72%|███████▏  | 525/730 [46:33<18:34,  5.44s/it]
                                                 
{'train/ce_loss': 1.9296875, 'train/diffusion_loss': 0.4374392330646515, 'epoch': 7.19}

 72%|███████▏  | 525/730 [46:33<18:34,  5.44s/it]
                                                 
{'train/learning_rate_real': 4.824645008385399e-06, 'epoch': 7.19}

 72%|███████▏  | 525/730 [46:33<18:34,  5.44s/it]
                                                 
{'debug/num_tok_total': 2590.0, 'debug/num_tok_loss': 1755.0, 'debug/num_lat_total': 2590.0, 'debug/num_lat_loss': 1755.0, 'epoch': 7.19}

 72%|███████▏  | 525/730 [46:35<18:34,  5.44s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.38194751739501953, 'epoch': 7.19}

 72%|███████▏  | 525/730 [46:35<18:34,  5.44s/it]
                                                 
{'train/learning_rate_real': 4.824645008385399e-06, 'epoch': 7.19}

 72%|███████▏  | 525/730 [46:35<18:34,  5.44s/it]
 72%|███████▏  | 526/730 [46:35<18:07,  5.33s/it]
                                                 
{'debug/num_tok_total': 2629.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2629.0, 'debug/num_lat_loss': 1780.0, 'epoch': 7.21}

 72%|███████▏  | 526/730 [46:36<18:07,  5.33s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.3838163912296295, 'epoch': 7.21}

 72%|███████▏  | 526/730 [46:36<18:07,  5.33s/it]
                                                 
{'train/learning_rate_real': 4.780942313172602e-06, 'epoch': 7.21}

 72%|███████▏  | 526/730 [46:36<18:07,  5.33s/it]
                                                 
{'debug/num_tok_total': 2439.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2439.0, 'debug/num_lat_loss': 1784.0, 'epoch': 7.21}

 72%|███████▏  | 526/730 [46:37<18:07,  5.33s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.4489106833934784, 'epoch': 7.21}

 72%|███████▏  | 526/730 [46:37<18:07,  5.33s/it]
                                                 
{'train/learning_rate_real': 4.780942313172602e-06, 'epoch': 7.21}

 72%|███████▏  | 526/730 [46:37<18:07,  5.33s/it]
                                                 
{'debug/num_tok_total': 2700.0, 'debug/num_tok_loss': 1616.0, 'debug/num_lat_total': 2700.0, 'debug/num_lat_loss': 1616.0, 'epoch': 7.21}

 72%|███████▏  | 526/730 [46:39<18:07,  5.33s/it]
                                                 
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.3480779826641083, 'epoch': 7.21}

 72%|███████▏  | 526/730 [46:39<18:07,  5.33s/it]
                                                 
{'train/learning_rate_real': 4.780942313172602e-06, 'epoch': 7.21}

 72%|███████▏  | 526/730 [46:39<18:07,  5.33s/it]
                                                 
{'debug/num_tok_total': 2372.0, 'debug/num_tok_loss': 1749.0, 'debug/num_lat_total': 2372.0, 'debug/num_lat_loss': 1749.0, 'epoch': 7.21}

 72%|███████▏  | 526/730 [46:40<18:07,  5.33s/it]
                                                 
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.42179641127586365, 'epoch': 7.21}

 72%|███████▏  | 526/730 [46:40<18:07,  5.33s/it]
                                                 
{'train/learning_rate_real': 4.780942313172602e-06, 'epoch': 7.21}

 72%|███████▏  | 526/730 [46:40<18:07,  5.33s/it]
 72%|███████▏  | 527/730 [46:40<17:58,  5.31s/it]
                                                 
{'debug/num_tok_total': 2909.0, 'debug/num_tok_loss': 1812.0, 'debug/num_lat_total': 2909.0, 'debug/num_lat_loss': 1812.0, 'epoch': 7.22}

 72%|███████▏  | 527/730 [46:42<17:58,  5.31s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.3573102355003357, 'epoch': 7.22}

 72%|███████▏  | 527/730 [46:42<17:58,  5.31s/it]
                                                 
{'train/learning_rate_real': 4.73739160158663e-06, 'epoch': 7.22}

 72%|███████▏  | 527/730 [46:42<17:58,  5.31s/it]
                                                 
{'debug/num_tok_total': 2655.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2655.0, 'debug/num_lat_loss': 1788.0, 'epoch': 7.22}

 72%|███████▏  | 527/730 [46:43<17:58,  5.31s/it]
                                                 
{'train/ce_loss': 1.90625, 'train/diffusion_loss': 0.4007185399532318, 'epoch': 7.22}

 72%|███████▏  | 527/730 [46:43<17:58,  5.31s/it]
                                                 
{'train/learning_rate_real': 4.73739160158663e-06, 'epoch': 7.22}

 72%|███████▏  | 527/730 [46:43<17:58,  5.31s/it]
                                                 
{'debug/num_tok_total': 2691.0, 'debug/num_tok_loss': 1704.0, 'debug/num_lat_total': 2691.0, 'debug/num_lat_loss': 1704.0, 'epoch': 7.22}

 72%|███████▏  | 527/730 [46:44<17:58,  5.31s/it]
                                                 
{'train/ce_loss': 1.609375, 'train/diffusion_loss': 0.3736780881881714, 'epoch': 7.22}

 72%|███████▏  | 527/730 [46:44<17:58,  5.31s/it]
                                                 
{'train/learning_rate_real': 4.73739160158663e-06, 'epoch': 7.22}

 72%|███████▏  | 527/730 [46:44<17:58,  5.31s/it]
                                                 
{'debug/num_tok_total': 2145.0, 'debug/num_tok_loss': 1716.0, 'debug/num_lat_total': 2145.0, 'debug/num_lat_loss': 1716.0, 'epoch': 7.22}

 72%|███████▏  | 527/730 [46:45<17:58,  5.31s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.4473676383495331, 'epoch': 7.22}

 72%|███████▏  | 527/730 [46:45<17:58,  5.31s/it]
                                                 
{'train/learning_rate_real': 4.73739160158663e-06, 'epoch': 7.22}

 72%|███████▏  | 527/730 [46:45<17:58,  5.31s/it]
 72%|███████▏  | 528/730 [46:46<18:01,  5.35s/it]
                                                 
{'debug/num_tok_total': 2880.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2880.0, 'debug/num_lat_loss': 1789.0, 'epoch': 7.23}

 72%|███████▏  | 528/730 [46:47<18:01,  5.35s/it]
                                                 
{'train/ce_loss': 1.640625, 'train/diffusion_loss': 0.37767571210861206, 'epoch': 7.23}

 72%|███████▏  | 528/730 [46:47<18:01,  5.35s/it]
                                                 
{'train/learning_rate_real': 4.693993731114929e-06, 'epoch': 7.23}

 72%|███████▏  | 528/730 [46:47<18:01,  5.35s/it]
                                                 
{'debug/num_tok_total': 2698.0, 'debug/num_tok_loss': 1812.0, 'debug/num_lat_total': 2698.0, 'debug/num_lat_loss': 1812.0, 'epoch': 7.23}

 72%|███████▏  | 528/730 [46:48<18:01,  5.35s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.38745903968811035, 'epoch': 7.23}

 72%|███████▏  | 528/730 [46:48<18:01,  5.35s/it]
                                                 
{'train/learning_rate_real': 4.693993731114929e-06, 'epoch': 7.23}

 72%|███████▏  | 528/730 [46:48<18:01,  5.35s/it]
                                                 
{'debug/num_tok_total': 3108.0, 'debug/num_tok_loss': 1804.0, 'debug/num_lat_total': 3108.0, 'debug/num_lat_loss': 1804.0, 'epoch': 7.23}

 72%|███████▏  | 528/730 [46:49<18:01,  5.35s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.3078138530254364, 'epoch': 7.23}

 72%|███████▏  | 528/730 [46:49<18:01,  5.35s/it]
                                                 
{'train/learning_rate_real': 4.693993731114929e-06, 'epoch': 7.23}

 72%|███████▏  | 528/730 [46:49<18:01,  5.35s/it]
                                                 
{'debug/num_tok_total': 2636.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2636.0, 'debug/num_lat_loss': 1783.0, 'epoch': 7.23}

 72%|███████▏  | 528/730 [46:51<18:01,  5.35s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.4015379846096039, 'epoch': 7.23}

 72%|███████▏  | 528/730 [46:51<18:01,  5.35s/it]
                                                 
{'train/learning_rate_real': 4.693993731114929e-06, 'epoch': 7.23}

 72%|███████▏  | 528/730 [46:51<18:01,  5.35s/it]
 72%|███████▏  | 529/730 [46:51<18:01,  5.38s/it]
                                                 
{'debug/num_tok_total': 2200.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 2200.0, 'debug/num_lat_loss': 1769.0, 'epoch': 7.25}

 72%|███████▏  | 529/730 [46:52<18:01,  5.38s/it]
                                                 
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.4916262924671173, 'epoch': 7.25}

 72%|███████▏  | 529/730 [46:52<18:01,  5.38s/it]
                                                 
{'train/learning_rate_real': 4.650749556235602e-06, 'epoch': 7.25}

 72%|███████▏  | 529/730 [46:52<18:01,  5.38s/it]
                                                 
{'debug/num_tok_total': 2905.0, 'debug/num_tok_loss': 1813.0, 'debug/num_lat_total': 2905.0, 'debug/num_lat_loss': 1813.0, 'epoch': 7.25}

 72%|███████▏  | 529/730 [46:54<18:01,  5.38s/it]
                                                 
{'train/ce_loss': 1.9296875, 'train/diffusion_loss': 0.3476724922657013, 'epoch': 7.25}

 72%|███████▏  | 529/730 [46:54<18:01,  5.38s/it]
                                                 
{'train/learning_rate_real': 4.650749556235602e-06, 'epoch': 7.25}

 72%|███████▏  | 529/730 [46:54<18:01,  5.38s/it]
                                                 
{'debug/num_tok_total': 2451.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 2451.0, 'debug/num_lat_loss': 1797.0, 'epoch': 7.25}

 72%|███████▏  | 529/730 [46:55<18:01,  5.38s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.4196511209011078, 'epoch': 7.25}

 72%|███████▏  | 529/730 [46:55<18:01,  5.38s/it]
                                                 
{'train/learning_rate_real': 4.650749556235602e-06, 'epoch': 7.25}

 72%|███████▏  | 529/730 [46:55<18:01,  5.38s/it]
                                                 
{'debug/num_tok_total': 3302.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 3302.0, 'debug/num_lat_loss': 1784.0, 'epoch': 7.25}

 72%|███████▏  | 529/730 [46:56<18:01,  5.38s/it]
                                                 
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.2985173463821411, 'epoch': 7.25}

 72%|███████▏  | 529/730 [46:56<18:01,  5.38s/it]
                                                 
{'train/learning_rate_real': 4.650749556235602e-06, 'epoch': 7.25}

 72%|███████▏  | 529/730 [46:56<18:01,  5.38s/it]03/16/2026 07:34:29 - INFO - __main__ - LoRA debug step 530: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 73%|███████▎  | 530/730 [46:57<17:54,  5.37s/it]
                                                 
{'loss': 2.4434, 'grad_norm': 1.1598719358444214, 'learning_rate': 4.650749556235602e-06, 'epoch': 7.26}

 73%|███████▎  | 530/730 [46:57<17:54,  5.37s/it]
                                                 
{'debug/num_tok_total': 2894.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 2894.0, 'debug/num_lat_loss': 1797.0, 'epoch': 7.26}

 73%|███████▎  | 530/730 [46:58<17:54,  5.37s/it]
                                                 
{'train/ce_loss': 1.625, 'train/diffusion_loss': 0.3587283790111542, 'epoch': 7.26}

 73%|███████▎  | 530/730 [46:58<17:54,  5.37s/it]
                                                 
{'train/learning_rate_real': 4.607659928400579e-06, 'epoch': 7.26}

 73%|███████▎  | 530/730 [46:58<17:54,  5.37s/it]
                                                 
{'debug/num_tok_total': 1769.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 1769.0, 'debug/num_lat_loss': 1769.0, 'epoch': 7.26}

 73%|███████▎  | 530/730 [46:59<17:54,  5.37s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.5675220489501953, 'epoch': 7.26}

 73%|███████▎  | 530/730 [46:59<17:54,  5.37s/it]
                                                 
{'train/learning_rate_real': 4.607659928400579e-06, 'epoch': 7.26}

 73%|███████▎  | 530/730 [46:59<17:54,  5.37s/it]
                                                 
{'debug/num_tok_total': 2439.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2439.0, 'debug/num_lat_loss': 1796.0, 'epoch': 7.26}

 73%|███████▎  | 530/730 [47:00<17:54,  5.37s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.4797578752040863, 'epoch': 7.26}

 73%|███████▎  | 530/730 [47:00<17:54,  5.37s/it]
                                                 
{'train/learning_rate_real': 4.607659928400579e-06, 'epoch': 7.26}

 73%|███████▎  | 530/730 [47:00<17:54,  5.37s/it]
                                                 
{'debug/num_tok_total': 2901.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2901.0, 'debug/num_lat_loss': 1802.0, 'epoch': 7.26}

 73%|███████▎  | 530/730 [47:01<17:54,  5.37s/it]
                                                 
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.3402682840824127, 'epoch': 7.26}

 73%|███████▎  | 530/730 [47:01<17:54,  5.37s/it]
                                                 
{'train/learning_rate_real': 4.607659928400579e-06, 'epoch': 7.26}

 73%|███████▎  | 530/730 [47:01<17:54,  5.37s/it]
 73%|███████▎  | 531/730 [47:02<17:36,  5.31s/it]
                                                 
{'debug/num_tok_total': 2684.0, 'debug/num_tok_loss': 1815.0, 'debug/num_lat_total': 2684.0, 'debug/num_lat_loss': 1815.0, 'epoch': 7.27}

 73%|███████▎  | 531/730 [47:03<17:36,  5.31s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.39923813939094543, 'epoch': 7.27}

 73%|███████▎  | 531/730 [47:03<17:36,  5.31s/it]
                                                 
{'train/learning_rate_real': 4.564725696018849e-06, 'epoch': 7.27}

 73%|███████▎  | 531/730 [47:03<17:36,  5.31s/it]
                                                 
{'debug/num_tok_total': 2587.0, 'debug/num_tok_loss': 1760.0, 'debug/num_lat_total': 2587.0, 'debug/num_lat_loss': 1760.0, 'epoch': 7.27}

 73%|███████▎  | 531/730 [47:04<17:36,  5.31s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.4195687174797058, 'epoch': 7.27}

 73%|███████▎  | 531/730 [47:04<17:36,  5.31s/it]
                                                 
{'train/learning_rate_real': 4.564725696018849e-06, 'epoch': 7.27}

 73%|███████▎  | 531/730 [47:04<17:36,  5.31s/it]
                                                 
{'debug/num_tok_total': 2638.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2638.0, 'debug/num_lat_loss': 1787.0, 'epoch': 7.27}

 73%|███████▎  | 531/730 [47:05<17:36,  5.31s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.4344101548194885, 'epoch': 7.27}

 73%|███████▎  | 531/730 [47:05<17:36,  5.31s/it]
                                                 
{'train/learning_rate_real': 4.564725696018849e-06, 'epoch': 7.27}

 73%|███████▎  | 531/730 [47:05<17:36,  5.31s/it]
                                                 
{'debug/num_tok_total': 1958.0, 'debug/num_tok_loss': 1753.0, 'debug/num_lat_total': 1958.0, 'debug/num_lat_loss': 1753.0, 'epoch': 7.27}

 73%|███████▎  | 531/730 [47:06<17:36,  5.31s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.5388801693916321, 'epoch': 7.27}

 73%|███████▎  | 531/730 [47:06<17:36,  5.31s/it]
                                                 
{'train/learning_rate_real': 4.564725696018849e-06, 'epoch': 7.27}

 73%|███████▎  | 531/730 [47:06<17:36,  5.31s/it]
 73%|███████▎  | 532/730 [47:07<17:18,  5.25s/it]
                                                 
{'debug/num_tok_total': 2598.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2598.0, 'debug/num_lat_loss': 1763.0, 'epoch': 7.29}

 73%|███████▎  | 532/730 [47:08<17:18,  5.25s/it]
                                                 
{'train/ce_loss': 1.6171875, 'train/diffusion_loss': 0.38455724716186523, 'epoch': 7.29}

 73%|███████▎  | 532/730 [47:08<17:18,  5.25s/it]
                                                 
{'train/learning_rate_real': 4.5219477044397525e-06, 'epoch': 7.29}

 73%|███████▎  | 532/730 [47:08<17:18,  5.25s/it]
                                                 
{'debug/num_tok_total': 2803.0, 'debug/num_tok_loss': 1753.0, 'debug/num_lat_total': 2803.0, 'debug/num_lat_loss': 1753.0, 'epoch': 7.29}

 73%|███████▎  | 532/730 [47:09<17:18,  5.25s/it]
                                                 
{'train/ce_loss': 1.6484375, 'train/diffusion_loss': 0.38055655360221863, 'epoch': 7.29}

 73%|███████▎  | 532/730 [47:09<17:18,  5.25s/it]
                                                 
{'train/learning_rate_real': 4.5219477044397525e-06, 'epoch': 7.29}

 73%|███████▎  | 532/730 [47:09<17:18,  5.25s/it]
                                                 
{'debug/num_tok_total': 2622.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2622.0, 'debug/num_lat_loss': 1776.0, 'epoch': 7.29}

 73%|███████▎  | 532/730 [47:10<17:18,  5.25s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.4435129463672638, 'epoch': 7.29}

 73%|███████▎  | 532/730 [47:10<17:18,  5.25s/it]
                                                 
{'train/learning_rate_real': 4.5219477044397525e-06, 'epoch': 7.29}

 73%|███████▎  | 532/730 [47:10<17:18,  5.25s/it]
                                                 
{'debug/num_tok_total': 3063.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 3063.0, 'debug/num_lat_loss': 1772.0, 'epoch': 7.29}

 73%|███████▎  | 532/730 [47:12<17:18,  5.25s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.3226284086704254, 'epoch': 7.29}

 73%|███████▎  | 532/730 [47:12<17:18,  5.25s/it]
                                                 
{'train/learning_rate_real': 4.5219477044397525e-06, 'epoch': 7.29}

 73%|███████▎  | 532/730 [47:12<17:18,  5.25s/it]
 73%|███████▎  | 533/730 [47:12<17:17,  5.27s/it]
                                                 
{'debug/num_tok_total': 2460.0, 'debug/num_tok_loss': 1811.0, 'debug/num_lat_total': 2460.0, 'debug/num_lat_loss': 1811.0, 'epoch': 7.3}

 73%|███████▎  | 533/730 [47:13<17:17,  5.27s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.4392894506454468, 'epoch': 7.3}

 73%|███████▎  | 533/730 [47:13<17:17,  5.27s/it]
                                                 
{'train/learning_rate_real': 4.479326795936338e-06, 'epoch': 7.3}

 73%|███████▎  | 533/730 [47:13<17:17,  5.27s/it]
                                                 
{'debug/num_tok_total': 2849.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2849.0, 'debug/num_lat_loss': 1770.0, 'epoch': 7.3}

 73%|███████▎  | 533/730 [47:15<17:17,  5.27s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.34575310349464417, 'epoch': 7.3}

 73%|███████▎  | 533/730 [47:15<17:17,  5.27s/it]
                                                 
{'train/learning_rate_real': 4.479326795936338e-06, 'epoch': 7.3}

 73%|███████▎  | 533/730 [47:15<17:17,  5.27s/it]
                                                 
{'debug/num_tok_total': 3087.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 3087.0, 'debug/num_lat_loss': 1787.0, 'epoch': 7.3}

 73%|███████▎  | 533/730 [47:16<17:17,  5.27s/it]
                                                 
{'train/ce_loss': 1.90625, 'train/diffusion_loss': 0.2958372235298157, 'epoch': 7.3}

 73%|███████▎  | 533/730 [47:16<17:17,  5.27s/it]
                                                 
{'train/learning_rate_real': 4.479326795936338e-06, 'epoch': 7.3}

 73%|███████▎  | 533/730 [47:16<17:17,  5.27s/it]
                                                 
{'debug/num_tok_total': 3015.0, 'debug/num_tok_loss': 1754.0, 'debug/num_lat_total': 3015.0, 'debug/num_lat_loss': 1754.0, 'epoch': 7.3}

 73%|███████▎  | 533/730 [47:17<17:17,  5.27s/it]
                                                 
{'train/ce_loss': 1.671875, 'train/diffusion_loss': 0.29717665910720825, 'epoch': 7.3}

 73%|███████▎  | 533/730 [47:17<17:17,  5.27s/it]
                                                 
{'train/learning_rate_real': 4.479326795936338e-06, 'epoch': 7.3}

 73%|███████▎  | 533/730 [47:17<17:17,  5.27s/it]
 73%|███████▎  | 534/730 [47:18<17:33,  5.38s/it]
                                                 
{'debug/num_tok_total': 2233.0, 'debug/num_tok_loss': 1806.0, 'debug/num_lat_total': 2233.0, 'debug/num_lat_loss': 1806.0, 'epoch': 7.32}

 73%|███████▎  | 534/730 [47:19<17:33,  5.38s/it]
                                                 
{'train/ce_loss': 1.921875, 'train/diffusion_loss': 0.48870232701301575, 'epoch': 7.32}

 73%|███████▎  | 534/730 [47:19<17:33,  5.38s/it]
                                                 
{'train/learning_rate_real': 4.436863809688806e-06, 'epoch': 7.32}

 73%|███████▎  | 534/730 [47:19<17:33,  5.38s/it]
                                                 
{'debug/num_tok_total': 3054.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 3054.0, 'debug/num_lat_loss': 1772.0, 'epoch': 7.32}

 73%|███████▎  | 534/730 [47:20<17:33,  5.38s/it]
                                                 
{'train/ce_loss': 1.6875, 'train/diffusion_loss': 0.2935459017753601, 'epoch': 7.32}

 73%|███████▎  | 534/730 [47:20<17:33,  5.38s/it]
                                                 
{'train/learning_rate_real': 4.436863809688806e-06, 'epoch': 7.32}

 73%|███████▎  | 534/730 [47:20<17:33,  5.38s/it]
                                                 
{'debug/num_tok_total': 2450.0, 'debug/num_tok_loss': 1810.0, 'debug/num_lat_total': 2450.0, 'debug/num_lat_loss': 1810.0, 'epoch': 7.32}

 73%|███████▎  | 534/730 [47:21<17:33,  5.38s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.4642660319805145, 'epoch': 7.32}

 73%|███████▎  | 534/730 [47:21<17:33,  5.38s/it]
                                                 
{'train/learning_rate_real': 4.436863809688806e-06, 'epoch': 7.32}

 73%|███████▎  | 534/730 [47:21<17:33,  5.38s/it]
                                                 
{'debug/num_tok_total': 3059.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 3059.0, 'debug/num_lat_loss': 1777.0, 'epoch': 7.32}

 73%|███████▎  | 534/730 [47:23<17:33,  5.38s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.3130846619606018, 'epoch': 7.32}

 73%|███████▎  | 534/730 [47:23<17:33,  5.38s/it]
                                                 
{'train/learning_rate_real': 4.436863809688806e-06, 'epoch': 7.32}

 73%|███████▎  | 534/730 [47:23<17:33,  5.38s/it]
 73%|███████▎  | 535/730 [47:23<17:28,  5.38s/it]
                                                 
{'debug/num_tok_total': 2237.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 2237.0, 'debug/num_lat_loss': 1801.0, 'epoch': 7.33}

 73%|███████▎  | 535/730 [47:24<17:28,  5.38s/it]
                                                 
{'train/ce_loss': 1.6875, 'train/diffusion_loss': 0.46884414553642273, 'epoch': 7.33}

 73%|███████▎  | 535/730 [47:24<17:28,  5.38s/it]
                                                 
{'train/learning_rate_real': 4.394559581767937e-06, 'epoch': 7.33}

 73%|███████▎  | 535/730 [47:24<17:28,  5.38s/it]
                                                 
{'debug/num_tok_total': 2246.0, 'debug/num_tok_loss': 1808.0, 'debug/num_lat_total': 2246.0, 'debug/num_lat_loss': 1808.0, 'epoch': 7.33}

 73%|███████▎  | 535/730 [47:25<17:28,  5.38s/it]
                                                 
{'train/ce_loss': 1.6015625, 'train/diffusion_loss': 0.46303117275238037, 'epoch': 7.33}

 73%|███████▎  | 535/730 [47:25<17:28,  5.38s/it]
                                                 
{'train/learning_rate_real': 4.394559581767937e-06, 'epoch': 7.33}

 73%|███████▎  | 535/730 [47:25<17:28,  5.38s/it]
                                                 
{'debug/num_tok_total': 2487.0, 'debug/num_tok_loss': 1821.0, 'debug/num_lat_total': 2487.0, 'debug/num_lat_loss': 1821.0, 'epoch': 7.33}

 73%|███████▎  | 535/730 [47:26<17:28,  5.38s/it]
                                                 
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.44739916920661926, 'epoch': 7.33}

 73%|███████▎  | 535/730 [47:26<17:28,  5.38s/it]
                                                 
{'train/learning_rate_real': 4.394559581767937e-06, 'epoch': 7.33}

 73%|███████▎  | 535/730 [47:26<17:28,  5.38s/it]
                                                 
{'debug/num_tok_total': 2669.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 2669.0, 'debug/num_lat_loss': 1801.0, 'epoch': 7.33}

 73%|███████▎  | 535/730 [47:28<17:28,  5.38s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.3915025293827057, 'epoch': 7.33}

 73%|███████▎  | 535/730 [47:28<17:28,  5.38s/it]
                                                 
{'train/learning_rate_real': 4.394559581767937e-06, 'epoch': 7.33}

 73%|███████▎  | 535/730 [47:28<17:28,  5.38s/it]
 73%|███████▎  | 536/730 [47:28<17:00,  5.26s/it]
                                                 
{'debug/num_tok_total': 2652.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2652.0, 'debug/num_lat_loss': 1796.0, 'epoch': 7.34}

 73%|███████▎  | 536/730 [47:29<17:00,  5.26s/it]
                                                 
{'train/ce_loss': 1.6015625, 'train/diffusion_loss': 0.41723179817199707, 'epoch': 7.34}

 73%|███████▎  | 536/730 [47:29<17:00,  5.26s/it]
                                                 
{'train/learning_rate_real': 4.352414945118663e-06, 'epoch': 7.34}

 73%|███████▎  | 536/730 [47:29<17:00,  5.26s/it]
                                                 
{'debug/num_tok_total': 2425.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2425.0, 'debug/num_lat_loss': 1782.0, 'epoch': 7.34}

 73%|███████▎  | 536/730 [47:30<17:00,  5.26s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.4095483124256134, 'epoch': 7.34}

 73%|███████▎  | 536/730 [47:30<17:00,  5.26s/it]
                                                 
{'train/learning_rate_real': 4.352414945118663e-06, 'epoch': 7.34}

 73%|███████▎  | 536/730 [47:30<17:00,  5.26s/it]
                                                 
{'debug/num_tok_total': 2851.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2851.0, 'debug/num_lat_loss': 1776.0, 'epoch': 7.34}

 73%|███████▎  | 536/730 [47:32<17:00,  5.26s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.3790554702281952, 'epoch': 7.34}

 73%|███████▎  | 536/730 [47:32<17:00,  5.26s/it]
                                                 
{'train/learning_rate_real': 4.352414945118663e-06, 'epoch': 7.34}

 73%|███████▎  | 536/730 [47:32<17:00,  5.26s/it]
                                                 
{'debug/num_tok_total': 2923.0, 'debug/num_tok_loss': 1706.0, 'debug/num_lat_total': 2923.0, 'debug/num_lat_loss': 1706.0, 'epoch': 7.34}

 73%|███████▎  | 536/730 [47:33<17:00,  5.26s/it]
                                                 
{'train/ce_loss': 1.640625, 'train/diffusion_loss': 0.312753289937973, 'epoch': 7.34}

 73%|███████▎  | 536/730 [47:33<17:00,  5.26s/it]
                                                 
{'train/learning_rate_real': 4.352414945118663e-06, 'epoch': 7.34}

 73%|███████▎  | 536/730 [47:33<17:00,  5.26s/it]
 74%|███████▎  | 537/730 [47:33<17:00,  5.29s/it]
                                                 
{'debug/num_tok_total': 2643.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2643.0, 'debug/num_lat_loss': 1781.0, 'epoch': 7.36}

 74%|███████▎  | 537/730 [47:35<17:00,  5.29s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.4084358215332031, 'epoch': 7.36}

 74%|███████▎  | 537/730 [47:35<17:00,  5.29s/it]
                                                 
{'train/learning_rate_real': 4.31043072954366e-06, 'epoch': 7.36}

 74%|███████▎  | 537/730 [47:35<17:00,  5.29s/it]
                                                 
{'debug/num_tok_total': 2468.0, 'debug/num_tok_loss': 1800.0, 'debug/num_lat_total': 2468.0, 'debug/num_lat_loss': 1800.0, 'epoch': 7.36}

 74%|███████▎  | 537/730 [47:36<17:00,  5.29s/it]
                                                 
{'train/ce_loss': 1.6875, 'train/diffusion_loss': 0.4373239576816559, 'epoch': 7.36}

 74%|███████▎  | 537/730 [47:36<17:00,  5.29s/it]
                                                 
{'train/learning_rate_real': 4.31043072954366e-06, 'epoch': 7.36}

 74%|███████▎  | 537/730 [47:36<17:00,  5.29s/it]
                                                 
{'debug/num_tok_total': 2215.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2215.0, 'debug/num_lat_loss': 1786.0, 'epoch': 7.36}

 74%|███████▎  | 537/730 [47:37<17:00,  5.29s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.4702085852622986, 'epoch': 7.36}

 74%|███████▎  | 537/730 [47:37<17:00,  5.29s/it]
                                                 
{'train/learning_rate_real': 4.31043072954366e-06, 'epoch': 7.36}

 74%|███████▎  | 537/730 [47:37<17:00,  5.29s/it]
                                                 
{'debug/num_tok_total': 2663.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2663.0, 'debug/num_lat_loss': 1795.0, 'epoch': 7.36}

 74%|███████▎  | 537/730 [47:38<17:00,  5.29s/it]
                                                 
{'train/ce_loss': 1.6328125, 'train/diffusion_loss': 0.40620049834251404, 'epoch': 7.36}

 74%|███████▎  | 537/730 [47:38<17:00,  5.29s/it]
                                                 
{'train/learning_rate_real': 4.31043072954366e-06, 'epoch': 7.36}

 74%|███████▎  | 537/730 [47:38<17:00,  5.29s/it]
 74%|███████▎  | 538/730 [47:39<16:45,  5.23s/it]
                                                 
{'debug/num_tok_total': 3253.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 3253.0, 'debug/num_lat_loss': 1763.0, 'epoch': 7.37}

 74%|███████▎  | 538/730 [47:40<16:45,  5.23s/it]
                                                 
{'train/ce_loss': 2.03125, 'train/diffusion_loss': 0.2733267843723297, 'epoch': 7.37}

 74%|███████▎  | 538/730 [47:40<16:45,  5.23s/it]
                                                 
{'train/learning_rate_real': 4.268607761687019e-06, 'epoch': 7.37}

 74%|███████▎  | 538/730 [47:40<16:45,  5.23s/it]
                                                 
{'debug/num_tok_total': 2682.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2682.0, 'debug/num_lat_loss': 1788.0, 'epoch': 7.37}

 74%|███████▎  | 538/730 [47:41<16:45,  5.23s/it]
                                                 
{'train/ce_loss': 1.6171875, 'train/diffusion_loss': 0.3908580243587494, 'epoch': 7.37}

 74%|███████▎  | 538/730 [47:41<16:45,  5.23s/it]
                                                 
{'train/learning_rate_real': 4.268607761687019e-06, 'epoch': 7.37}

 74%|███████▎  | 538/730 [47:41<16:45,  5.23s/it]
                                                 
{'debug/num_tok_total': 2617.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2617.0, 'debug/num_lat_loss': 1763.0, 'epoch': 7.37}

 74%|███████▎  | 538/730 [47:43<16:45,  5.23s/it]
                                                 
{'train/ce_loss': 1.8984375, 'train/diffusion_loss': 0.4306962490081787, 'epoch': 7.37}

 74%|███████▎  | 538/730 [47:43<16:45,  5.23s/it]
                                                 
{'train/learning_rate_real': 4.268607761687019e-06, 'epoch': 7.37}

 74%|███████▎  | 538/730 [47:43<16:45,  5.23s/it]
                                                 
{'debug/num_tok_total': 2635.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2635.0, 'debug/num_lat_loss': 1777.0, 'epoch': 7.37}

 74%|███████▎  | 538/730 [47:44<16:45,  5.23s/it]
                                                 
{'train/ce_loss': 1.8984375, 'train/diffusion_loss': 0.37691691517829895, 'epoch': 7.37}

 74%|███████▎  | 538/730 [47:44<16:45,  5.23s/it]
                                                 
{'train/learning_rate_real': 4.268607761687019e-06, 'epoch': 7.37}

 74%|███████▎  | 538/730 [47:44<16:45,  5.23s/it]
 74%|███████▍  | 539/730 [47:44<17:03,  5.36s/it]
                                                 
{'debug/num_tok_total': 3097.0, 'debug/num_tok_loss': 1798.0, 'debug/num_lat_total': 3097.0, 'debug/num_lat_loss': 1798.0, 'epoch': 7.38}

 74%|███████▍  | 539/730 [47:46<17:03,  5.36s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.33729854226112366, 'epoch': 7.38}

 74%|███████▍  | 539/730 [47:46<17:03,  5.36s/it]
                                                 
{'train/learning_rate_real': 4.226946865017939e-06, 'epoch': 7.38}

 74%|███████▍  | 539/730 [47:46<17:03,  5.36s/it]
                                                 
{'debug/num_tok_total': 2406.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2406.0, 'debug/num_lat_loss': 1771.0, 'epoch': 7.38}

 74%|███████▍  | 539/730 [47:47<17:03,  5.36s/it]
                                                 
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.4161483645439148, 'epoch': 7.38}

 74%|███████▍  | 539/730 [47:47<17:03,  5.36s/it]
                                                 
{'train/learning_rate_real': 4.226946865017939e-06, 'epoch': 7.38}

 74%|███████▍  | 539/730 [47:47<17:03,  5.36s/it]
                                                 
{'debug/num_tok_total': 3063.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 3063.0, 'debug/num_lat_loss': 1775.0, 'epoch': 7.38}

 74%|███████▍  | 539/730 [47:48<17:03,  5.36s/it]
                                                 
{'train/ce_loss': 1.6328125, 'train/diffusion_loss': 0.34313905239105225, 'epoch': 7.38}

 74%|███████▍  | 539/730 [47:48<17:03,  5.36s/it]
                                                 
{'train/learning_rate_real': 4.226946865017939e-06, 'epoch': 7.38}

 74%|███████▍  | 539/730 [47:48<17:03,  5.36s/it]
                                                 
{'debug/num_tok_total': 1991.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 1991.0, 'debug/num_lat_loss': 1767.0, 'epoch': 7.38}

 74%|███████▍  | 539/730 [47:49<17:03,  5.36s/it]
                                                 
{'train/ce_loss': 1.90625, 'train/diffusion_loss': 0.5337827205657959, 'epoch': 7.38}

 74%|███████▍  | 539/730 [47:49<17:03,  5.36s/it]
                                                 
{'train/learning_rate_real': 4.226946865017939e-06, 'epoch': 7.38}

 74%|███████▍  | 539/730 [47:49<17:03,  5.36s/it]03/16/2026 07:35:22 - INFO - __main__ - LoRA debug step 540: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 74%|███████▍  | 540/730 [47:50<17:02,  5.38s/it]
                                                 
{'loss': 2.5372, 'grad_norm': 1.265709638595581, 'learning_rate': 4.226946865017939e-06, 'epoch': 7.4}

 74%|███████▍  | 540/730 [47:50<17:02,  5.38s/it]
                                                 
{'debug/num_tok_total': 2415.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2415.0, 'debug/num_lat_loss': 1781.0, 'epoch': 7.4}

 74%|███████▍  | 540/730 [47:51<17:02,  5.38s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.4418313503265381, 'epoch': 7.4}

 74%|███████▍  | 540/730 [47:51<17:02,  5.38s/it]
                                                 
{'train/learning_rate_real': 4.185448859814554e-06, 'epoch': 7.4}

 74%|███████▍  | 540/730 [47:51<17:02,  5.38s/it]
                                                 
{'debug/num_tok_total': 2339.0, 'debug/num_tok_loss': 1675.0, 'debug/num_lat_total': 2339.0, 'debug/num_lat_loss': 1675.0, 'epoch': 7.4}

 74%|███████▍  | 540/730 [47:52<17:02,  5.38s/it]
                                                 
{'train/ce_loss': 1.640625, 'train/diffusion_loss': 0.4057674705982208, 'epoch': 7.4}

 74%|███████▍  | 540/730 [47:52<17:02,  5.38s/it]
                                                 
{'train/learning_rate_real': 4.185448859814554e-06, 'epoch': 7.4}

 74%|███████▍  | 540/730 [47:52<17:02,  5.38s/it]
                                                 
{'debug/num_tok_total': 2405.0, 'debug/num_tok_loss': 1757.0, 'debug/num_lat_total': 2405.0, 'debug/num_lat_loss': 1757.0, 'epoch': 7.4}

 74%|███████▍  | 540/730 [47:53<17:02,  5.38s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.44093719124794006, 'epoch': 7.4}

 74%|███████▍  | 540/730 [47:53<17:02,  5.38s/it]
                                                 
{'train/learning_rate_real': 4.185448859814554e-06, 'epoch': 7.4}

 74%|███████▍  | 540/730 [47:53<17:02,  5.38s/it]
                                                 
{'debug/num_tok_total': 2188.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 2188.0, 'debug/num_lat_loss': 1765.0, 'epoch': 7.4}

 74%|███████▍  | 540/730 [47:54<17:02,  5.38s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.48902931809425354, 'epoch': 7.4}

 74%|███████▍  | 540/730 [47:54<17:02,  5.38s/it]
                                                 
{'train/learning_rate_real': 4.185448859814554e-06, 'epoch': 7.4}

 74%|███████▍  | 540/730 [47:54<17:02,  5.38s/it]
 74%|███████▍  | 541/730 [47:55<16:38,  5.28s/it]
                                                 
{'debug/num_tok_total': 2663.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2663.0, 'debug/num_lat_loss': 1790.0, 'epoch': 7.41}

 74%|███████▍  | 541/730 [47:56<16:38,  5.28s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.4004761874675751, 'epoch': 7.41}

 74%|███████▍  | 541/730 [47:56<16:38,  5.28s/it]
                                                 
{'train/learning_rate_real': 4.1441145631477465e-06, 'epoch': 7.41}

 74%|███████▍  | 541/730 [47:56<16:38,  5.28s/it]
                                                 
{'debug/num_tok_total': 2204.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2204.0, 'debug/num_lat_loss': 1788.0, 'epoch': 7.41}

 74%|███████▍  | 541/730 [47:57<16:38,  5.28s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.4746943712234497, 'epoch': 7.41}

 74%|███████▍  | 541/730 [47:57<16:38,  5.28s/it]
                                                 
{'train/learning_rate_real': 4.1441145631477465e-06, 'epoch': 7.41}

 74%|███████▍  | 541/730 [47:57<16:38,  5.28s/it]
                                                 
{'debug/num_tok_total': 2849.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2849.0, 'debug/num_lat_loss': 1778.0, 'epoch': 7.41}

 74%|███████▍  | 541/730 [47:58<16:38,  5.28s/it]
                                                 
{'train/ce_loss': 2.296875, 'train/diffusion_loss': 0.3939989507198334, 'epoch': 7.41}

 74%|███████▍  | 541/730 [47:58<16:38,  5.28s/it]
                                                 
{'train/learning_rate_real': 4.1441145631477465e-06, 'epoch': 7.41}

 74%|███████▍  | 541/730 [47:58<16:38,  5.28s/it]
                                                 
{'debug/num_tok_total': 2737.0, 'debug/num_tok_loss': 1667.0, 'debug/num_lat_total': 2737.0, 'debug/num_lat_loss': 1667.0, 'epoch': 7.41}

 74%|███████▍  | 541/730 [47:59<16:38,  5.28s/it]
                                                 
{'train/ce_loss': 1.65625, 'train/diffusion_loss': 0.3877655863761902, 'epoch': 7.41}

 74%|███████▍  | 541/730 [47:59<16:38,  5.28s/it]
                                                 
{'train/learning_rate_real': 4.1441145631477465e-06, 'epoch': 7.41}

 74%|███████▍  | 541/730 [47:59<16:38,  5.28s/it]
 74%|███████▍  | 542/730 [48:00<16:28,  5.26s/it]
                                                 
{'debug/num_tok_total': 2263.0, 'debug/num_tok_loss': 1597.0, 'debug/num_lat_total': 2263.0, 'debug/num_lat_loss': 1597.0, 'epoch': 7.42}

 74%|███████▍  | 542/730 [48:01<16:28,  5.26s/it]
                                                 
{'train/ce_loss': 1.6640625, 'train/diffusion_loss': 0.41057872772216797, 'epoch': 7.42}

 74%|███████▍  | 542/730 [48:01<16:28,  5.26s/it]
                                                 
{'train/learning_rate_real': 4.10294478886509e-06, 'epoch': 7.42}

 74%|███████▍  | 542/730 [48:01<16:28,  5.26s/it]
                                                 
{'debug/num_tok_total': 2624.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 2624.0, 'debug/num_lat_loss': 1768.0, 'epoch': 7.42}

 74%|███████▍  | 542/730 [48:02<16:28,  5.26s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.41963034868240356, 'epoch': 7.42}

 74%|███████▍  | 542/730 [48:02<16:28,  5.26s/it]
                                                 
{'train/learning_rate_real': 4.10294478886509e-06, 'epoch': 7.42}

 74%|███████▍  | 542/730 [48:02<16:28,  5.26s/it]
                                                 
{'debug/num_tok_total': 2602.0, 'debug/num_tok_loss': 1656.0, 'debug/num_lat_total': 2602.0, 'debug/num_lat_loss': 1656.0, 'epoch': 7.42}

 74%|███████▍  | 542/730 [48:04<16:28,  5.26s/it]
                                                 
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.4068897068500519, 'epoch': 7.42}

 74%|███████▍  | 542/730 [48:04<16:28,  5.26s/it]
                                                 
{'train/learning_rate_real': 4.10294478886509e-06, 'epoch': 7.42}

 74%|███████▍  | 542/730 [48:04<16:28,  5.26s/it]
                                                 
{'debug/num_tok_total': 2405.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2405.0, 'debug/num_lat_loss': 1770.0, 'epoch': 7.42}

 74%|███████▍  | 542/730 [48:05<16:28,  5.26s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.4170279800891876, 'epoch': 7.42}

 74%|███████▍  | 542/730 [48:05<16:28,  5.26s/it]
                                                 
{'train/learning_rate_real': 4.10294478886509e-06, 'epoch': 7.42}

 74%|███████▍  | 542/730 [48:05<16:28,  5.26s/it]
 74%|███████▍  | 543/730 [48:05<16:20,  5.25s/it]
                                                 
{'debug/num_tok_total': 2187.0, 'debug/num_tok_loss': 1662.0, 'debug/num_lat_total': 2187.0, 'debug/num_lat_loss': 1662.0, 'epoch': 7.44}

 74%|███████▍  | 543/730 [48:06<16:20,  5.25s/it]
                                                 
{'train/ce_loss': 1.578125, 'train/diffusion_loss': 0.45762717723846436, 'epoch': 7.44}

 74%|███████▍  | 543/730 [48:06<16:20,  5.25s/it]
                                                 
{'train/learning_rate_real': 4.061940347574805e-06, 'epoch': 7.44}

 74%|███████▍  | 543/730 [48:06<16:20,  5.25s/it]
                                                 
{'debug/num_tok_total': 2062.0, 'debug/num_tok_loss': 1622.0, 'debug/num_lat_total': 2062.0, 'debug/num_lat_loss': 1622.0, 'epoch': 7.44}

 74%|███████▍  | 543/730 [48:07<16:20,  5.25s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.48587432503700256, 'epoch': 7.44}

 74%|███████▍  | 543/730 [48:07<16:20,  5.25s/it]
                                                 
{'train/learning_rate_real': 4.061940347574805e-06, 'epoch': 7.44}

 74%|███████▍  | 543/730 [48:07<16:20,  5.25s/it]
                                                 
{'debug/num_tok_total': 1981.0, 'debug/num_tok_loss': 1760.0, 'debug/num_lat_total': 1981.0, 'debug/num_lat_loss': 1760.0, 'epoch': 7.44}

 74%|███████▍  | 543/730 [48:08<16:20,  5.25s/it]
                                                 
{'train/ce_loss': 1.875, 'train/diffusion_loss': 0.5117278099060059, 'epoch': 7.44}

 74%|███████▍  | 543/730 [48:08<16:20,  5.25s/it]
                                                 
{'train/learning_rate_real': 4.061940347574805e-06, 'epoch': 7.44}

 74%|███████▍  | 543/730 [48:08<16:20,  5.25s/it]
                                                 
{'debug/num_tok_total': 2918.0, 'debug/num_tok_loss': 1714.0, 'debug/num_lat_total': 2918.0, 'debug/num_lat_loss': 1714.0, 'epoch': 7.44}

 74%|███████▍  | 543/730 [48:10<16:20,  5.25s/it]
                                                 
{'train/ce_loss': 1.65625, 'train/diffusion_loss': 0.31773754954338074, 'epoch': 7.44}

 74%|███████▍  | 543/730 [48:10<16:20,  5.25s/it]
                                                 
{'train/learning_rate_real': 4.061940347574805e-06, 'epoch': 7.44}

 74%|███████▍  | 543/730 [48:10<16:20,  5.25s/it]
 75%|███████▍  | 544/730 [48:10<16:02,  5.17s/it]
                                                 
{'debug/num_tok_total': 3054.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 3054.0, 'debug/num_lat_loss': 1775.0, 'epoch': 7.45}

 75%|███████▍  | 544/730 [48:11<16:02,  5.17s/it]
                                                 
{'train/ce_loss': 1.90625, 'train/diffusion_loss': 0.31697341799736023, 'epoch': 7.45}

 75%|███████▍  | 544/730 [48:11<16:02,  5.17s/it]
                                                 
{'train/learning_rate_real': 4.021102046629799e-06, 'epoch': 7.45}

 75%|███████▍  | 544/730 [48:11<16:02,  5.17s/it]
                                                 
{'debug/num_tok_total': 2421.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2421.0, 'debug/num_lat_loss': 1770.0, 'epoch': 7.45}

 75%|███████▍  | 544/730 [48:13<16:02,  5.17s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.4358188807964325, 'epoch': 7.45}

 75%|███████▍  | 544/730 [48:13<16:02,  5.17s/it]
                                                 
{'train/learning_rate_real': 4.021102046629799e-06, 'epoch': 7.45}

 75%|███████▍  | 544/730 [48:13<16:02,  5.17s/it]
                                                 
{'debug/num_tok_total': 2407.0, 'debug/num_tok_loss': 1663.0, 'debug/num_lat_total': 2407.0, 'debug/num_lat_loss': 1663.0, 'epoch': 7.45}

 75%|███████▍  | 544/730 [48:14<16:02,  5.17s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.4026140868663788, 'epoch': 7.45}

 75%|███████▍  | 544/730 [48:14<16:02,  5.17s/it]
                                                 
{'train/learning_rate_real': 4.021102046629799e-06, 'epoch': 7.45}

 75%|███████▍  | 544/730 [48:14<16:02,  5.17s/it]
                                                 
{'debug/num_tok_total': 2197.0, 'debug/num_tok_loss': 1554.0, 'debug/num_lat_total': 2197.0, 'debug/num_lat_loss': 1554.0, 'epoch': 7.45}

 75%|███████▍  | 544/730 [48:15<16:02,  5.17s/it]
                                                 
{'train/ce_loss': 1.6484375, 'train/diffusion_loss': 0.4100596010684967, 'epoch': 7.45}

 75%|███████▍  | 544/730 [48:15<16:02,  5.17s/it]
                                                 
{'train/learning_rate_real': 4.021102046629799e-06, 'epoch': 7.45}

 75%|███████▍  | 544/730 [48:15<16:02,  5.17s/it]
 75%|███████▍  | 545/730 [48:15<15:56,  5.17s/it]
                                                 
{'debug/num_tok_total': 2632.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2632.0, 'debug/num_lat_loss': 1774.0, 'epoch': 7.47}

 75%|███████▍  | 545/730 [48:16<15:56,  5.17s/it]
                                                 
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.38504743576049805, 'epoch': 7.47}

 75%|███████▍  | 545/730 [48:16<15:56,  5.17s/it]
                                                 
{'train/learning_rate_real': 3.980430690111786e-06, 'epoch': 7.47}

 75%|███████▍  | 545/730 [48:16<15:56,  5.17s/it]
                                                 
{'debug/num_tok_total': 2433.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2433.0, 'debug/num_lat_loss': 1783.0, 'epoch': 7.47}

 75%|███████▍  | 545/730 [48:18<15:56,  5.17s/it]
                                                 
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.4343603849411011, 'epoch': 7.47}

 75%|███████▍  | 545/730 [48:18<15:56,  5.17s/it]
                                                 
{'train/learning_rate_real': 3.980430690111786e-06, 'epoch': 7.47}

 75%|███████▍  | 545/730 [48:18<15:56,  5.17s/it]
                                                 
{'debug/num_tok_total': 3275.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 3275.0, 'debug/num_lat_loss': 1778.0, 'epoch': 7.47}

 75%|███████▍  | 545/730 [48:19<15:56,  5.17s/it]
                                                 
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.24168667197227478, 'epoch': 7.47}

 75%|███████▍  | 545/730 [48:19<15:56,  5.17s/it]
                                                 
{'train/learning_rate_real': 3.980430690111786e-06, 'epoch': 7.47}

 75%|███████▍  | 545/730 [48:19<15:56,  5.17s/it]
                                                 
{'debug/num_tok_total': 2321.0, 'debug/num_tok_loss': 1679.0, 'debug/num_lat_total': 2321.0, 'debug/num_lat_loss': 1679.0, 'epoch': 7.47}

 75%|███████▍  | 545/730 [48:20<15:56,  5.17s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.42443209886550903, 'epoch': 7.47}

 75%|███████▍  | 545/730 [48:20<15:56,  5.17s/it]
                                                 
{'train/learning_rate_real': 3.980430690111786e-06, 'epoch': 7.47}

 75%|███████▍  | 545/730 [48:20<15:56,  5.17s/it]
 75%|███████▍  | 546/730 [48:21<16:07,  5.26s/it]
                                                 
{'debug/num_tok_total': 2197.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2197.0, 'debug/num_lat_loss': 1781.0, 'epoch': 7.48}

 75%|███████▍  | 546/730 [48:22<16:07,  5.26s/it]
                                                 
{'train/ce_loss': 1.90625, 'train/diffusion_loss': 0.4720486104488373, 'epoch': 7.48}

 75%|███████▍  | 546/730 [48:22<16:07,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.9399270788154305e-06, 'epoch': 7.48}

 75%|███████▍  | 546/730 [48:22<16:07,  5.26s/it]
                                                 
{'debug/num_tok_total': 3037.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 3037.0, 'debug/num_lat_loss': 1763.0, 'epoch': 7.48}

 75%|███████▍  | 546/730 [48:23<16:07,  5.26s/it]
                                                 
{'train/ce_loss': 1.890625, 'train/diffusion_loss': 0.29837027192115784, 'epoch': 7.48}

 75%|███████▍  | 546/730 [48:23<16:07,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.9399270788154305e-06, 'epoch': 7.48}

 75%|███████▍  | 546/730 [48:23<16:07,  5.26s/it]
                                                 
{'debug/num_tok_total': 2711.0, 'debug/num_tok_loss': 1660.0, 'debug/num_lat_total': 2711.0, 'debug/num_lat_loss': 1660.0, 'epoch': 7.48}

 75%|███████▍  | 546/730 [48:24<16:07,  5.26s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.31753769516944885, 'epoch': 7.48}

 75%|███████▍  | 546/730 [48:24<16:07,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.9399270788154305e-06, 'epoch': 7.48}

 75%|███████▍  | 546/730 [48:24<16:07,  5.26s/it]
                                                 
{'debug/num_tok_total': 2318.0, 'debug/num_tok_loss': 1646.0, 'debug/num_lat_total': 2318.0, 'debug/num_lat_loss': 1646.0, 'epoch': 7.48}

 75%|███████▍  | 546/730 [48:26<16:07,  5.26s/it]
                                                 
{'train/ce_loss': 1.8515625, 'train/diffusion_loss': 0.41015323996543884, 'epoch': 7.48}

 75%|███████▍  | 546/730 [48:26<16:07,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.9399270788154305e-06, 'epoch': 7.48}

 75%|███████▍  | 546/730 [48:26<16:07,  5.26s/it]
 75%|███████▍  | 547/730 [48:26<16:02,  5.26s/it]
                                                 
{'debug/num_tok_total': 2830.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 2830.0, 'debug/num_lat_loss': 1767.0, 'epoch': 7.49}

 75%|███████▍  | 547/730 [48:27<16:02,  5.26s/it]
                                                 
{'train/ce_loss': 1.671875, 'train/diffusion_loss': 0.37406599521636963, 'epoch': 7.49}

 75%|███████▍  | 547/730 [48:27<16:02,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.899592010232607e-06, 'epoch': 7.49}

 75%|███████▍  | 547/730 [48:27<16:02,  5.26s/it]
                                                 
{'debug/num_tok_total': 2640.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2640.0, 'debug/num_lat_loss': 1781.0, 'epoch': 7.49}

 75%|███████▍  | 547/730 [48:28<16:02,  5.26s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.4041522443294525, 'epoch': 7.49}

 75%|███████▍  | 547/730 [48:28<16:02,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.899592010232607e-06, 'epoch': 7.49}

 75%|███████▍  | 547/730 [48:28<16:02,  5.26s/it]
                                                 
{'debug/num_tok_total': 2849.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2849.0, 'debug/num_lat_loss': 1784.0, 'epoch': 7.49}

 75%|███████▍  | 547/730 [48:30<16:02,  5.26s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.34697240591049194, 'epoch': 7.49}

 75%|███████▍  | 547/730 [48:30<16:02,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.899592010232607e-06, 'epoch': 7.49}

 75%|███████▍  | 547/730 [48:30<16:02,  5.26s/it]
                                                 
{'debug/num_tok_total': 2481.0, 'debug/num_tok_loss': 1593.0, 'debug/num_lat_total': 2481.0, 'debug/num_lat_loss': 1593.0, 'epoch': 7.49}

 75%|███████▍  | 547/730 [48:31<16:02,  5.26s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.3803322911262512, 'epoch': 7.49}

 75%|███████▍  | 547/730 [48:31<16:02,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.899592010232607e-06, 'epoch': 7.49}

 75%|███████▍  | 547/730 [48:31<16:02,  5.26s/it]
 75%|███████▌  | 548/730 [48:31<16:02,  5.29s/it]
                                                 
{'debug/num_tok_total': 2237.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2237.0, 'debug/num_lat_loss': 1796.0, 'epoch': 7.51}

 75%|███████▌  | 548/730 [48:32<16:02,  5.29s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.5047414898872375, 'epoch': 7.51}

 75%|███████▌  | 548/730 [48:32<16:02,  5.29s/it]
                                                 
{'train/learning_rate_real': 3.859426278536683e-06, 'epoch': 7.51}

 75%|███████▌  | 548/730 [48:32<16:02,  5.29s/it]
                                                 
{'debug/num_tok_total': 2539.0, 'debug/num_tok_loss': 1631.0, 'debug/num_lat_total': 2539.0, 'debug/num_lat_loss': 1631.0, 'epoch': 7.51}

 75%|███████▌  | 548/730 [48:34<16:02,  5.29s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.3775019645690918, 'epoch': 7.51}

 75%|███████▌  | 548/730 [48:34<16:02,  5.29s/it]
                                                 
{'train/learning_rate_real': 3.859426278536683e-06, 'epoch': 7.51}

 75%|███████▌  | 548/730 [48:34<16:02,  5.29s/it]
                                                 
{'debug/num_tok_total': 2834.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 2834.0, 'debug/num_lat_loss': 1767.0, 'epoch': 7.51}

 75%|███████▌  | 548/730 [48:35<16:02,  5.29s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.36412084102630615, 'epoch': 7.51}

 75%|███████▌  | 548/730 [48:35<16:02,  5.29s/it]
                                                 
{'train/learning_rate_real': 3.859426278536683e-06, 'epoch': 7.51}

 75%|███████▌  | 548/730 [48:35<16:02,  5.29s/it]
                                                 
{'debug/num_tok_total': 2349.0, 'debug/num_tok_loss': 1625.0, 'debug/num_lat_total': 2349.0, 'debug/num_lat_loss': 1625.0, 'epoch': 7.51}

 75%|███████▌  | 548/730 [48:36<16:02,  5.29s/it]
                                                 
{'train/ce_loss': 1.96875, 'train/diffusion_loss': 0.4071674346923828, 'epoch': 7.51}

 75%|███████▌  | 548/730 [48:36<16:02,  5.29s/it]
                                                 
{'train/learning_rate_real': 3.859426278536683e-06, 'epoch': 7.51}

 75%|███████▌  | 548/730 [48:36<16:02,  5.29s/it]
 75%|███████▌  | 549/730 [48:37<15:56,  5.28s/it]
                                                 
{'debug/num_tok_total': 2428.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2428.0, 'debug/num_lat_loss': 1789.0, 'epoch': 7.52}

 75%|███████▌  | 549/730 [48:38<15:56,  5.28s/it]
                                                 
{'train/ce_loss': 1.9765625, 'train/diffusion_loss': 0.43682992458343506, 'epoch': 7.52}

 75%|███████▌  | 549/730 [48:38<15:56,  5.28s/it]
                                                 
{'train/learning_rate_real': 3.819430674566872e-06, 'epoch': 7.52}

 75%|███████▌  | 549/730 [48:38<15:56,  5.28s/it]
                                                 
{'debug/num_tok_total': 2958.0, 'debug/num_tok_loss': 1670.0, 'debug/num_lat_total': 2958.0, 'debug/num_lat_loss': 1670.0, 'epoch': 7.52}

 75%|███████▌  | 549/730 [48:39<15:56,  5.28s/it]
                                                 
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.3279258906841278, 'epoch': 7.52}

 75%|███████▌  | 549/730 [48:39<15:56,  5.28s/it]
                                                 
{'train/learning_rate_real': 3.819430674566872e-06, 'epoch': 7.52}

 75%|███████▌  | 549/730 [48:39<15:56,  5.28s/it]
                                                 
{'debug/num_tok_total': 2415.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2415.0, 'debug/num_lat_loss': 1778.0, 'epoch': 7.52}

 75%|███████▌  | 549/730 [48:40<15:56,  5.28s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.49888876080513, 'epoch': 7.52}

 75%|███████▌  | 549/730 [48:40<15:56,  5.28s/it]
                                                 
{'train/learning_rate_real': 3.819430674566872e-06, 'epoch': 7.52}

 75%|███████▌  | 549/730 [48:40<15:56,  5.28s/it]
                                                 
{'debug/num_tok_total': 2173.0, 'debug/num_tok_loss': 1746.0, 'debug/num_lat_total': 2173.0, 'debug/num_lat_loss': 1746.0, 'epoch': 7.52}

 75%|███████▌  | 549/730 [48:41<15:56,  5.28s/it]
                                                 
{'train/ce_loss': 1.6171875, 'train/diffusion_loss': 0.4969797730445862, 'epoch': 7.52}

 75%|███████▌  | 549/730 [48:41<15:56,  5.28s/it]
                                                 
{'train/learning_rate_real': 3.819430674566872e-06, 'epoch': 7.52}

 75%|███████▌  | 549/730 [48:41<15:56,  5.28s/it]03/16/2026 07:36:14 - INFO - __main__ - LoRA debug step 550: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 75%|███████▌  | 550/730 [48:42<15:46,  5.26s/it]
                                                 
{'loss': 2.5704, 'grad_norm': 1.1224170923233032, 'learning_rate': 3.819430674566872e-06, 'epoch': 7.53}

 75%|███████▌  | 550/730 [48:42<15:46,  5.26s/it]
                                                 
{'debug/num_tok_total': 2601.0, 'debug/num_tok_loss': 1544.0, 'debug/num_lat_total': 2601.0, 'debug/num_lat_loss': 1544.0, 'epoch': 7.53}

 75%|███████▌  | 550/730 [48:43<15:46,  5.26s/it]
                                                 
{'train/ce_loss': 1.671875, 'train/diffusion_loss': 0.35317564010620117, 'epoch': 7.53}

 75%|███████▌  | 550/730 [48:43<15:46,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.7796059858126927e-06, 'epoch': 7.53}

 75%|███████▌  | 550/730 [48:43<15:46,  5.26s/it]
                                                 
{'debug/num_tok_total': 2622.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2622.0, 'debug/num_lat_loss': 1766.0, 'epoch': 7.53}

 75%|███████▌  | 550/730 [48:44<15:46,  5.26s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.36522096395492554, 'epoch': 7.53}

 75%|███████▌  | 550/730 [48:44<15:46,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.7796059858126927e-06, 'epoch': 7.53}

 75%|███████▌  | 550/730 [48:44<15:46,  5.26s/it]
                                                 
{'debug/num_tok_total': 2870.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2870.0, 'debug/num_lat_loss': 1785.0, 'epoch': 7.53}

 75%|███████▌  | 550/730 [48:46<15:46,  5.26s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.380035400390625, 'epoch': 7.53}

 75%|███████▌  | 550/730 [48:46<15:46,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.7796059858126927e-06, 'epoch': 7.53}

 75%|███████▌  | 550/730 [48:46<15:46,  5.26s/it]
                                                 
{'debug/num_tok_total': 2431.0, 'debug/num_tok_loss': 1674.0, 'debug/num_lat_total': 2431.0, 'debug/num_lat_loss': 1674.0, 'epoch': 7.53}

 75%|███████▌  | 550/730 [48:47<15:46,  5.26s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.41258060932159424, 'epoch': 7.53}

 75%|███████▌  | 550/730 [48:47<15:46,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.7796059858126927e-06, 'epoch': 7.53}

 75%|███████▌  | 550/730 [48:47<15:46,  5.26s/it]
 75%|███████▌  | 551/730 [48:47<15:48,  5.30s/it]
                                                 
{'debug/num_tok_total': 2867.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2867.0, 'debug/num_lat_loss': 1791.0, 'epoch': 7.55}

 75%|███████▌  | 551/730 [48:48<15:48,  5.30s/it]
                                                 
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.35512274503707886, 'epoch': 7.55}

 75%|███████▌  | 551/730 [48:48<15:48,  5.30s/it]
                                                 
{'train/learning_rate_real': 3.739952996398427e-06, 'epoch': 7.55}

 75%|███████▌  | 551/730 [48:48<15:48,  5.30s/it]
                                                 
{'debug/num_tok_total': 2633.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2633.0, 'debug/num_lat_loss': 1778.0, 'epoch': 7.55}

 75%|███████▌  | 551/730 [48:50<15:48,  5.30s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.39710748195648193, 'epoch': 7.55}

 75%|███████▌  | 551/730 [48:50<15:48,  5.30s/it]
                                                 
{'train/learning_rate_real': 3.739952996398427e-06, 'epoch': 7.55}

 75%|███████▌  | 551/730 [48:50<15:48,  5.30s/it]
                                                 
{'debug/num_tok_total': 2410.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2410.0, 'debug/num_lat_loss': 1763.0, 'epoch': 7.55}

 75%|███████▌  | 551/730 [48:51<15:48,  5.30s/it]
                                                 
{'train/ce_loss': 1.921875, 'train/diffusion_loss': 0.4415169656276703, 'epoch': 7.55}

 75%|███████▌  | 551/730 [48:51<15:48,  5.30s/it]
                                                 
{'train/learning_rate_real': 3.739952996398427e-06, 'epoch': 7.55}

 75%|███████▌  | 551/730 [48:51<15:48,  5.30s/it]
                                                 
{'debug/num_tok_total': 2134.0, 'debug/num_tok_loss': 1530.0, 'debug/num_lat_total': 2134.0, 'debug/num_lat_loss': 1530.0, 'epoch': 7.55}

 75%|███████▌  | 551/730 [48:52<15:48,  5.30s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.438183456659317, 'epoch': 7.55}

 75%|███████▌  | 551/730 [48:52<15:48,  5.30s/it]
                                                 
{'train/learning_rate_real': 3.739952996398427e-06, 'epoch': 7.55}

 75%|███████▌  | 551/730 [48:52<15:48,  5.30s/it]
 76%|███████▌  | 552/730 [48:52<15:37,  5.26s/it]
                                                 
{'debug/num_tok_total': 2416.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2416.0, 'debug/num_lat_loss': 1787.0, 'epoch': 7.56}

 76%|███████▌  | 552/730 [48:54<15:37,  5.26s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.4395983815193176, 'epoch': 7.56}

 76%|███████▌  | 552/730 [48:54<15:37,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.7004724870677133e-06, 'epoch': 7.56}

 76%|███████▌  | 552/730 [48:54<15:37,  5.26s/it]
                                                 
{'debug/num_tok_total': 3077.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 3077.0, 'debug/num_lat_loss': 1785.0, 'epoch': 7.56}

 76%|███████▌  | 552/730 [48:55<15:37,  5.26s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.2828432023525238, 'epoch': 7.56}

 76%|███████▌  | 552/730 [48:55<15:37,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.7004724870677133e-06, 'epoch': 7.56}

 76%|███████▌  | 552/730 [48:55<15:37,  5.26s/it]
                                                 
{'debug/num_tok_total': 2659.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2659.0, 'debug/num_lat_loss': 1794.0, 'epoch': 7.56}

 76%|███████▌  | 552/730 [48:56<15:37,  5.26s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.41582196950912476, 'epoch': 7.56}

 76%|███████▌  | 552/730 [48:56<15:37,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.7004724870677133e-06, 'epoch': 7.56}

 76%|███████▌  | 552/730 [48:56<15:37,  5.26s/it]
                                                 
{'debug/num_tok_total': 2597.0, 'debug/num_tok_loss': 1746.0, 'debug/num_lat_total': 2597.0, 'debug/num_lat_loss': 1746.0, 'epoch': 7.56}

 76%|███████▌  | 552/730 [48:57<15:37,  5.26s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.40991100668907166, 'epoch': 7.56}

 76%|███████▌  | 552/730 [48:57<15:37,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.7004724870677133e-06, 'epoch': 7.56}

 76%|███████▌  | 552/730 [48:57<15:37,  5.26s/it]
 76%|███████▌  | 553/730 [48:58<15:41,  5.32s/it]
                                                 
{'debug/num_tok_total': 3093.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 3093.0, 'debug/num_lat_loss': 1794.0, 'epoch': 7.58}

 76%|███████▌  | 553/730 [48:59<15:41,  5.32s/it]
                                                 
{'train/ce_loss': 2.015625, 'train/diffusion_loss': 0.2898784875869751, 'epoch': 7.58}

 76%|███████▌  | 553/730 [48:59<15:41,  5.32s/it]
                                                 
{'train/learning_rate_real': 3.6611652351681568e-06, 'epoch': 7.58}

 76%|███████▌  | 553/730 [48:59<15:41,  5.32s/it]
                                                 
{'debug/num_tok_total': 3076.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 3076.0, 'debug/num_lat_loss': 1786.0, 'epoch': 7.58}

 76%|███████▌  | 553/730 [49:01<15:41,  5.32s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.30902954936027527, 'epoch': 7.58}

 76%|███████▌  | 553/730 [49:01<15:41,  5.32s/it]
                                                 
{'train/learning_rate_real': 3.6611652351681568e-06, 'epoch': 7.58}

 76%|███████▌  | 553/730 [49:01<15:41,  5.32s/it]
                                                 
{'debug/num_tok_total': 2708.0, 'debug/num_tok_loss': 1701.0, 'debug/num_lat_total': 2708.0, 'debug/num_lat_loss': 1701.0, 'epoch': 7.58}

 76%|███████▌  | 553/730 [49:02<15:41,  5.32s/it]
                                                 
{'train/ce_loss': 1.9296875, 'train/diffusion_loss': 0.36524373292922974, 'epoch': 7.58}

 76%|███████▌  | 553/730 [49:02<15:41,  5.32s/it]
                                                 
{'train/learning_rate_real': 3.6611652351681568e-06, 'epoch': 7.58}

 76%|███████▌  | 553/730 [49:02<15:41,  5.32s/it]
                                                 
{'debug/num_tok_total': 2872.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2872.0, 'debug/num_lat_loss': 1794.0, 'epoch': 7.58}

 76%|███████▌  | 553/730 [49:03<15:41,  5.32s/it]
                                                 
{'train/ce_loss': 1.9375, 'train/diffusion_loss': 0.3450227975845337, 'epoch': 7.58}

 76%|███████▌  | 553/730 [49:03<15:41,  5.32s/it]
                                                 
{'train/learning_rate_real': 3.6611652351681568e-06, 'epoch': 7.58}

 76%|███████▌  | 553/730 [49:03<15:41,  5.32s/it]
 76%|███████▌  | 554/730 [49:04<15:58,  5.45s/it]
                                                 
{'debug/num_tok_total': 2401.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2401.0, 'debug/num_lat_loss': 1774.0, 'epoch': 7.59}

 76%|███████▌  | 554/730 [49:05<15:58,  5.45s/it]
                                                 
{'train/ce_loss': 1.9375, 'train/diffusion_loss': 0.43930497765541077, 'epoch': 7.59}

 76%|███████▌  | 554/730 [49:05<15:58,  5.45s/it]
                                                 
{'train/learning_rate_real': 3.6220320146360244e-06, 'epoch': 7.59}

 76%|███████▌  | 554/730 [49:05<15:58,  5.45s/it]
                                                 
{'debug/num_tok_total': 2396.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2396.0, 'debug/num_lat_loss': 1771.0, 'epoch': 7.59}

 76%|███████▌  | 554/730 [49:06<15:58,  5.45s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.4401986598968506, 'epoch': 7.59}

 76%|███████▌  | 554/730 [49:06<15:58,  5.45s/it]
                                                 
{'train/learning_rate_real': 3.6220320146360244e-06, 'epoch': 7.59}

 76%|███████▌  | 554/730 [49:06<15:58,  5.45s/it]
                                                 
{'debug/num_tok_total': 2431.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2431.0, 'debug/num_lat_loss': 1783.0, 'epoch': 7.59}

 76%|███████▌  | 554/730 [49:07<15:58,  5.45s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.43268758058547974, 'epoch': 7.59}

 76%|███████▌  | 554/730 [49:07<15:58,  5.45s/it]
                                                 
{'train/learning_rate_real': 3.6220320146360244e-06, 'epoch': 7.59}

 76%|███████▌  | 554/730 [49:07<15:58,  5.45s/it]
                                                 
{'debug/num_tok_total': 3098.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 3098.0, 'debug/num_lat_loss': 1795.0, 'epoch': 7.59}

 76%|███████▌  | 554/730 [49:08<15:58,  5.45s/it]
                                                 
{'train/ce_loss': 1.6171875, 'train/diffusion_loss': 0.3015957474708557, 'epoch': 7.59}

 76%|███████▌  | 554/730 [49:08<15:58,  5.45s/it]
                                                 
{'train/learning_rate_real': 3.6220320146360244e-06, 'epoch': 7.59}

 76%|███████▌  | 554/730 [49:08<15:58,  5.45s/it]
 76%|███████▌  | 555/730 [49:09<15:47,  5.41s/it]
                                                 
{'debug/num_tok_total': 2434.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2434.0, 'debug/num_lat_loss': 1786.0, 'epoch': 7.6}

 76%|███████▌  | 555/730 [49:10<15:47,  5.41s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.4460858106613159, 'epoch': 7.6}

 76%|███████▌  | 555/730 [49:10<15:47,  5.41s/it]
                                                 
{'train/learning_rate_real': 3.5830735959810082e-06, 'epoch': 7.6}

 76%|███████▌  | 555/730 [49:10<15:47,  5.41s/it]
                                                 
{'debug/num_tok_total': 2696.0, 'debug/num_tok_loss': 1819.0, 'debug/num_lat_total': 2696.0, 'debug/num_lat_loss': 1819.0, 'epoch': 7.6}

 76%|███████▌  | 555/730 [49:11<15:47,  5.41s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.4096073806285858, 'epoch': 7.6}

 76%|███████▌  | 555/730 [49:11<15:47,  5.41s/it]
                                                 
{'train/learning_rate_real': 3.5830735959810082e-06, 'epoch': 7.6}

 76%|███████▌  | 555/730 [49:11<15:47,  5.41s/it]
                                                 
{'debug/num_tok_total': 1803.0, 'debug/num_tok_loss': 1803.0, 'debug/num_lat_total': 1803.0, 'debug/num_lat_loss': 1803.0, 'epoch': 7.6}

 76%|███████▌  | 555/730 [49:12<15:47,  5.41s/it]
                                                 
{'train/ce_loss': 1.890625, 'train/diffusion_loss': 0.5851296782493591, 'epoch': 7.6}

 76%|███████▌  | 555/730 [49:12<15:47,  5.41s/it]
                                                 
{'train/learning_rate_real': 3.5830735959810082e-06, 'epoch': 7.6}

 76%|███████▌  | 555/730 [49:12<15:47,  5.41s/it]
                                                 
{'debug/num_tok_total': 2642.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2642.0, 'debug/num_lat_loss': 1783.0, 'epoch': 7.6}

 76%|███████▌  | 555/730 [49:13<15:47,  5.41s/it]
                                                 
{'train/ce_loss': 1.609375, 'train/diffusion_loss': 0.3945103883743286, 'epoch': 7.6}

 76%|███████▌  | 555/730 [49:13<15:47,  5.41s/it]
                                                 
{'train/learning_rate_real': 3.5830735959810082e-06, 'epoch': 7.6}

 76%|███████▌  | 555/730 [49:13<15:47,  5.41s/it]
 76%|███████▌  | 556/730 [49:14<15:14,  5.26s/it]
                                                 
{'debug/num_tok_total': 2875.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2875.0, 'debug/num_lat_loss': 1790.0, 'epoch': 7.62}

 76%|███████▌  | 556/730 [49:15<15:14,  5.26s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.33037760853767395, 'epoch': 7.62}

 76%|███████▌  | 556/730 [49:15<15:14,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.5442907462710605e-06, 'epoch': 7.62}

 76%|███████▌  | 556/730 [49:15<15:14,  5.26s/it]
                                                 
{'debug/num_tok_total': 2438.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2438.0, 'debug/num_lat_loss': 1793.0, 'epoch': 7.62}

 76%|███████▌  | 556/730 [49:16<15:14,  5.26s/it]
                                                 
{'train/ce_loss': 2.140625, 'train/diffusion_loss': 0.43046316504478455, 'epoch': 7.62}

 76%|███████▌  | 556/730 [49:16<15:14,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.5442907462710605e-06, 'epoch': 7.62}

 76%|███████▌  | 556/730 [49:16<15:14,  5.26s/it]
                                                 
{'debug/num_tok_total': 2851.0, 'debug/num_tok_loss': 1671.0, 'debug/num_lat_total': 2851.0, 'debug/num_lat_loss': 1671.0, 'epoch': 7.62}

 76%|███████▌  | 556/730 [49:18<15:14,  5.26s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.3255270719528198, 'epoch': 7.62}

 76%|███████▌  | 556/730 [49:18<15:14,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.5442907462710605e-06, 'epoch': 7.62}

 76%|███████▌  | 556/730 [49:18<15:14,  5.26s/it]
                                                 
{'debug/num_tok_total': 3092.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 3092.0, 'debug/num_lat_loss': 1786.0, 'epoch': 7.62}

 76%|███████▌  | 556/730 [49:19<15:14,  5.26s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.30197182297706604, 'epoch': 7.62}

 76%|███████▌  | 556/730 [49:19<15:14,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.5442907462710605e-06, 'epoch': 7.62}

 76%|███████▌  | 556/730 [49:19<15:14,  5.26s/it]
 76%|███████▋  | 557/730 [49:19<15:25,  5.35s/it]
                                                 
{'debug/num_tok_total': 2435.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2435.0, 'debug/num_lat_loss': 1783.0, 'epoch': 7.63}

 76%|███████▋  | 557/730 [49:20<15:25,  5.35s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.462166428565979, 'epoch': 7.63}

 76%|███████▋  | 557/730 [49:20<15:25,  5.35s/it]
                                                 
{'train/learning_rate_real': 3.5056842291172912e-06, 'epoch': 7.63}

 76%|███████▋  | 557/730 [49:20<15:25,  5.35s/it]
                                                 
{'debug/num_tok_total': 2612.0, 'debug/num_tok_loss': 1759.0, 'debug/num_lat_total': 2612.0, 'debug/num_lat_loss': 1759.0, 'epoch': 7.63}

 76%|███████▋  | 557/730 [49:22<15:25,  5.35s/it]
                                                 
{'train/ce_loss': 1.8515625, 'train/diffusion_loss': 0.4133981764316559, 'epoch': 7.63}

 76%|███████▋  | 557/730 [49:22<15:25,  5.35s/it]
                                                 
{'train/learning_rate_real': 3.5056842291172912e-06, 'epoch': 7.63}

 76%|███████▋  | 557/730 [49:22<15:25,  5.35s/it]
                                                 
{'debug/num_tok_total': 2635.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2635.0, 'debug/num_lat_loss': 1773.0, 'epoch': 7.63}

 76%|███████▋  | 557/730 [49:23<15:25,  5.35s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.391751766204834, 'epoch': 7.63}

 76%|███████▋  | 557/730 [49:23<15:25,  5.35s/it]
                                                 
{'train/learning_rate_real': 3.5056842291172912e-06, 'epoch': 7.63}

 76%|███████▋  | 557/730 [49:23<15:25,  5.35s/it]
                                                 
{'debug/num_tok_total': 2398.0, 'debug/num_tok_loss': 1761.0, 'debug/num_lat_total': 2398.0, 'debug/num_lat_loss': 1761.0, 'epoch': 7.63}

 76%|███████▋  | 557/730 [49:24<15:25,  5.35s/it]
                                                 
{'train/ce_loss': 1.6171875, 'train/diffusion_loss': 0.45379704236984253, 'epoch': 7.63}

 76%|███████▋  | 557/730 [49:24<15:25,  5.35s/it]
                                                 
{'train/learning_rate_real': 3.5056842291172912e-06, 'epoch': 7.63}

 76%|███████▋  | 557/730 [49:24<15:25,  5.35s/it]
 76%|███████▋  | 558/730 [49:25<15:16,  5.33s/it]
                                                 
{'debug/num_tok_total': 2812.0, 'debug/num_tok_loss': 1756.0, 'debug/num_lat_total': 2812.0, 'debug/num_lat_loss': 1756.0, 'epoch': 7.64}

 76%|███████▋  | 558/730 [49:26<15:16,  5.33s/it]
                                                 
{'train/ce_loss': 1.6171875, 'train/diffusion_loss': 0.3835338056087494, 'epoch': 7.64}

 76%|███████▋  | 558/730 [49:26<15:16,  5.33s/it]
                                                 
{'train/learning_rate_real': 3.4672548046589105e-06, 'epoch': 7.64}

 76%|███████▋  | 558/730 [49:26<15:16,  5.33s/it]
                                                 
{'debug/num_tok_total': 2408.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2408.0, 'debug/num_lat_loss': 1763.0, 'epoch': 7.64}

 76%|███████▋  | 558/730 [49:27<15:16,  5.33s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.42137765884399414, 'epoch': 7.64}

 76%|███████▋  | 558/730 [49:27<15:16,  5.33s/it]
                                                 
{'train/learning_rate_real': 3.4672548046589105e-06, 'epoch': 7.64}

 76%|███████▋  | 558/730 [49:27<15:16,  5.33s/it]
                                                 
{'debug/num_tok_total': 2840.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2840.0, 'debug/num_lat_loss': 1766.0, 'epoch': 7.64}

 76%|███████▋  | 558/730 [49:28<15:16,  5.33s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.3423376679420471, 'epoch': 7.64}

 76%|███████▋  | 558/730 [49:28<15:16,  5.33s/it]
                                                 
{'train/learning_rate_real': 3.4672548046589105e-06, 'epoch': 7.64}

 76%|███████▋  | 558/730 [49:28<15:16,  5.33s/it]
                                                 
{'debug/num_tok_total': 2611.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2611.0, 'debug/num_lat_loss': 1766.0, 'epoch': 7.64}

 76%|███████▋  | 558/730 [49:29<15:16,  5.33s/it]
                                                 
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.3939228951931, 'epoch': 7.64}

 76%|███████▋  | 558/730 [49:29<15:16,  5.33s/it]
                                                 
{'train/learning_rate_real': 3.4672548046589105e-06, 'epoch': 7.64}

 76%|███████▋  | 558/730 [49:29<15:16,  5.33s/it]
 77%|███████▋  | 559/730 [49:30<15:02,  5.28s/it]
                                                 
{'debug/num_tok_total': 2420.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2420.0, 'debug/num_lat_loss': 1764.0, 'epoch': 7.66}

 77%|███████▋  | 559/730 [49:31<15:02,  5.28s/it]
                                                 
{'train/ce_loss': 1.6171875, 'train/diffusion_loss': 0.42871248722076416, 'epoch': 7.66}

 77%|███████▋  | 559/730 [49:31<15:02,  5.28s/it]
                                                 
{'train/learning_rate_real': 3.4290032295483012e-06, 'epoch': 7.66}

 77%|███████▋  | 559/730 [49:31<15:02,  5.28s/it]
                                                 
{'debug/num_tok_total': 2878.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2878.0, 'debug/num_lat_loss': 1793.0, 'epoch': 7.66}

 77%|███████▋  | 559/730 [49:32<15:02,  5.28s/it]
                                                 
{'train/ce_loss': 1.8515625, 'train/diffusion_loss': 0.37304413318634033, 'epoch': 7.66}

 77%|███████▋  | 559/730 [49:32<15:02,  5.28s/it]
                                                 
{'train/learning_rate_real': 3.4290032295483012e-06, 'epoch': 7.66}

 77%|███████▋  | 559/730 [49:32<15:02,  5.28s/it]
                                                 
{'debug/num_tok_total': 2410.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2410.0, 'debug/num_lat_loss': 1778.0, 'epoch': 7.66}

 77%|███████▋  | 559/730 [49:33<15:02,  5.28s/it]
                                                 
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.43494951725006104, 'epoch': 7.66}

 77%|███████▋  | 559/730 [49:33<15:02,  5.28s/it]
                                                 
{'train/learning_rate_real': 3.4290032295483012e-06, 'epoch': 7.66}

 77%|███████▋  | 559/730 [49:33<15:02,  5.28s/it]
                                                 
{'debug/num_tok_total': 2427.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2427.0, 'debug/num_lat_loss': 1782.0, 'epoch': 7.66}

 77%|███████▋  | 559/730 [49:35<15:02,  5.28s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.4354192018508911, 'epoch': 7.66}

 77%|███████▋  | 559/730 [49:35<15:02,  5.28s/it]
                                                 
{'train/learning_rate_real': 3.4290032295483012e-06, 'epoch': 7.66}

 77%|███████▋  | 559/730 [49:35<15:02,  5.28s/it]03/16/2026 07:37:07 - INFO - __main__ - LoRA debug step 560: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 77%|███████▋  | 560/730 [49:35<14:53,  5.26s/it]
                                                 
{'loss': 2.4945, 'grad_norm': 1.1772764921188354, 'learning_rate': 3.4290032295483012e-06, 'epoch': 7.67}

 77%|███████▋  | 560/730 [49:35<14:53,  5.26s/it]
                                                 
{'debug/num_tok_total': 3055.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 3055.0, 'debug/num_lat_loss': 1777.0, 'epoch': 7.67}

 77%|███████▋  | 560/730 [49:36<14:53,  5.26s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.3003534972667694, 'epoch': 7.67}

 77%|███████▋  | 560/730 [49:36<14:53,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.390930256936077e-06, 'epoch': 7.67}

 77%|███████▋  | 560/730 [49:36<14:53,  5.26s/it]
                                                 
{'debug/num_tok_total': 2630.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2630.0, 'debug/num_lat_loss': 1778.0, 'epoch': 7.67}

 77%|███████▋  | 560/730 [49:38<14:53,  5.26s/it]
                                                 
{'train/ce_loss': 1.90625, 'train/diffusion_loss': 0.41532668471336365, 'epoch': 7.67}

 77%|███████▋  | 560/730 [49:38<14:53,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.390930256936077e-06, 'epoch': 7.67}

 77%|███████▋  | 560/730 [49:38<14:53,  5.26s/it]
                                                 
{'debug/num_tok_total': 2602.0, 'debug/num_tok_loss': 1725.0, 'debug/num_lat_total': 2602.0, 'debug/num_lat_loss': 1725.0, 'epoch': 7.67}

 77%|███████▋  | 560/730 [49:39<14:53,  5.26s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.37217092514038086, 'epoch': 7.67}

 77%|███████▋  | 560/730 [49:39<14:53,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.390930256936077e-06, 'epoch': 7.67}

 77%|███████▋  | 560/730 [49:39<14:53,  5.26s/it]
                                                 
{'debug/num_tok_total': 3271.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 3271.0, 'debug/num_lat_loss': 1777.0, 'epoch': 7.67}

 77%|███████▋  | 560/730 [49:40<14:53,  5.26s/it]
                                                 
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.28343665599823, 'epoch': 7.67}

 77%|███████▋  | 560/730 [49:40<14:53,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.390930256936077e-06, 'epoch': 7.67}

 77%|███████▋  | 560/730 [49:40<14:53,  5.26s/it]
 77%|███████▋  | 561/730 [49:41<15:22,  5.46s/it]
                                                 
{'debug/num_tok_total': 2406.0, 'debug/num_tok_loss': 1762.0, 'debug/num_lat_total': 2406.0, 'debug/num_lat_loss': 1762.0, 'epoch': 7.68}

 77%|███████▋  | 561/730 [49:42<15:22,  5.46s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.4421650171279907, 'epoch': 7.68}

 77%|███████▋  | 561/730 [49:42<15:22,  5.46s/it]
                                                 
{'train/learning_rate_real': 3.3530366364562914e-06, 'epoch': 7.68}

 77%|███████▋  | 561/730 [49:42<15:22,  5.46s/it]
                                                 
{'debug/num_tok_total': 2312.0, 'debug/num_tok_loss': 1709.0, 'debug/num_lat_total': 2312.0, 'debug/num_lat_loss': 1709.0, 'epoch': 7.68}

 77%|███████▋  | 561/730 [49:43<15:22,  5.46s/it]
                                                 
{'train/ce_loss': 1.6640625, 'train/diffusion_loss': 0.4473550021648407, 'epoch': 7.68}

 77%|███████▋  | 561/730 [49:43<15:22,  5.46s/it]
                                                 
{'train/learning_rate_real': 3.3530366364562914e-06, 'epoch': 7.68}

 77%|███████▋  | 561/730 [49:43<15:22,  5.46s/it]
                                                 
{'debug/num_tok_total': 2424.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2424.0, 'debug/num_lat_loss': 1777.0, 'epoch': 7.68}

 77%|███████▋  | 561/730 [49:44<15:22,  5.46s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.4566288888454437, 'epoch': 7.68}

 77%|███████▋  | 561/730 [49:44<15:22,  5.46s/it]
                                                 
{'train/learning_rate_real': 3.3530366364562914e-06, 'epoch': 7.68}

 77%|███████▋  | 561/730 [49:44<15:22,  5.46s/it]
                                                 
{'debug/num_tok_total': 2014.0, 'debug/num_tok_loss': 1423.0, 'debug/num_lat_total': 2014.0, 'debug/num_lat_loss': 1423.0, 'epoch': 7.68}

 77%|███████▋  | 561/730 [49:46<15:22,  5.46s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.41504815220832825, 'epoch': 7.68}

 77%|███████▋  | 561/730 [49:46<15:22,  5.46s/it]
                                                 
{'train/learning_rate_real': 3.3530366364562914e-06, 'epoch': 7.68}

 77%|███████▋  | 561/730 [49:46<15:22,  5.46s/it]
 77%|███████▋  | 562/730 [49:46<14:54,  5.33s/it]
                                                 
{'debug/num_tok_total': 2440.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2440.0, 'debug/num_lat_loss': 1788.0, 'epoch': 7.7}

 77%|███████▋  | 562/730 [49:47<14:54,  5.33s/it]
                                                 
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.447273313999176, 'epoch': 7.7}

 77%|███████▋  | 562/730 [49:47<14:54,  5.33s/it]
                                                 
{'train/learning_rate_real': 3.3153231142116617e-06, 'epoch': 7.7}

 77%|███████▋  | 562/730 [49:47<14:54,  5.33s/it]
                                                 
{'debug/num_tok_total': 3263.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 3263.0, 'debug/num_lat_loss': 1764.0, 'epoch': 7.7}

 77%|███████▋  | 562/730 [49:49<14:54,  5.33s/it]
                                                 
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.25116926431655884, 'epoch': 7.7}

 77%|███████▋  | 562/730 [49:49<14:54,  5.33s/it]
                                                 
{'train/learning_rate_real': 3.3153231142116617e-06, 'epoch': 7.7}

 77%|███████▋  | 562/730 [49:49<14:54,  5.33s/it]
                                                 
{'debug/num_tok_total': 2846.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2846.0, 'debug/num_lat_loss': 1777.0, 'epoch': 7.7}

 77%|███████▋  | 562/730 [49:50<14:54,  5.33s/it]
                                                 
{'train/ce_loss': 1.6875, 'train/diffusion_loss': 0.3709633946418762, 'epoch': 7.7}

 77%|███████▋  | 562/730 [49:50<14:54,  5.33s/it]
                                                 
{'train/learning_rate_real': 3.3153231142116617e-06, 'epoch': 7.7}

 77%|███████▋  | 562/730 [49:50<14:54,  5.33s/it]
                                                 
{'debug/num_tok_total': 1985.0, 'debug/num_tok_loss': 1762.0, 'debug/num_lat_total': 1985.0, 'debug/num_lat_loss': 1762.0, 'epoch': 7.7}

 77%|███████▋  | 562/730 [49:51<14:54,  5.33s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.5031887292861938, 'epoch': 7.7}

 77%|███████▋  | 562/730 [49:51<14:54,  5.33s/it]
                                                 
{'train/learning_rate_real': 3.3153231142116617e-06, 'epoch': 7.7}

 77%|███████▋  | 562/730 [49:51<14:54,  5.33s/it]
 77%|███████▋  | 563/730 [49:51<14:51,  5.34s/it]
                                                 
{'debug/num_tok_total': 2485.0, 'debug/num_tok_loss': 1615.0, 'debug/num_lat_total': 2485.0, 'debug/num_lat_loss': 1615.0, 'epoch': 7.71}

 77%|███████▋  | 563/730 [49:52<14:51,  5.34s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.35747194290161133, 'epoch': 7.71}

 77%|███████▋  | 563/730 [49:53<14:51,  5.34s/it]
                                                 
{'train/learning_rate_real': 3.277790432758862e-06, 'epoch': 7.71}

 77%|███████▋  | 563/730 [49:53<14:51,  5.34s/it]
                                                 
{'debug/num_tok_total': 2031.0, 'debug/num_tok_loss': 1814.0, 'debug/num_lat_total': 2031.0, 'debug/num_lat_loss': 1814.0, 'epoch': 7.71}

 77%|███████▋  | 563/730 [49:54<14:51,  5.34s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.519098162651062, 'epoch': 7.71}

 77%|███████▋  | 563/730 [49:54<14:51,  5.34s/it]
                                                 
{'train/learning_rate_real': 3.277790432758862e-06, 'epoch': 7.71}

 77%|███████▋  | 563/730 [49:54<14:51,  5.34s/it]
                                                 
{'debug/num_tok_total': 2931.0, 'debug/num_tok_loss': 1711.0, 'debug/num_lat_total': 2931.0, 'debug/num_lat_loss': 1711.0, 'epoch': 7.71}

 77%|███████▋  | 563/730 [49:55<14:51,  5.34s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.32325708866119385, 'epoch': 7.71}

 77%|███████▋  | 563/730 [49:55<14:51,  5.34s/it]
                                                 
{'train/learning_rate_real': 3.277790432758862e-06, 'epoch': 7.71}

 77%|███████▋  | 563/730 [49:55<14:51,  5.34s/it]
                                                 
{'debug/num_tok_total': 3299.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 3299.0, 'debug/num_lat_loss': 1786.0, 'epoch': 7.71}

 77%|███████▋  | 563/730 [49:56<14:51,  5.34s/it]
                                                 
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.2574474811553955, 'epoch': 7.71}

 77%|███████▋  | 563/730 [49:56<14:51,  5.34s/it]
                                                 
{'train/learning_rate_real': 3.277790432758862e-06, 'epoch': 7.71}

 77%|███████▋  | 563/730 [49:56<14:51,  5.34s/it]
 77%|███████▋  | 564/730 [49:57<14:47,  5.35s/it]
                                                 
{'debug/num_tok_total': 2653.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2653.0, 'debug/num_lat_loss': 1780.0, 'epoch': 7.73}

 77%|███████▋  | 564/730 [49:58<14:47,  5.35s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.39371904730796814, 'epoch': 7.73}

 77%|███████▋  | 564/730 [49:58<14:47,  5.35s/it]
                                                 
{'train/learning_rate_real': 3.240439331093942e-06, 'epoch': 7.73}

 77%|███████▋  | 564/730 [49:58<14:47,  5.35s/it]
                                                 
{'debug/num_tok_total': 2686.0, 'debug/num_tok_loss': 1807.0, 'debug/num_lat_total': 2686.0, 'debug/num_lat_loss': 1807.0, 'epoch': 7.73}

 77%|███████▋  | 564/730 [49:59<14:47,  5.35s/it]
                                                 
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.4176764190196991, 'epoch': 7.73}

 77%|███████▋  | 564/730 [49:59<14:47,  5.35s/it]
                                                 
{'train/learning_rate_real': 3.240439331093942e-06, 'epoch': 7.73}

 77%|███████▋  | 564/730 [49:59<14:47,  5.35s/it]
                                                 
{'debug/num_tok_total': 2666.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2666.0, 'debug/num_lat_loss': 1792.0, 'epoch': 7.73}

 77%|███████▋  | 564/730 [50:00<14:47,  5.35s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.4258557856082916, 'epoch': 7.73}

 77%|███████▋  | 564/730 [50:00<14:47,  5.35s/it]
                                                 
{'train/learning_rate_real': 3.240439331093942e-06, 'epoch': 7.73}

 77%|███████▋  | 564/730 [50:00<14:47,  5.35s/it]
                                                 
{'debug/num_tok_total': 2893.0, 'debug/num_tok_loss': 1810.0, 'debug/num_lat_total': 2893.0, 'debug/num_lat_loss': 1810.0, 'epoch': 7.73}

 77%|███████▋  | 564/730 [50:02<14:47,  5.35s/it]
                                                 
{'train/ce_loss': 1.875, 'train/diffusion_loss': 0.340121865272522, 'epoch': 7.73}

 77%|███████▋  | 564/730 [50:02<14:47,  5.35s/it]
                                                 
{'train/learning_rate_real': 3.240439331093942e-06, 'epoch': 7.73}

 77%|███████▋  | 564/730 [50:02<14:47,  5.35s/it]
 77%|███████▋  | 565/730 [50:02<14:46,  5.37s/it]
                                                 
{'debug/num_tok_total': 2221.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2221.0, 'debug/num_lat_loss': 1777.0, 'epoch': 7.74}

 77%|███████▋  | 565/730 [50:03<14:46,  5.37s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.48557475209236145, 'epoch': 7.74}

 77%|███████▋  | 565/730 [50:03<14:46,  5.37s/it]
                                                 
{'train/learning_rate_real': 3.203270544637732e-06, 'epoch': 7.74}

 77%|███████▋  | 565/730 [50:03<14:46,  5.37s/it]
                                                 
{'debug/num_tok_total': 2425.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2425.0, 'debug/num_lat_loss': 1764.0, 'epoch': 7.74}

 77%|███████▋  | 565/730 [50:04<14:46,  5.37s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.4270690977573395, 'epoch': 7.74}

 77%|███████▋  | 565/730 [50:04<14:46,  5.37s/it]
                                                 
{'train/learning_rate_real': 3.203270544637732e-06, 'epoch': 7.74}

 77%|███████▋  | 565/730 [50:04<14:46,  5.37s/it]
                                                 
{'debug/num_tok_total': 2687.0, 'debug/num_tok_loss': 1804.0, 'debug/num_lat_total': 2687.0, 'debug/num_lat_loss': 1804.0, 'epoch': 7.74}

 77%|███████▋  | 565/730 [50:06<14:46,  5.37s/it]
                                                 
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.4182634949684143, 'epoch': 7.74}

 77%|███████▋  | 565/730 [50:06<14:46,  5.37s/it]
                                                 
{'train/learning_rate_real': 3.203270544637732e-06, 'epoch': 7.74}

 77%|███████▋  | 565/730 [50:06<14:46,  5.37s/it]
                                                 
{'debug/num_tok_total': 2012.0, 'debug/num_tok_loss': 1592.0, 'debug/num_lat_total': 2012.0, 'debug/num_lat_loss': 1592.0, 'epoch': 7.74}

 77%|███████▋  | 565/730 [50:07<14:46,  5.37s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.4573726952075958, 'epoch': 7.74}

 77%|███████▋  | 565/730 [50:07<14:46,  5.37s/it]
                                                 
{'train/learning_rate_real': 3.203270544637732e-06, 'epoch': 7.74}

 77%|███████▋  | 565/730 [50:07<14:46,  5.37s/it]
 78%|███████▊  | 566/730 [50:07<14:23,  5.26s/it]
                                                 
{'debug/num_tok_total': 2571.0, 'debug/num_tok_loss': 1728.0, 'debug/num_lat_total': 2571.0, 'debug/num_lat_loss': 1728.0, 'epoch': 7.75}

 78%|███████▊  | 566/730 [50:08<14:23,  5.26s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.4133783280849457, 'epoch': 7.75}

 78%|███████▊  | 566/730 [50:08<14:23,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.1662848052214033e-06, 'epoch': 7.75}

 78%|███████▊  | 566/730 [50:08<14:23,  5.26s/it]
                                                 
{'debug/num_tok_total': 3058.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 3058.0, 'debug/num_lat_loss': 1778.0, 'epoch': 7.75}

 78%|███████▊  | 566/730 [50:10<14:23,  5.26s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.3346664607524872, 'epoch': 7.75}

 78%|███████▊  | 566/730 [50:10<14:23,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.1662848052214033e-06, 'epoch': 7.75}

 78%|███████▊  | 566/730 [50:10<14:23,  5.26s/it]
                                                 
{'debug/num_tok_total': 2401.0, 'debug/num_tok_loss': 1755.0, 'debug/num_lat_total': 2401.0, 'debug/num_lat_loss': 1755.0, 'epoch': 7.75}

 78%|███████▊  | 566/730 [50:11<14:23,  5.26s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.43124109506607056, 'epoch': 7.75}

 78%|███████▊  | 566/730 [50:11<14:23,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.1662848052214033e-06, 'epoch': 7.75}

 78%|███████▊  | 566/730 [50:11<14:23,  5.26s/it]
                                                 
{'debug/num_tok_total': 2879.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 2879.0, 'debug/num_lat_loss': 1797.0, 'epoch': 7.75}

 78%|███████▊  | 566/730 [50:12<14:23,  5.26s/it]
                                                 
{'train/ce_loss': 2.15625, 'train/diffusion_loss': 0.3557266592979431, 'epoch': 7.75}

 78%|███████▊  | 566/730 [50:12<14:23,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.1662848052214033e-06, 'epoch': 7.75}

 78%|███████▊  | 566/730 [50:12<14:23,  5.26s/it]
 78%|███████▊  | 567/730 [50:13<14:34,  5.36s/it]
                                                 
{'debug/num_tok_total': 2863.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2863.0, 'debug/num_lat_loss': 1776.0, 'epoch': 7.77}

 78%|███████▊  | 567/730 [50:14<14:34,  5.36s/it]
                                                 
{'train/ce_loss': 1.8515625, 'train/diffusion_loss': 0.361041784286499, 'epoch': 7.77}

 78%|███████▊  | 567/730 [50:14<14:34,  5.36s/it]
                                                 
{'train/learning_rate_real': 3.129482841072033e-06, 'epoch': 7.77}

 78%|███████▊  | 567/730 [50:14<14:34,  5.36s/it]
                                                 
{'debug/num_tok_total': 2857.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2857.0, 'debug/num_lat_loss': 1782.0, 'epoch': 7.77}

 78%|███████▊  | 567/730 [50:15<14:34,  5.36s/it]
                                                 
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.37784963846206665, 'epoch': 7.77}

 78%|███████▊  | 567/730 [50:15<14:34,  5.36s/it]
                                                 
{'train/learning_rate_real': 3.129482841072033e-06, 'epoch': 7.77}

 78%|███████▊  | 567/730 [50:15<14:34,  5.36s/it]
                                                 
{'debug/num_tok_total': 2814.0, 'debug/num_tok_loss': 1757.0, 'debug/num_lat_total': 2814.0, 'debug/num_lat_loss': 1757.0, 'epoch': 7.77}

 78%|███████▊  | 567/730 [50:16<14:34,  5.36s/it]
                                                 
{'train/ce_loss': 1.671875, 'train/diffusion_loss': 0.33241772651672363, 'epoch': 7.77}

 78%|███████▊  | 567/730 [50:16<14:34,  5.36s/it]
                                                 
{'train/learning_rate_real': 3.129482841072033e-06, 'epoch': 7.77}

 78%|███████▊  | 567/730 [50:16<14:34,  5.36s/it]
                                                 
{'debug/num_tok_total': 2811.0, 'debug/num_tok_loss': 1747.0, 'debug/num_lat_total': 2811.0, 'debug/num_lat_loss': 1747.0, 'epoch': 7.77}

 78%|███████▊  | 567/730 [50:18<14:34,  5.36s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.3363363444805145, 'epoch': 7.77}

 78%|███████▊  | 567/730 [50:18<14:34,  5.36s/it]
                                                 
{'train/learning_rate_real': 3.129482841072033e-06, 'epoch': 7.77}

 78%|███████▊  | 567/730 [50:18<14:34,  5.36s/it]
 78%|███████▊  | 568/730 [50:18<14:31,  5.38s/it]
                                                 
{'debug/num_tok_total': 2235.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2235.0, 'debug/num_lat_loss': 1793.0, 'epoch': 7.78}

 78%|███████▊  | 568/730 [50:19<14:31,  5.38s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.4794442653656006, 'epoch': 7.78}

 78%|███████▊  | 568/730 [50:19<14:31,  5.38s/it]
                                                 
{'train/learning_rate_real': 3.0928653767982695e-06, 'epoch': 7.78}

 78%|███████▊  | 568/730 [50:19<14:31,  5.38s/it]
                                                 
{'debug/num_tok_total': 3116.0, 'debug/num_tok_loss': 1812.0, 'debug/num_lat_total': 3116.0, 'debug/num_lat_loss': 1812.0, 'epoch': 7.78}

 78%|███████▊  | 568/730 [50:21<14:31,  5.38s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.3123010993003845, 'epoch': 7.78}

 78%|███████▊  | 568/730 [50:21<14:31,  5.38s/it]
                                                 
{'train/learning_rate_real': 3.0928653767982695e-06, 'epoch': 7.78}

 78%|███████▊  | 568/730 [50:21<14:31,  5.38s/it]
                                                 
{'debug/num_tok_total': 2906.0, 'debug/num_tok_loss': 1810.0, 'debug/num_lat_total': 2906.0, 'debug/num_lat_loss': 1810.0, 'epoch': 7.78}

 78%|███████▊  | 568/730 [50:22<14:31,  5.38s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.37748152017593384, 'epoch': 7.78}

 78%|███████▊  | 568/730 [50:22<14:31,  5.38s/it]
                                                 
{'train/learning_rate_real': 3.0928653767982695e-06, 'epoch': 7.78}

 78%|███████▊  | 568/730 [50:22<14:31,  5.38s/it]
                                                 
{'debug/num_tok_total': 3061.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 3061.0, 'debug/num_lat_loss': 1777.0, 'epoch': 7.78}

 78%|███████▊  | 568/730 [50:23<14:31,  5.38s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.3027379512786865, 'epoch': 7.78}

 78%|███████▊  | 568/730 [50:23<14:31,  5.38s/it]
                                                 
{'train/learning_rate_real': 3.0928653767982695e-06, 'epoch': 7.78}

 78%|███████▊  | 568/730 [50:23<14:31,  5.38s/it]
 78%|███████▊  | 569/730 [50:24<14:34,  5.43s/it]
                                                 
{'debug/num_tok_total': 2613.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2613.0, 'debug/num_lat_loss': 1771.0, 'epoch': 7.79}

 78%|███████▊  | 569/730 [50:25<14:34,  5.43s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.40023860335350037, 'epoch': 7.79}

 78%|███████▊  | 569/730 [50:25<14:34,  5.43s/it]
                                                 
{'train/learning_rate_real': 3.056433133376077e-06, 'epoch': 7.79}

 78%|███████▊  | 569/730 [50:25<14:34,  5.43s/it]
                                                 
{'debug/num_tok_total': 2643.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2643.0, 'debug/num_lat_loss': 1780.0, 'epoch': 7.79}

 78%|███████▊  | 569/730 [50:26<14:34,  5.43s/it]
                                                 
{'train/ce_loss': 1.9375, 'train/diffusion_loss': 0.4276621639728546, 'epoch': 7.79}

 78%|███████▊  | 569/730 [50:26<14:34,  5.43s/it]
                                                 
{'train/learning_rate_real': 3.056433133376077e-06, 'epoch': 7.79}

 78%|███████▊  | 569/730 [50:26<14:34,  5.43s/it]
                                                 
{'debug/num_tok_total': 2683.0, 'debug/num_tok_loss': 1810.0, 'debug/num_lat_total': 2683.0, 'debug/num_lat_loss': 1810.0, 'epoch': 7.79}

 78%|███████▊  | 569/730 [50:27<14:34,  5.43s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.4126534163951874, 'epoch': 7.79}

 78%|███████▊  | 569/730 [50:27<14:34,  5.43s/it]
                                                 
{'train/learning_rate_real': 3.056433133376077e-06, 'epoch': 7.79}

 78%|███████▊  | 569/730 [50:27<14:34,  5.43s/it]
                                                 
{'debug/num_tok_total': 2603.0, 'debug/num_tok_loss': 1752.0, 'debug/num_lat_total': 2603.0, 'debug/num_lat_loss': 1752.0, 'epoch': 7.79}

 78%|███████▊  | 569/730 [50:29<14:34,  5.43s/it]
                                                 
{'train/ce_loss': 1.65625, 'train/diffusion_loss': 0.39523473381996155, 'epoch': 7.79}

 78%|███████▊  | 569/730 [50:29<14:34,  5.43s/it]
                                                 
{'train/learning_rate_real': 3.056433133376077e-06, 'epoch': 7.79}

 78%|███████▊  | 569/730 [50:29<14:34,  5.43s/it]03/16/2026 07:38:01 - INFO - __main__ - LoRA debug step 570: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 78%|███████▊  | 570/730 [50:29<14:28,  5.43s/it]
                                                 
{'loss': 2.4669, 'grad_norm': 1.2171447277069092, 'learning_rate': 3.056433133376077e-06, 'epoch': 7.81}

 78%|███████▊  | 570/730 [50:29<14:28,  5.43s/it]
                                                 
{'debug/num_tok_total': 2865.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2865.0, 'debug/num_lat_loss': 1784.0, 'epoch': 7.81}

 78%|███████▊  | 570/730 [50:30<14:28,  5.43s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.3865640163421631, 'epoch': 7.81}

 78%|███████▊  | 570/730 [50:30<14:28,  5.43s/it]
                                                 
{'train/learning_rate_real': 3.0201868281345207e-06, 'epoch': 7.81}

 78%|███████▊  | 570/730 [50:30<14:28,  5.43s/it]
                                                 
{'debug/num_tok_total': 2577.0, 'debug/num_tok_loss': 1737.0, 'debug/num_lat_total': 2577.0, 'debug/num_lat_loss': 1737.0, 'epoch': 7.81}

 78%|███████▊  | 570/730 [50:32<14:28,  5.43s/it]
                                                 
{'train/ce_loss': 1.9140625, 'train/diffusion_loss': 0.37444987893104553, 'epoch': 7.81}

 78%|███████▊  | 570/730 [50:32<14:28,  5.43s/it]
                                                 
{'train/learning_rate_real': 3.0201868281345207e-06, 'epoch': 7.81}

 78%|███████▊  | 570/730 [50:32<14:28,  5.43s/it]
                                                 
{'debug/num_tok_total': 2866.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2866.0, 'debug/num_lat_loss': 1779.0, 'epoch': 7.81}

 78%|███████▊  | 570/730 [50:33<14:28,  5.43s/it]
                                                 
{'train/ce_loss': 1.546875, 'train/diffusion_loss': 0.3480437994003296, 'epoch': 7.81}

 78%|███████▊  | 570/730 [50:33<14:28,  5.43s/it]
                                                 
{'train/learning_rate_real': 3.0201868281345207e-06, 'epoch': 7.81}

 78%|███████▊  | 570/730 [50:33<14:28,  5.43s/it]
                                                 
{'debug/num_tok_total': 2448.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2448.0, 'debug/num_lat_loss': 1785.0, 'epoch': 7.81}

 78%|███████▊  | 570/730 [50:34<14:28,  5.43s/it]
                                                 
{'train/ce_loss': 1.953125, 'train/diffusion_loss': 0.44967976212501526, 'epoch': 7.81}

 78%|███████▊  | 570/730 [50:34<14:28,  5.43s/it]
                                                 
{'train/learning_rate_real': 3.0201868281345207e-06, 'epoch': 7.81}

 78%|███████▊  | 570/730 [50:34<14:28,  5.43s/it]
 78%|███████▊  | 571/730 [50:35<14:19,  5.41s/it]
                                                 
{'debug/num_tok_total': 2173.0, 'debug/num_tok_loss': 1751.0, 'debug/num_lat_total': 2173.0, 'debug/num_lat_loss': 1751.0, 'epoch': 7.82}

 78%|███████▊  | 571/730 [50:36<14:19,  5.41s/it]
                                                 
{'train/ce_loss': 1.9296875, 'train/diffusion_loss': 0.5380390286445618, 'epoch': 7.82}

 78%|███████▊  | 571/730 [50:36<14:19,  5.41s/it]
                                                 
{'train/learning_rate_real': 2.984127174741666e-06, 'epoch': 7.82}

 78%|███████▊  | 571/730 [50:36<14:19,  5.41s/it]
                                                 
{'debug/num_tok_total': 2655.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2655.0, 'debug/num_lat_loss': 1788.0, 'epoch': 7.82}

 78%|███████▊  | 571/730 [50:37<14:19,  5.41s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.41260913014411926, 'epoch': 7.82}

 78%|███████▊  | 571/730 [50:37<14:19,  5.41s/it]
                                                 
{'train/learning_rate_real': 2.984127174741666e-06, 'epoch': 7.82}

 78%|███████▊  | 571/730 [50:37<14:19,  5.41s/it]
                                                 
{'debug/num_tok_total': 2866.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2866.0, 'debug/num_lat_loss': 1784.0, 'epoch': 7.82}

 78%|███████▊  | 571/730 [50:38<14:19,  5.41s/it]
                                                 
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.3388899564743042, 'epoch': 7.82}

 78%|███████▊  | 571/730 [50:38<14:19,  5.41s/it]
                                                 
{'train/learning_rate_real': 2.984127174741666e-06, 'epoch': 7.82}

 78%|███████▊  | 571/730 [50:38<14:19,  5.41s/it]
                                                 
{'debug/num_tok_total': 2399.0, 'debug/num_tok_loss': 1739.0, 'debug/num_lat_total': 2399.0, 'debug/num_lat_loss': 1739.0, 'epoch': 7.82}

 78%|███████▊  | 571/730 [50:39<14:19,  5.41s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.44461512565612793, 'epoch': 7.82}

 78%|███████▊  | 571/730 [50:39<14:19,  5.41s/it]
                                                 
{'train/learning_rate_real': 2.984127174741666e-06, 'epoch': 7.82}

 78%|███████▊  | 571/730 [50:39<14:19,  5.41s/it]
 78%|███████▊  | 572/730 [50:40<14:18,  5.43s/it]
                                                 
{'debug/num_tok_total': 2645.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2645.0, 'debug/num_lat_loss': 1783.0, 'epoch': 7.84}

 78%|███████▊  | 572/730 [50:41<14:18,  5.43s/it]
                                                 
{'train/ce_loss': 1.953125, 'train/diffusion_loss': 0.43589550256729126, 'epoch': 7.84}

 78%|███████▊  | 572/730 [50:41<14:18,  5.43s/it]
                                                 
{'train/learning_rate_real': 2.948254883190514e-06, 'epoch': 7.84}

 78%|███████▊  | 572/730 [50:41<14:18,  5.43s/it]
                                                 
{'debug/num_tok_total': 2421.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2421.0, 'debug/num_lat_loss': 1774.0, 'epoch': 7.84}

 78%|███████▊  | 572/730 [50:42<14:18,  5.43s/it]
                                                 
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.4272382855415344, 'epoch': 7.84}

 78%|███████▊  | 572/730 [50:42<14:18,  5.43s/it]
                                                 
{'train/learning_rate_real': 2.948254883190514e-06, 'epoch': 7.84}

 78%|███████▊  | 572/730 [50:42<14:18,  5.43s/it]
                                                 
{'debug/num_tok_total': 2795.0, 'debug/num_tok_loss': 1738.0, 'debug/num_lat_total': 2795.0, 'debug/num_lat_loss': 1738.0, 'epoch': 7.84}

 78%|███████▊  | 572/730 [50:44<14:18,  5.43s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.3267253637313843, 'epoch': 7.84}

 78%|███████▊  | 572/730 [50:44<14:18,  5.43s/it]
                                                 
{'train/learning_rate_real': 2.948254883190514e-06, 'epoch': 7.84}

 78%|███████▊  | 572/730 [50:44<14:18,  5.43s/it]
                                                 
{'debug/num_tok_total': 3029.0, 'debug/num_tok_loss': 1752.0, 'debug/num_lat_total': 3029.0, 'debug/num_lat_loss': 1752.0, 'epoch': 7.84}

 78%|███████▊  | 572/730 [50:45<14:18,  5.43s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.2967897653579712, 'epoch': 7.84}

 78%|███████▊  | 572/730 [50:45<14:18,  5.43s/it]
                                                 
{'train/learning_rate_real': 2.948254883190514e-06, 'epoch': 7.84}

 78%|███████▊  | 572/730 [50:45<14:18,  5.43s/it]
 78%|███████▊  | 573/730 [50:45<14:11,  5.43s/it]
                                                 
{'debug/num_tok_total': 2831.0, 'debug/num_tok_loss': 1733.0, 'debug/num_lat_total': 2831.0, 'debug/num_lat_loss': 1733.0, 'epoch': 7.85}

 78%|███████▊  | 573/730 [50:47<14:11,  5.43s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.34459465742111206, 'epoch': 7.85}

 78%|███████▊  | 573/730 [50:47<14:11,  5.43s/it]
                                                 
{'train/learning_rate_real': 2.912570659785016e-06, 'epoch': 7.85}

 78%|███████▊  | 573/730 [50:47<14:11,  5.43s/it]
                                                 
{'debug/num_tok_total': 2239.0, 'debug/num_tok_loss': 1816.0, 'debug/num_lat_total': 2239.0, 'debug/num_lat_loss': 1816.0, 'epoch': 7.85}

 78%|███████▊  | 573/730 [50:48<14:11,  5.43s/it]
                                                 
{'train/ce_loss': 1.9609375, 'train/diffusion_loss': 0.47724080085754395, 'epoch': 7.85}

 78%|███████▊  | 573/730 [50:48<14:11,  5.43s/it]
                                                 
{'train/learning_rate_real': 2.912570659785016e-06, 'epoch': 7.85}

 78%|███████▊  | 573/730 [50:48<14:11,  5.43s/it]
                                                 
{'debug/num_tok_total': 2875.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2875.0, 'debug/num_lat_loss': 1788.0, 'epoch': 7.85}

 78%|███████▊  | 573/730 [50:49<14:11,  5.43s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.349178671836853, 'epoch': 7.85}

 78%|███████▊  | 573/730 [50:49<14:11,  5.43s/it]
                                                 
{'train/learning_rate_real': 2.912570659785016e-06, 'epoch': 7.85}

 78%|███████▊  | 573/730 [50:49<14:11,  5.43s/it]
                                                 
{'debug/num_tok_total': 3338.0, 'debug/num_tok_loss': 1811.0, 'debug/num_lat_total': 3338.0, 'debug/num_lat_loss': 1811.0, 'epoch': 7.85}

 78%|███████▊  | 573/730 [50:51<14:11,  5.43s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.2758176028728485, 'epoch': 7.85}

 78%|███████▊  | 573/730 [50:51<14:11,  5.43s/it]
                                                 
{'train/learning_rate_real': 2.912570659785016e-06, 'epoch': 7.85}

 78%|███████▊  | 573/730 [50:51<14:11,  5.43s/it]
 79%|███████▊  | 574/730 [50:51<14:13,  5.47s/it]
                                                 
{'debug/num_tok_total': 2140.0, 'debug/num_tok_loss': 1702.0, 'debug/num_lat_total': 2140.0, 'debug/num_lat_loss': 1702.0, 'epoch': 7.86}

 79%|███████▊  | 574/730 [50:52<14:13,  5.47s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.4596053957939148, 'epoch': 7.86}

 79%|███████▊  | 574/730 [50:52<14:13,  5.47s/it]
                                                 
{'train/learning_rate_real': 2.87707520712617e-06, 'epoch': 7.86}

 79%|███████▊  | 574/730 [50:52<14:13,  5.47s/it]
                                                 
{'debug/num_tok_total': 2209.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2209.0, 'debug/num_lat_loss': 1776.0, 'epoch': 7.86}

 79%|███████▊  | 574/730 [50:53<14:13,  5.47s/it]
                                                 
{'train/ce_loss': 1.9140625, 'train/diffusion_loss': 0.49156123399734497, 'epoch': 7.86}

 79%|███████▊  | 574/730 [50:53<14:13,  5.47s/it]
                                                 
{'train/learning_rate_real': 2.87707520712617e-06, 'epoch': 7.86}

 79%|███████▊  | 574/730 [50:53<14:13,  5.47s/it]
                                                 
{'debug/num_tok_total': 3285.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 3285.0, 'debug/num_lat_loss': 1780.0, 'epoch': 7.86}

 79%|███████▊  | 574/730 [50:55<14:13,  5.47s/it]
                                                 
{'train/ce_loss': 1.6328125, 'train/diffusion_loss': 0.2816939353942871, 'epoch': 7.86}

 79%|███████▊  | 574/730 [50:55<14:13,  5.47s/it]
                                                 
{'train/learning_rate_real': 2.87707520712617e-06, 'epoch': 7.86}

 79%|███████▊  | 574/730 [50:55<14:13,  5.47s/it]
                                                 
{'debug/num_tok_total': 2654.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2654.0, 'debug/num_lat_loss': 1786.0, 'epoch': 7.86}

 79%|███████▊  | 574/730 [50:56<14:13,  5.47s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.40930789709091187, 'epoch': 7.86}

 79%|███████▊  | 574/730 [50:56<14:13,  5.47s/it]
                                                 
{'train/learning_rate_real': 2.87707520712617e-06, 'epoch': 7.86}

 79%|███████▊  | 574/730 [50:56<14:13,  5.47s/it]
 79%|███████▉  | 575/730 [50:56<14:00,  5.42s/it]
                                                 
{'debug/num_tok_total': 2620.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2620.0, 'debug/num_lat_loss': 1775.0, 'epoch': 7.88}

 79%|███████▉  | 575/730 [50:57<14:00,  5.42s/it]
                                                 
{'train/ce_loss': 1.6171875, 'train/diffusion_loss': 0.4136548638343811, 'epoch': 7.88}

 79%|███████▉  | 575/730 [50:57<14:00,  5.42s/it]
                                                 
{'train/learning_rate_real': 2.8417692240982133e-06, 'epoch': 7.88}

 79%|███████▉  | 575/730 [50:57<14:00,  5.42s/it]
                                                 
{'debug/num_tok_total': 2442.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2442.0, 'debug/num_lat_loss': 1786.0, 'epoch': 7.88}

 79%|███████▉  | 575/730 [50:59<14:00,  5.42s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.4371531903743744, 'epoch': 7.88}

 79%|███████▉  | 575/730 [50:59<14:00,  5.42s/it]
                                                 
{'train/learning_rate_real': 2.8417692240982133e-06, 'epoch': 7.88}

 79%|███████▉  | 575/730 [50:59<14:00,  5.42s/it]
                                                 
{'debug/num_tok_total': 2401.0, 'debug/num_tok_loss': 1750.0, 'debug/num_lat_total': 2401.0, 'debug/num_lat_loss': 1750.0, 'epoch': 7.88}

 79%|███████▉  | 575/730 [51:00<14:00,  5.42s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.4163949191570282, 'epoch': 7.88}

 79%|███████▉  | 575/730 [51:00<14:00,  5.42s/it]
                                                 
{'train/learning_rate_real': 2.8417692240982133e-06, 'epoch': 7.88}

 79%|███████▉  | 575/730 [51:00<14:00,  5.42s/it]
                                                 
{'debug/num_tok_total': 2643.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2643.0, 'debug/num_lat_loss': 1779.0, 'epoch': 7.88}

 79%|███████▉  | 575/730 [51:01<14:00,  5.42s/it]
                                                 
{'train/ce_loss': 1.65625, 'train/diffusion_loss': 0.38238945603370667, 'epoch': 7.88}

 79%|███████▉  | 575/730 [51:01<14:00,  5.42s/it]
                                                 
{'train/learning_rate_real': 2.8417692240982133e-06, 'epoch': 7.88}

 79%|███████▉  | 575/730 [51:01<14:00,  5.42s/it]
 79%|███████▉  | 576/730 [51:02<13:48,  5.38s/it]
                                                 
{'debug/num_tok_total': 2674.0, 'debug/num_tok_loss': 1803.0, 'debug/num_lat_total': 2674.0, 'debug/num_lat_loss': 1803.0, 'epoch': 7.89}

 79%|███████▉  | 576/730 [51:03<13:48,  5.38s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.40002450346946716, 'epoch': 7.89}

 79%|███████▉  | 576/730 [51:03<13:48,  5.38s/it]
                                                 
{'train/learning_rate_real': 2.806653405854817e-06, 'epoch': 7.89}

 79%|███████▉  | 576/730 [51:03<13:48,  5.38s/it]
                                                 
{'debug/num_tok_total': 3273.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 3273.0, 'debug/num_lat_loss': 1773.0, 'epoch': 7.89}

 79%|███████▉  | 576/730 [51:04<13:48,  5.38s/it]
                                                 
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.2618025243282318, 'epoch': 7.89}

 79%|███████▉  | 576/730 [51:04<13:48,  5.38s/it]
                                                 
{'train/learning_rate_real': 2.806653405854817e-06, 'epoch': 7.89}

 79%|███████▉  | 576/730 [51:04<13:48,  5.38s/it]
                                                 
{'debug/num_tok_total': 3126.0, 'debug/num_tok_loss': 1813.0, 'debug/num_lat_total': 3126.0, 'debug/num_lat_loss': 1813.0, 'epoch': 7.89}

 79%|███████▉  | 576/730 [51:06<13:48,  5.38s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.3356093168258667, 'epoch': 7.89}

 79%|███████▉  | 576/730 [51:06<13:48,  5.38s/it]
                                                 
{'train/learning_rate_real': 2.806653405854817e-06, 'epoch': 7.89}

 79%|███████▉  | 576/730 [51:06<13:48,  5.38s/it]
                                                 
{'debug/num_tok_total': 3282.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 3282.0, 'debug/num_lat_loss': 1782.0, 'epoch': 7.89}

 79%|███████▉  | 576/730 [51:07<13:48,  5.38s/it]
                                                 
{'train/ce_loss': 1.6328125, 'train/diffusion_loss': 0.2568773627281189, 'epoch': 7.89}

 79%|███████▉  | 576/730 [51:07<13:48,  5.38s/it]
                                                 
{'train/learning_rate_real': 2.806653405854817e-06, 'epoch': 7.89}

 79%|███████▉  | 576/730 [51:07<13:48,  5.38s/it]
 79%|███████▉  | 577/730 [51:07<14:07,  5.54s/it]
                                                 
{'debug/num_tok_total': 2625.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2625.0, 'debug/num_lat_loss': 1772.0, 'epoch': 7.9}

 79%|███████▉  | 577/730 [51:09<14:07,  5.54s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.4047796428203583, 'epoch': 7.9}

 79%|███████▉  | 577/730 [51:09<14:07,  5.54s/it]
                                                 
{'train/learning_rate_real': 2.771728443805427e-06, 'epoch': 7.9}

 79%|███████▉  | 577/730 [51:09<14:07,  5.54s/it]
                                                 
{'debug/num_tok_total': 2488.0, 'debug/num_tok_loss': 1599.0, 'debug/num_lat_total': 2488.0, 'debug/num_lat_loss': 1599.0, 'epoch': 7.9}

 79%|███████▉  | 577/730 [51:10<14:07,  5.54s/it]
                                                 
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.3945554494857788, 'epoch': 7.9}

 79%|███████▉  | 577/730 [51:10<14:07,  5.54s/it]
                                                 
{'train/learning_rate_real': 2.771728443805427e-06, 'epoch': 7.9}

 79%|███████▉  | 577/730 [51:10<14:07,  5.54s/it]
                                                 
{'debug/num_tok_total': 2604.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2604.0, 'debug/num_lat_loss': 1773.0, 'epoch': 7.9}

 79%|███████▉  | 577/730 [51:11<14:07,  5.54s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.4065757095813751, 'epoch': 7.9}

 79%|███████▉  | 577/730 [51:11<14:07,  5.54s/it]
                                                 
{'train/learning_rate_real': 2.771728443805427e-06, 'epoch': 7.9}

 79%|███████▉  | 577/730 [51:11<14:07,  5.54s/it]
                                                 
{'debug/num_tok_total': 2448.0, 'debug/num_tok_loss': 1596.0, 'debug/num_lat_total': 2448.0, 'debug/num_lat_loss': 1596.0, 'epoch': 7.9}

 79%|███████▉  | 577/730 [51:12<14:07,  5.54s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.3715749979019165, 'epoch': 7.9}

 79%|███████▉  | 577/730 [51:12<14:07,  5.54s/it]
                                                 
{'train/learning_rate_real': 2.771728443805427e-06, 'epoch': 7.9}

 79%|███████▉  | 577/730 [51:12<14:07,  5.54s/it]
 79%|███████▉  | 578/730 [51:13<13:57,  5.51s/it]
                                                 
{'debug/num_tok_total': 2683.0, 'debug/num_tok_loss': 1807.0, 'debug/num_lat_total': 2683.0, 'debug/num_lat_loss': 1807.0, 'epoch': 7.92}

 79%|███████▉  | 578/730 [51:14<13:57,  5.51s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.39397451281547546, 'epoch': 7.92}

 79%|███████▉  | 578/730 [51:14<13:57,  5.51s/it]
                                                 
{'train/learning_rate_real': 2.736995025601649e-06, 'epoch': 7.92}

 79%|███████▉  | 578/730 [51:14<13:57,  5.51s/it]
                                                 
{'debug/num_tok_total': 2402.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2402.0, 'debug/num_lat_loss': 1763.0, 'epoch': 7.92}

 79%|███████▉  | 578/730 [51:15<13:57,  5.51s/it]
                                                 
{'train/ce_loss': 1.671875, 'train/diffusion_loss': 0.4299086630344391, 'epoch': 7.92}

 79%|███████▉  | 578/730 [51:15<13:57,  5.51s/it]
                                                 
{'train/learning_rate_real': 2.736995025601649e-06, 'epoch': 7.92}

 79%|███████▉  | 578/730 [51:15<13:57,  5.51s/it]
                                                 
{'debug/num_tok_total': 2814.0, 'debug/num_tok_loss': 1760.0, 'debug/num_lat_total': 2814.0, 'debug/num_lat_loss': 1760.0, 'epoch': 7.92}

 79%|███████▉  | 578/730 [51:17<13:57,  5.51s/it]
                                                 
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.3570418059825897, 'epoch': 7.92}

 79%|███████▉  | 578/730 [51:17<13:57,  5.51s/it]
                                                 
{'train/learning_rate_real': 2.736995025601649e-06, 'epoch': 7.92}

 79%|███████▉  | 578/730 [51:17<13:57,  5.51s/it]
                                                 
{'debug/num_tok_total': 2501.0, 'debug/num_tok_loss': 1616.0, 'debug/num_lat_total': 2501.0, 'debug/num_lat_loss': 1616.0, 'epoch': 7.92}

 79%|███████▉  | 578/730 [51:18<13:57,  5.51s/it]
                                                 
{'train/ce_loss': 1.6484375, 'train/diffusion_loss': 0.3783414363861084, 'epoch': 7.92}

 79%|███████▉  | 578/730 [51:18<13:57,  5.51s/it]
                                                 
{'train/learning_rate_real': 2.736995025601649e-06, 'epoch': 7.92}

 79%|███████▉  | 578/730 [51:18<13:57,  5.51s/it]
 79%|███████▉  | 579/730 [51:18<13:40,  5.44s/it]
                                                 
{'debug/num_tok_total': 2116.0, 'debug/num_tok_loss': 1624.0, 'debug/num_lat_total': 2116.0, 'debug/num_lat_loss': 1624.0, 'epoch': 7.93}

 79%|███████▉  | 579/730 [51:19<13:40,  5.44s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.4777570068836212, 'epoch': 7.93}

 79%|███████▉  | 579/730 [51:19<13:40,  5.44s/it]
                                                 
{'train/learning_rate_real': 2.7024538351237057e-06, 'epoch': 7.93}

 79%|███████▉  | 579/730 [51:19<13:40,  5.44s/it]
                                                 
{'debug/num_tok_total': 2857.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2857.0, 'debug/num_lat_loss': 1781.0, 'epoch': 7.93}

 79%|███████▉  | 579/730 [51:21<13:40,  5.44s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.3636435568332672, 'epoch': 7.93}

 79%|███████▉  | 579/730 [51:21<13:40,  5.44s/it]
                                                 
{'train/learning_rate_real': 2.7024538351237057e-06, 'epoch': 7.93}

 79%|███████▉  | 579/730 [51:21<13:40,  5.44s/it]
                                                 
{'debug/num_tok_total': 2223.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2223.0, 'debug/num_lat_loss': 1799.0, 'epoch': 7.93}

 79%|███████▉  | 579/730 [51:22<13:40,  5.44s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.4607008099555969, 'epoch': 7.93}

 79%|███████▉  | 579/730 [51:22<13:40,  5.44s/it]
                                                 
{'train/learning_rate_real': 2.7024538351237057e-06, 'epoch': 7.93}

 79%|███████▉  | 579/730 [51:22<13:40,  5.44s/it]
                                                 
{'debug/num_tok_total': 2629.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2629.0, 'debug/num_lat_loss': 1775.0, 'epoch': 7.93}

 79%|███████▉  | 579/730 [51:23<13:40,  5.44s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.38739609718322754, 'epoch': 7.93}

 79%|███████▉  | 579/730 [51:23<13:40,  5.44s/it]
                                                 
{'train/learning_rate_real': 2.7024538351237057e-06, 'epoch': 7.93}

 79%|███████▉  | 579/730 [51:23<13:40,  5.44s/it]03/16/2026 07:38:55 - INFO - __main__ - LoRA debug step 580: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 79%|███████▉  | 580/730 [51:23<13:28,  5.39s/it]
                                                 
{'loss': 2.4662, 'grad_norm': 1.2165049314498901, 'learning_rate': 2.7024538351237057e-06, 'epoch': 7.95}

 79%|███████▉  | 580/730 [51:23<13:28,  5.39s/it]
                                                 
{'debug/num_tok_total': 2896.0, 'debug/num_tok_loss': 1808.0, 'debug/num_lat_total': 2896.0, 'debug/num_lat_loss': 1808.0, 'epoch': 7.95}

 79%|███████▉  | 580/730 [51:25<13:28,  5.39s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.3835728168487549, 'epoch': 7.95}

 79%|███████▉  | 580/730 [51:25<13:28,  5.39s/it]
                                                 
{'train/learning_rate_real': 2.6681055524669694e-06, 'epoch': 7.95}

 79%|███████▉  | 580/730 [51:25<13:28,  5.39s/it]
                                                 
{'debug/num_tok_total': 2662.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2662.0, 'debug/num_lat_loss': 1784.0, 'epoch': 7.95}

 79%|███████▉  | 580/730 [51:26<13:28,  5.39s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.37859082221984863, 'epoch': 7.95}

 79%|███████▉  | 580/730 [51:26<13:28,  5.39s/it]
                                                 
{'train/learning_rate_real': 2.6681055524669694e-06, 'epoch': 7.95}

 79%|███████▉  | 580/730 [51:26<13:28,  5.39s/it]
                                                 
{'debug/num_tok_total': 2471.0, 'debug/num_tok_loss': 1807.0, 'debug/num_lat_total': 2471.0, 'debug/num_lat_loss': 1807.0, 'epoch': 7.95}

 79%|███████▉  | 580/730 [51:27<13:28,  5.39s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.43487682938575745, 'epoch': 7.95}

 79%|███████▉  | 580/730 [51:27<13:28,  5.39s/it]
                                                 
{'train/learning_rate_real': 2.6681055524669694e-06, 'epoch': 7.95}

 79%|███████▉  | 580/730 [51:27<13:28,  5.39s/it]
                                                 
{'debug/num_tok_total': 2659.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 2659.0, 'debug/num_lat_loss': 1797.0, 'epoch': 7.95}

 79%|███████▉  | 580/730 [51:28<13:28,  5.39s/it]
                                                 
{'train/ce_loss': 1.953125, 'train/diffusion_loss': 0.39776819944381714, 'epoch': 7.95}

 79%|███████▉  | 580/730 [51:28<13:28,  5.39s/it]
                                                 
{'train/learning_rate_real': 2.6681055524669694e-06, 'epoch': 7.95}

 79%|███████▉  | 580/730 [51:28<13:28,  5.39s/it]
 80%|███████▉  | 581/730 [51:29<13:15,  5.34s/it]
                                                 
{'debug/num_tok_total': 2644.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2644.0, 'debug/num_lat_loss': 1778.0, 'epoch': 7.96}

 80%|███████▉  | 581/730 [51:30<13:15,  5.34s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.40125375986099243, 'epoch': 7.96}

 80%|███████▉  | 581/730 [51:30<13:15,  5.34s/it]
                                                 
{'train/learning_rate_real': 2.6339508539285696e-06, 'epoch': 7.96}

 80%|███████▉  | 581/730 [51:30<13:15,  5.34s/it]
                                                 
{'debug/num_tok_total': 2245.0, 'debug/num_tok_loss': 1610.0, 'debug/num_lat_total': 2245.0, 'debug/num_lat_loss': 1610.0, 'epoch': 7.96}

 80%|███████▉  | 581/730 [51:31<13:15,  5.34s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.4165666103363037, 'epoch': 7.96}

 80%|███████▉  | 581/730 [51:31<13:15,  5.34s/it]
                                                 
{'train/learning_rate_real': 2.6339508539285696e-06, 'epoch': 7.96}

 80%|███████▉  | 581/730 [51:31<13:15,  5.34s/it]
                                                 
{'debug/num_tok_total': 2907.0, 'debug/num_tok_loss': 1800.0, 'debug/num_lat_total': 2907.0, 'debug/num_lat_loss': 1800.0, 'epoch': 7.96}

 80%|███████▉  | 581/730 [51:32<13:15,  5.34s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.3482736647129059, 'epoch': 7.96}

 80%|███████▉  | 581/730 [51:32<13:15,  5.34s/it]
                                                 
{'train/learning_rate_real': 2.6339508539285696e-06, 'epoch': 7.96}

 80%|███████▉  | 581/730 [51:32<13:15,  5.34s/it]
                                                 
{'debug/num_tok_total': 2629.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2629.0, 'debug/num_lat_loss': 1779.0, 'epoch': 7.96}

 80%|███████▉  | 581/730 [51:34<13:15,  5.34s/it]
                                                 
{'train/ce_loss': 1.8828125, 'train/diffusion_loss': 0.4021114110946655, 'epoch': 7.96}

 80%|███████▉  | 581/730 [51:34<13:15,  5.34s/it]
                                                 
{'train/learning_rate_real': 2.6339508539285696e-06, 'epoch': 7.96}

 80%|███████▉  | 581/730 [51:34<13:15,  5.34s/it]
 80%|███████▉  | 582/730 [51:34<13:05,  5.31s/it]
                                                 
{'debug/num_tok_total': 2632.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2632.0, 'debug/num_lat_loss': 1781.0, 'epoch': 7.97}

 80%|███████▉  | 582/730 [51:35<13:05,  5.31s/it]
                                                 
{'train/ce_loss': 1.671875, 'train/diffusion_loss': 0.42549845576286316, 'epoch': 7.97}

 80%|███████▉  | 582/730 [51:35<13:05,  5.31s/it]
                                                 
{'train/learning_rate_real': 2.5999904119940784e-06, 'epoch': 7.97}

 80%|███████▉  | 582/730 [51:35<13:05,  5.31s/it]
                                                 
{'debug/num_tok_total': 2925.0, 'debug/num_tok_loss': 1812.0, 'debug/num_lat_total': 2925.0, 'debug/num_lat_loss': 1812.0, 'epoch': 7.97}

 80%|███████▉  | 582/730 [51:36<13:05,  5.31s/it]
                                                 
{'train/ce_loss': 1.6640625, 'train/diffusion_loss': 0.3622713088989258, 'epoch': 7.97}

 80%|███████▉  | 582/730 [51:36<13:05,  5.31s/it]
                                                 
{'train/learning_rate_real': 2.5999904119940784e-06, 'epoch': 7.97}

 80%|███████▉  | 582/730 [51:36<13:05,  5.31s/it]
                                                 
{'debug/num_tok_total': 2904.0, 'debug/num_tok_loss': 1804.0, 'debug/num_lat_total': 2904.0, 'debug/num_lat_loss': 1804.0, 'epoch': 7.97}

 80%|███████▉  | 582/730 [51:38<13:05,  5.31s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.32749444246292114, 'epoch': 7.97}

 80%|███████▉  | 582/730 [51:38<13:05,  5.31s/it]
                                                 
{'train/learning_rate_real': 2.5999904119940784e-06, 'epoch': 7.97}

 80%|███████▉  | 582/730 [51:38<13:05,  5.31s/it]
                                                 
{'debug/num_tok_total': 2849.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2849.0, 'debug/num_lat_loss': 1782.0, 'epoch': 7.97}

 80%|███████▉  | 582/730 [51:39<13:05,  5.31s/it]
                                                 
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.3703305125236511, 'epoch': 7.97}

 80%|███████▉  | 582/730 [51:39<13:05,  5.31s/it]
                                                 
{'train/learning_rate_real': 2.5999904119940784e-06, 'epoch': 7.97}

 80%|███████▉  | 582/730 [51:39<13:05,  5.31s/it]
 80%|███████▉  | 583/730 [51:39<13:07,  5.36s/it]
                                                 
{'debug/num_tok_total': 2435.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2435.0, 'debug/num_lat_loss': 1775.0, 'epoch': 7.99}

 80%|███████▉  | 583/730 [51:40<13:07,  5.36s/it]
                                                 
{'train/ce_loss': 1.9296875, 'train/diffusion_loss': 0.45139092206954956, 'epoch': 7.99}

 80%|███████▉  | 583/730 [51:40<13:07,  5.36s/it]
                                                 
{'train/learning_rate_real': 2.5662248953242805e-06, 'epoch': 7.99}

 80%|███████▉  | 583/730 [51:40<13:07,  5.36s/it]
                                                 
{'debug/num_tok_total': 2417.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2417.0, 'debug/num_lat_loss': 1778.0, 'epoch': 7.99}

 80%|███████▉  | 583/730 [51:42<13:07,  5.36s/it]
                                                 
{'train/ce_loss': 1.8984375, 'train/diffusion_loss': 0.4718853235244751, 'epoch': 7.99}

 80%|███████▉  | 583/730 [51:42<13:07,  5.36s/it]
                                                 
{'train/learning_rate_real': 2.5662248953242805e-06, 'epoch': 7.99}

 80%|███████▉  | 583/730 [51:42<13:07,  5.36s/it]
                                                 
{'debug/num_tok_total': 2657.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 2657.0, 'debug/num_lat_loss': 1797.0, 'epoch': 7.99}

 80%|███████▉  | 583/730 [51:43<13:07,  5.36s/it]
                                                 
{'train/ce_loss': 1.8984375, 'train/diffusion_loss': 0.408611536026001, 'epoch': 7.99}

 80%|███████▉  | 583/730 [51:43<13:07,  5.36s/it]
                                                 
{'train/learning_rate_real': 2.5662248953242805e-06, 'epoch': 7.99}

 80%|███████▉  | 583/730 [51:43<13:07,  5.36s/it]
                                                 
{'debug/num_tok_total': 722.0, 'debug/num_tok_loss': 369.0, 'debug/num_lat_total': 722.0, 'debug/num_lat_loss': 369.0, 'epoch': 7.99}

 80%|███████▉  | 583/730 [51:43<13:07,  5.36s/it]
                                                 
{'train/ce_loss': 1.4921875, 'train/diffusion_loss': 0.21506242454051971, 'epoch': 7.99}

 80%|███████▉  | 583/730 [51:43<13:07,  5.36s/it]
                                                 
{'train/learning_rate_real': 2.5662248953242805e-06, 'epoch': 7.99}

 80%|███████▉  | 583/730 [51:43<13:07,  5.36s/it]
 80%|████████  | 584/730 [51:44<12:13,  5.02s/it]
                                                 
{'debug/num_tok_total': 2451.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2451.0, 'debug/num_lat_loss': 1791.0, 'epoch': 8.0}

 80%|████████  | 584/730 [51:45<12:13,  5.02s/it]
                                                 
{'train/ce_loss': 1.890625, 'train/diffusion_loss': 0.42565661668777466, 'epoch': 8.0}

 80%|████████  | 584/730 [51:45<12:13,  5.02s/it]
                                                 
{'train/learning_rate_real': 2.532654968741996e-06, 'epoch': 8.0}

 80%|████████  | 584/730 [51:45<12:13,  5.02s/it]
                                                 
{'debug/num_tok_total': 2202.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2202.0, 'debug/num_lat_loss': 1771.0, 'epoch': 8.0}

 80%|████████  | 584/730 [51:46<12:13,  5.02s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.5132241249084473, 'epoch': 8.0}

 80%|████████  | 584/730 [51:46<12:13,  5.02s/it]
                                                 
{'train/learning_rate_real': 2.532654968741996e-06, 'epoch': 8.0}

 80%|████████  | 584/730 [51:46<12:13,  5.02s/it]
                                                 
{'debug/num_tok_total': 2396.0, 'debug/num_tok_loss': 1760.0, 'debug/num_lat_total': 2396.0, 'debug/num_lat_loss': 1760.0, 'epoch': 8.0}

 80%|████████  | 584/730 [51:47<12:13,  5.02s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.44498375058174133, 'epoch': 8.0}

 80%|████████  | 584/730 [51:47<12:13,  5.02s/it]
                                                 
{'train/learning_rate_real': 2.532654968741996e-06, 'epoch': 8.0}

 80%|████████  | 584/730 [51:47<12:13,  5.02s/it]
                                                 
{'debug/num_tok_total': 2527.0, 'debug/num_tok_loss': 1741.0, 'debug/num_lat_total': 2527.0, 'debug/num_lat_loss': 1741.0, 'epoch': 8.0}

 80%|████████  | 584/730 [51:48<12:13,  5.02s/it]
                                                 
{'train/ce_loss': 1.8515625, 'train/diffusion_loss': 0.4109252691268921, 'epoch': 8.0}

 80%|████████  | 584/730 [51:48<12:13,  5.02s/it]
                                                 
{'train/learning_rate_real': 2.532654968741996e-06, 'epoch': 8.0}

 80%|████████  | 584/730 [51:48<12:13,  5.02s/it]
 80%|████████  | 585/730 [51:49<12:12,  5.05s/it]
                                                 
{'debug/num_tok_total': 2445.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2445.0, 'debug/num_lat_loss': 1795.0, 'epoch': 8.01}

 80%|████████  | 585/730 [51:50<12:12,  5.05s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.4403439462184906, 'epoch': 8.01}

 80%|████████  | 585/730 [51:50<12:12,  5.05s/it]
                                                 
{'train/learning_rate_real': 2.4992812932189857e-06, 'epoch': 8.01}

 80%|████████  | 585/730 [51:50<12:12,  5.05s/it]
                                                 
{'debug/num_tok_total': 2637.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2637.0, 'debug/num_lat_loss': 1775.0, 'epoch': 8.01}

 80%|████████  | 585/730 [51:51<12:12,  5.05s/it]
                                                 
{'train/ce_loss': 1.671875, 'train/diffusion_loss': 0.4128667712211609, 'epoch': 8.01}

 80%|████████  | 585/730 [51:51<12:12,  5.05s/it]
                                                 
{'train/learning_rate_real': 2.4992812932189857e-06, 'epoch': 8.01}

 80%|████████  | 585/730 [51:51<12:12,  5.05s/it]
                                                 
{'debug/num_tok_total': 3102.0, 'debug/num_tok_loss': 1808.0, 'debug/num_lat_total': 3102.0, 'debug/num_lat_loss': 1808.0, 'epoch': 8.01}

 80%|████████  | 585/730 [51:52<12:12,  5.05s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.3025169372558594, 'epoch': 8.01}

 80%|████████  | 585/730 [51:53<12:12,  5.05s/it]
                                                 
{'train/learning_rate_real': 2.4992812932189857e-06, 'epoch': 8.01}

 80%|████████  | 585/730 [51:53<12:12,  5.05s/it]
                                                 
{'debug/num_tok_total': 2625.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2625.0, 'debug/num_lat_loss': 1775.0, 'epoch': 8.01}

 80%|████████  | 585/730 [51:54<12:12,  5.05s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.36739349365234375, 'epoch': 8.01}

 80%|████████  | 585/730 [51:54<12:12,  5.05s/it]
                                                 
{'train/learning_rate_real': 2.4992812932189857e-06, 'epoch': 8.01}

 80%|████████  | 585/730 [51:54<12:12,  5.05s/it]
 80%|████████  | 586/730 [51:54<12:21,  5.15s/it]
                                                 
{'debug/num_tok_total': 2651.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2651.0, 'debug/num_lat_loss': 1791.0, 'epoch': 8.03}

 80%|████████  | 586/730 [51:55<12:21,  5.15s/it]
                                                 
{'train/ce_loss': 1.6640625, 'train/diffusion_loss': 0.3769001364707947, 'epoch': 8.03}

 80%|████████  | 586/730 [51:55<12:21,  5.15s/it]
                                                 
{'train/learning_rate_real': 2.466104525862957e-06, 'epoch': 8.03}

 80%|████████  | 586/730 [51:55<12:21,  5.15s/it]
                                                 
{'debug/num_tok_total': 1951.0, 'debug/num_tok_loss': 1641.0, 'debug/num_lat_total': 1951.0, 'debug/num_lat_loss': 1641.0, 'epoch': 8.03}

 80%|████████  | 586/730 [51:56<12:21,  5.15s/it]
                                                 
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.5023387670516968, 'epoch': 8.03}

 80%|████████  | 586/730 [51:56<12:21,  5.15s/it]
                                                 
{'train/learning_rate_real': 2.466104525862957e-06, 'epoch': 8.03}

 80%|████████  | 586/730 [51:56<12:21,  5.15s/it]
                                                 
{'debug/num_tok_total': 2020.0, 'debug/num_tok_loss': 1798.0, 'debug/num_lat_total': 2020.0, 'debug/num_lat_loss': 1798.0, 'epoch': 8.03}

 80%|████████  | 586/730 [51:57<12:21,  5.15s/it]
                                                 
{'train/ce_loss': 1.875, 'train/diffusion_loss': 0.5088688731193542, 'epoch': 8.03}

 80%|████████  | 586/730 [51:57<12:21,  5.15s/it]
                                                 
{'train/learning_rate_real': 2.466104525862957e-06, 'epoch': 8.03}

 80%|████████  | 586/730 [51:57<12:21,  5.15s/it]
                                                 
{'debug/num_tok_total': 2673.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2673.0, 'debug/num_lat_loss': 1799.0, 'epoch': 8.03}

 80%|████████  | 586/730 [51:59<12:21,  5.15s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.3828412890434265, 'epoch': 8.03}

 80%|████████  | 586/730 [51:59<12:21,  5.15s/it]
                                                 
{'train/learning_rate_real': 2.466104525862957e-06, 'epoch': 8.03}

 80%|████████  | 586/730 [51:59<12:21,  5.15s/it]
 80%|████████  | 587/730 [51:59<12:04,  5.07s/it]
                                                 
{'debug/num_tok_total': 2667.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2667.0, 'debug/num_lat_loss': 1795.0, 'epoch': 8.04}

 80%|████████  | 587/730 [52:00<12:04,  5.07s/it]
                                                 
{'train/ce_loss': 1.90625, 'train/diffusion_loss': 0.3972609341144562, 'epoch': 8.04}

 80%|████████  | 587/730 [52:00<12:04,  5.07s/it]
                                                 
{'train/learning_rate_real': 2.4331253199045974e-06, 'epoch': 8.04}

 80%|████████  | 587/730 [52:00<12:04,  5.07s/it]
                                                 
{'debug/num_tok_total': 2167.0, 'debug/num_tok_loss': 1723.0, 'debug/num_lat_total': 2167.0, 'debug/num_lat_loss': 1723.0, 'epoch': 8.04}

 80%|████████  | 587/730 [52:01<12:04,  5.07s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.46457260847091675, 'epoch': 8.04}

 80%|████████  | 587/730 [52:01<12:04,  5.07s/it]
                                                 
{'train/learning_rate_real': 2.4331253199045974e-06, 'epoch': 8.04}

 80%|████████  | 587/730 [52:01<12:04,  5.07s/it]
                                                 
{'debug/num_tok_total': 2678.0, 'debug/num_tok_loss': 1800.0, 'debug/num_lat_total': 2678.0, 'debug/num_lat_loss': 1800.0, 'epoch': 8.04}

 80%|████████  | 587/730 [52:03<12:04,  5.07s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.4574020802974701, 'epoch': 8.04}

 80%|████████  | 587/730 [52:03<12:04,  5.07s/it]
                                                 
{'train/learning_rate_real': 2.4331253199045974e-06, 'epoch': 8.04}

 80%|████████  | 587/730 [52:03<12:04,  5.07s/it]
                                                 
{'debug/num_tok_total': 2408.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 2408.0, 'debug/num_lat_loss': 1765.0, 'epoch': 8.04}

 80%|████████  | 587/730 [52:04<12:04,  5.07s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.44553983211517334, 'epoch': 8.04}

 80%|████████  | 587/730 [52:04<12:04,  5.07s/it]
                                                 
{'train/learning_rate_real': 2.4331253199045974e-06, 'epoch': 8.04}

 80%|████████  | 587/730 [52:04<12:04,  5.07s/it]
 81%|████████  | 588/730 [52:04<12:06,  5.11s/it]
                                                 
{'debug/num_tok_total': 2218.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2218.0, 'debug/num_lat_loss': 1777.0, 'epoch': 8.05}

 81%|████████  | 588/730 [52:05<12:06,  5.11s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.506874680519104, 'epoch': 8.05}

 81%|████████  | 588/730 [52:05<12:06,  5.11s/it]
                                                 
{'train/learning_rate_real': 2.4003443246847444e-06, 'epoch': 8.05}

 81%|████████  | 588/730 [52:05<12:06,  5.11s/it]
                                                 
{'debug/num_tok_total': 2449.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2449.0, 'debug/num_lat_loss': 1789.0, 'epoch': 8.05}

 81%|████████  | 588/730 [52:06<12:06,  5.11s/it]
                                                 
{'train/ce_loss': 1.578125, 'train/diffusion_loss': 0.43931522965431213, 'epoch': 8.05}

 81%|████████  | 588/730 [52:06<12:06,  5.11s/it]
                                                 
{'train/learning_rate_real': 2.4003443246847444e-06, 'epoch': 8.05}

 81%|████████  | 588/730 [52:06<12:06,  5.11s/it]
                                                 
{'debug/num_tok_total': 2421.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2421.0, 'debug/num_lat_loss': 1783.0, 'epoch': 8.05}

 81%|████████  | 588/730 [52:08<12:06,  5.11s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.47150668501853943, 'epoch': 8.05}

 81%|████████  | 588/730 [52:08<12:06,  5.11s/it]
                                                 
{'train/learning_rate_real': 2.4003443246847444e-06, 'epoch': 8.05}

 81%|████████  | 588/730 [52:08<12:06,  5.11s/it]
                                                 
{'debug/num_tok_total': 2676.0, 'debug/num_tok_loss': 1807.0, 'debug/num_lat_total': 2676.0, 'debug/num_lat_loss': 1807.0, 'epoch': 8.05}

 81%|████████  | 588/730 [52:09<12:06,  5.11s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.4119138717651367, 'epoch': 8.05}

 81%|████████  | 588/730 [52:09<12:06,  5.11s/it]
                                                 
{'train/learning_rate_real': 2.4003443246847444e-06, 'epoch': 8.05}

 81%|████████  | 588/730 [52:09<12:06,  5.11s/it]
 81%|████████  | 589/730 [52:09<11:55,  5.08s/it]
                                                 
{'debug/num_tok_total': 2657.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2657.0, 'debug/num_lat_loss': 1790.0, 'epoch': 8.07}

 81%|████████  | 589/730 [52:11<11:55,  5.08s/it]
                                                 
{'train/ce_loss': 1.6328125, 'train/diffusion_loss': 0.38259297609329224, 'epoch': 8.07}

 81%|████████  | 589/730 [52:11<11:55,  5.08s/it]
                                                 
{'train/learning_rate_real': 2.3677621856415783e-06, 'epoch': 8.07}

 81%|████████  | 589/730 [52:11<11:55,  5.08s/it]
                                                 
{'debug/num_tok_total': 2827.0, 'debug/num_tok_loss': 1761.0, 'debug/num_lat_total': 2827.0, 'debug/num_lat_loss': 1761.0, 'epoch': 8.07}

 81%|████████  | 589/730 [52:12<11:55,  5.08s/it]
                                                 
{'train/ce_loss': 2.015625, 'train/diffusion_loss': 0.3626105785369873, 'epoch': 8.07}

 81%|████████  | 589/730 [52:12<11:55,  5.08s/it]
                                                 
{'train/learning_rate_real': 2.3677621856415783e-06, 'epoch': 8.07}

 81%|████████  | 589/730 [52:12<11:55,  5.08s/it]
                                                 
{'debug/num_tok_total': 3081.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 3081.0, 'debug/num_lat_loss': 1786.0, 'epoch': 8.07}

 81%|████████  | 589/730 [52:13<11:55,  5.08s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.32082292437553406, 'epoch': 8.07}

 81%|████████  | 589/730 [52:13<11:55,  5.08s/it]
                                                 
{'train/learning_rate_real': 2.3677621856415783e-06, 'epoch': 8.07}

 81%|████████  | 589/730 [52:13<11:55,  5.08s/it]
                                                 
{'debug/num_tok_total': 2171.0, 'debug/num_tok_loss': 1554.0, 'debug/num_lat_total': 2171.0, 'debug/num_lat_loss': 1554.0, 'epoch': 8.07}

 81%|████████  | 589/730 [52:14<11:55,  5.08s/it]
                                                 
{'train/ce_loss': 2.03125, 'train/diffusion_loss': 0.41037848591804504, 'epoch': 8.07}

 81%|████████  | 589/730 [52:14<11:55,  5.08s/it]
                                                 
{'train/learning_rate_real': 2.3677621856415783e-06, 'epoch': 8.07}

 81%|████████  | 589/730 [52:14<11:55,  5.08s/it]03/16/2026 07:39:47 - INFO - __main__ - LoRA debug step 590: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 81%|████████  | 590/730 [52:15<12:17,  5.27s/it]
                                                 
{'loss': 2.5744, 'grad_norm': 1.168031930923462, 'learning_rate': 2.3677621856415783e-06, 'epoch': 8.08}

 81%|████████  | 590/730 [52:15<12:17,  5.27s/it]
                                                 
{'debug/num_tok_total': 3115.0, 'debug/num_tok_loss': 1803.0, 'debug/num_lat_total': 3115.0, 'debug/num_lat_loss': 1803.0, 'epoch': 8.08}

 81%|████████  | 590/730 [52:16<12:17,  5.27s/it]
                                                 
{'train/ce_loss': 1.6015625, 'train/diffusion_loss': 0.30546632409095764, 'epoch': 8.08}

 81%|████████  | 590/730 [52:16<12:17,  5.27s/it]
                                                 
{'train/learning_rate_real': 2.335379544297915e-06, 'epoch': 8.08}

 81%|████████  | 590/730 [52:16<12:17,  5.27s/it]
                                                 
{'debug/num_tok_total': 3030.0, 'debug/num_tok_loss': 1753.0, 'debug/num_lat_total': 3030.0, 'debug/num_lat_loss': 1753.0, 'epoch': 8.08}

 81%|████████  | 590/730 [52:18<12:17,  5.27s/it]
                                                 
{'train/ce_loss': 1.578125, 'train/diffusion_loss': 0.31824129819869995, 'epoch': 8.08}

 81%|████████  | 590/730 [52:18<12:17,  5.27s/it]
                                                 
{'train/learning_rate_real': 2.335379544297915e-06, 'epoch': 8.08}

 81%|████████  | 590/730 [52:18<12:17,  5.27s/it]
                                                 
{'debug/num_tok_total': 2696.0, 'debug/num_tok_loss': 1814.0, 'debug/num_lat_total': 2696.0, 'debug/num_lat_loss': 1814.0, 'epoch': 8.08}

 81%|████████  | 590/730 [52:19<12:17,  5.27s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.4063245356082916, 'epoch': 8.08}

 81%|████████  | 590/730 [52:19<12:17,  5.27s/it]
                                                 
{'train/learning_rate_real': 2.335379544297915e-06, 'epoch': 8.08}

 81%|████████  | 590/730 [52:19<12:17,  5.27s/it]
                                                 
{'debug/num_tok_total': 3284.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 3284.0, 'debug/num_lat_loss': 1778.0, 'epoch': 8.08}

 81%|████████  | 590/730 [52:20<12:17,  5.27s/it]
                                                 
{'train/ce_loss': 1.6328125, 'train/diffusion_loss': 0.26908355951309204, 'epoch': 8.08}

 81%|████████  | 590/730 [52:20<12:17,  5.27s/it]
                                                 
{'train/learning_rate_real': 2.335379544297915e-06, 'epoch': 8.08}

 81%|████████  | 590/730 [52:20<12:17,  5.27s/it]
 81%|████████  | 591/730 [52:21<12:37,  5.45s/it]
                                                 
{'debug/num_tok_total': 2624.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2624.0, 'debug/num_lat_loss': 1773.0, 'epoch': 8.1}

 81%|████████  | 591/730 [52:22<12:37,  5.45s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.41196611523628235, 'epoch': 8.1}

 81%|████████  | 591/730 [52:22<12:37,  5.45s/it]
                                                 
{'train/learning_rate_real': 2.3031970382485918e-06, 'epoch': 8.1}

 81%|████████  | 591/730 [52:22<12:37,  5.45s/it]
                                                 
{'debug/num_tok_total': 2832.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2832.0, 'debug/num_lat_loss': 1766.0, 'epoch': 8.1}

 81%|████████  | 591/730 [52:23<12:37,  5.45s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.3586272895336151, 'epoch': 8.1}

 81%|████████  | 591/730 [52:23<12:37,  5.45s/it]
                                                 
{'train/learning_rate_real': 2.3031970382485918e-06, 'epoch': 8.1}

 81%|████████  | 591/730 [52:23<12:37,  5.45s/it]
                                                 
{'debug/num_tok_total': 2822.0, 'debug/num_tok_loss': 1761.0, 'debug/num_lat_total': 2822.0, 'debug/num_lat_loss': 1761.0, 'epoch': 8.1}

 81%|████████  | 591/730 [52:25<12:37,  5.45s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.3326472342014313, 'epoch': 8.1}

 81%|████████  | 591/730 [52:25<12:37,  5.45s/it]
                                                 
{'train/learning_rate_real': 2.3031970382485918e-06, 'epoch': 8.1}

 81%|████████  | 591/730 [52:25<12:37,  5.45s/it]
                                                 
{'debug/num_tok_total': 2612.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2612.0, 'debug/num_lat_loss': 1766.0, 'epoch': 8.1}

 81%|████████  | 591/730 [52:26<12:37,  5.45s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.38019707798957825, 'epoch': 8.1}

 81%|████████  | 591/730 [52:26<12:37,  5.45s/it]
                                                 
{'train/learning_rate_real': 2.3031970382485918e-06, 'epoch': 8.1}

 81%|████████  | 591/730 [52:26<12:37,  5.45s/it]
 81%|████████  | 592/730 [52:26<12:33,  5.46s/it]
                                                 
{'debug/num_tok_total': 2726.0, 'debug/num_tok_loss': 1604.0, 'debug/num_lat_total': 2726.0, 'debug/num_lat_loss': 1604.0, 'epoch': 8.11}

 81%|████████  | 592/730 [52:28<12:33,  5.46s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.3233419954776764, 'epoch': 8.11}

 81%|████████  | 592/730 [52:28<12:33,  5.46s/it]
                                                 
{'train/learning_rate_real': 2.271215301147887e-06, 'epoch': 8.11}

 81%|████████  | 592/730 [52:28<12:33,  5.46s/it]
                                                 
{'debug/num_tok_total': 3131.0, 'debug/num_tok_loss': 1817.0, 'debug/num_lat_total': 3131.0, 'debug/num_lat_loss': 1817.0, 'epoch': 8.11}

 81%|████████  | 592/730 [52:29<12:33,  5.46s/it]
                                                 
{'train/ce_loss': 1.59375, 'train/diffusion_loss': 0.3010649085044861, 'epoch': 8.11}

 81%|████████  | 592/730 [52:29<12:33,  5.46s/it]
                                                 
{'train/learning_rate_real': 2.271215301147887e-06, 'epoch': 8.11}

 81%|████████  | 592/730 [52:29<12:33,  5.46s/it]
                                                 
{'debug/num_tok_total': 2842.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2842.0, 'debug/num_lat_loss': 1772.0, 'epoch': 8.11}

 81%|████████  | 592/730 [52:30<12:33,  5.46s/it]
                                                 
{'train/ce_loss': 1.5859375, 'train/diffusion_loss': 0.35829347372055054, 'epoch': 8.11}

 81%|████████  | 592/730 [52:30<12:33,  5.46s/it]
                                                 
{'train/learning_rate_real': 2.271215301147887e-06, 'epoch': 8.11}

 81%|████████  | 592/730 [52:30<12:33,  5.46s/it]
                                                 
{'debug/num_tok_total': 3077.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 3077.0, 'debug/num_lat_loss': 1784.0, 'epoch': 8.11}

 81%|████████  | 592/730 [52:32<12:33,  5.46s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.3273492157459259, 'epoch': 8.11}

 81%|████████  | 592/730 [52:32<12:33,  5.46s/it]
                                                 
{'train/learning_rate_real': 2.271215301147887e-06, 'epoch': 8.11}

 81%|████████  | 592/730 [52:32<12:33,  5.46s/it]
 81%|████████  | 593/730 [52:33<12:57,  5.68s/it]
                                                 
{'debug/num_tok_total': 2480.0, 'debug/num_tok_loss': 1808.0, 'debug/num_lat_total': 2480.0, 'debug/num_lat_loss': 1808.0, 'epoch': 8.12}

 81%|████████  | 593/730 [52:34<12:57,  5.68s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.4071198105812073, 'epoch': 8.12}

 81%|████████  | 593/730 [52:34<12:57,  5.68s/it]
                                                 
{'train/learning_rate_real': 2.239434962697072e-06, 'epoch': 8.12}

 81%|████████  | 593/730 [52:34<12:57,  5.68s/it]
                                                 
{'debug/num_tok_total': 2878.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2878.0, 'debug/num_lat_loss': 1793.0, 'epoch': 8.12}

 81%|████████  | 593/730 [52:35<12:57,  5.68s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.3482542335987091, 'epoch': 8.12}

 81%|████████  | 593/730 [52:35<12:57,  5.68s/it]
                                                 
{'train/learning_rate_real': 2.239434962697072e-06, 'epoch': 8.12}

 81%|████████  | 593/730 [52:35<12:57,  5.68s/it]
                                                 
{'debug/num_tok_total': 3043.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 3043.0, 'debug/num_lat_loss': 1765.0, 'epoch': 8.12}

 81%|████████  | 593/730 [52:36<12:57,  5.68s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.31067174673080444, 'epoch': 8.12}

 81%|████████  | 593/730 [52:36<12:57,  5.68s/it]
                                                 
{'train/learning_rate_real': 2.239434962697072e-06, 'epoch': 8.12}

 81%|████████  | 593/730 [52:36<12:57,  5.68s/it]
                                                 
{'debug/num_tok_total': 2171.0, 'debug/num_tok_loss': 1738.0, 'debug/num_lat_total': 2171.0, 'debug/num_lat_loss': 1738.0, 'epoch': 8.12}

 81%|████████  | 593/730 [52:37<12:57,  5.68s/it]
                                                 
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.4833768904209137, 'epoch': 8.12}

 81%|████████  | 593/730 [52:37<12:57,  5.68s/it]
                                                 
{'train/learning_rate_real': 2.239434962697072e-06, 'epoch': 8.12}

 81%|████████  | 593/730 [52:37<12:57,  5.68s/it]
 81%|████████▏ | 594/730 [52:38<12:42,  5.61s/it]
                                                 
{'debug/num_tok_total': 2420.0, 'debug/num_tok_loss': 1675.0, 'debug/num_lat_total': 2420.0, 'debug/num_lat_loss': 1675.0, 'epoch': 8.14}

 81%|████████▏ | 594/730 [52:39<12:42,  5.61s/it]
                                                 
{'train/ce_loss': 1.671875, 'train/diffusion_loss': 0.4059217572212219, 'epoch': 8.14}

 81%|████████▏ | 594/730 [52:39<12:42,  5.61s/it]
                                                 
{'train/learning_rate_real': 2.2078566486319937e-06, 'epoch': 8.14}

 81%|████████▏ | 594/730 [52:39<12:42,  5.61s/it]
                                                 
{'debug/num_tok_total': 3018.0, 'debug/num_tok_loss': 1752.0, 'debug/num_lat_total': 3018.0, 'debug/num_lat_loss': 1752.0, 'epoch': 8.14}

 81%|████████▏ | 594/730 [52:40<12:42,  5.61s/it]
                                                 
{'train/ce_loss': 1.65625, 'train/diffusion_loss': 0.30129319429397583, 'epoch': 8.14}

 81%|████████▏ | 594/730 [52:40<12:42,  5.61s/it]
                                                 
{'train/learning_rate_real': 2.2078566486319937e-06, 'epoch': 8.14}

 81%|████████▏ | 594/730 [52:40<12:42,  5.61s/it]
                                                 
{'debug/num_tok_total': 2864.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2864.0, 'debug/num_lat_loss': 1794.0, 'epoch': 8.14}

 81%|████████▏ | 594/730 [52:42<12:42,  5.61s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.358121782541275, 'epoch': 8.14}

 81%|████████▏ | 594/730 [52:42<12:42,  5.61s/it]
                                                 
{'train/learning_rate_real': 2.2078566486319937e-06, 'epoch': 8.14}

 81%|████████▏ | 594/730 [52:42<12:42,  5.61s/it]
                                                 
{'debug/num_tok_total': 2412.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2412.0, 'debug/num_lat_loss': 1776.0, 'epoch': 8.14}

 81%|████████▏ | 594/730 [52:43<12:42,  5.61s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.438777357339859, 'epoch': 8.14}

 81%|████████▏ | 594/730 [52:43<12:42,  5.61s/it]
                                                 
{'train/learning_rate_real': 2.2078566486319937e-06, 'epoch': 8.14}

 81%|████████▏ | 594/730 [52:43<12:42,  5.61s/it]
 82%|████████▏ | 595/730 [52:43<12:26,  5.53s/it]
                                                 
{'debug/num_tok_total': 2412.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2412.0, 'debug/num_lat_loss': 1772.0, 'epoch': 8.15}

 82%|████████▏ | 595/730 [52:44<12:26,  5.53s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.4263143241405487, 'epoch': 8.15}

 82%|████████▏ | 595/730 [52:44<12:26,  5.53s/it]
                                                 
{'train/learning_rate_real': 2.17648098071076e-06, 'epoch': 8.15}

 82%|████████▏ | 595/730 [52:44<12:26,  5.53s/it]
                                                 
{'debug/num_tok_total': 2443.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2443.0, 'debug/num_lat_loss': 1788.0, 'epoch': 8.15}

 82%|████████▏ | 595/730 [52:45<12:26,  5.53s/it]
                                                 
{'train/ce_loss': 1.640625, 'train/diffusion_loss': 0.4225360155105591, 'epoch': 8.15}

 82%|████████▏ | 595/730 [52:45<12:26,  5.53s/it]
                                                 
{'train/learning_rate_real': 2.17648098071076e-06, 'epoch': 8.15}

 82%|████████▏ | 595/730 [52:45<12:26,  5.53s/it]
                                                 
{'debug/num_tok_total': 2441.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2441.0, 'debug/num_lat_loss': 1791.0, 'epoch': 8.15}

 82%|████████▏ | 595/730 [52:47<12:26,  5.53s/it]
                                                 
{'train/ce_loss': 1.8828125, 'train/diffusion_loss': 0.4602355360984802, 'epoch': 8.15}

 82%|████████▏ | 595/730 [52:47<12:26,  5.53s/it]
                                                 
{'train/learning_rate_real': 2.17648098071076e-06, 'epoch': 8.15}

 82%|████████▏ | 595/730 [52:47<12:26,  5.53s/it]
                                                 
{'debug/num_tok_total': 2275.0, 'debug/num_tok_loss': 1620.0, 'debug/num_lat_total': 2275.0, 'debug/num_lat_loss': 1620.0, 'epoch': 8.15}

 82%|████████▏ | 595/730 [52:48<12:26,  5.53s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.41237741708755493, 'epoch': 8.15}

 82%|████████▏ | 595/730 [52:48<12:26,  5.53s/it]
                                                 
{'train/learning_rate_real': 2.17648098071076e-06, 'epoch': 8.15}

 82%|████████▏ | 595/730 [52:48<12:26,  5.53s/it]
 82%|████████▏ | 596/730 [52:48<12:05,  5.41s/it]
                                                 
{'debug/num_tok_total': 2772.0, 'debug/num_tok_loss': 1695.0, 'debug/num_lat_total': 2772.0, 'debug/num_lat_loss': 1695.0, 'epoch': 8.16}

 82%|████████▏ | 596/730 [52:50<12:05,  5.41s/it]
                                                 
{'train/ce_loss': 1.8515625, 'train/diffusion_loss': 0.3481716513633728, 'epoch': 8.16}

 82%|████████▏ | 596/730 [52:50<12:05,  5.41s/it]
                                                 
{'train/learning_rate_real': 2.145308576701492e-06, 'epoch': 8.16}

 82%|████████▏ | 596/730 [52:50<12:05,  5.41s/it]
                                                 
{'debug/num_tok_total': 2312.0, 'debug/num_tok_loss': 1688.0, 'debug/num_lat_total': 2312.0, 'debug/num_lat_loss': 1688.0, 'epoch': 8.16}

 82%|████████▏ | 596/730 [52:51<12:05,  5.41s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.4391588270664215, 'epoch': 8.16}

 82%|████████▏ | 596/730 [52:51<12:05,  5.41s/it]
                                                 
{'train/learning_rate_real': 2.145308576701492e-06, 'epoch': 8.16}

 82%|████████▏ | 596/730 [52:51<12:05,  5.41s/it]
                                                 
{'debug/num_tok_total': 3068.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 3068.0, 'debug/num_lat_loss': 1780.0, 'epoch': 8.16}

 82%|████████▏ | 596/730 [52:52<12:05,  5.41s/it]
                                                 
{'train/ce_loss': 1.9921875, 'train/diffusion_loss': 0.31799617409706116, 'epoch': 8.16}

 82%|████████▏ | 596/730 [52:52<12:05,  5.41s/it]
                                                 
{'train/learning_rate_real': 2.145308576701492e-06, 'epoch': 8.16}

 82%|████████▏ | 596/730 [52:52<12:05,  5.41s/it]
                                                 
{'debug/num_tok_total': 2811.0, 'debug/num_tok_loss': 1759.0, 'debug/num_lat_total': 2811.0, 'debug/num_lat_loss': 1759.0, 'epoch': 8.16}

 82%|████████▏ | 596/730 [52:53<12:05,  5.41s/it]
                                                 
{'train/ce_loss': 1.6328125, 'train/diffusion_loss': 0.35171249508857727, 'epoch': 8.16}

 82%|████████▏ | 596/730 [52:53<12:05,  5.41s/it]
                                                 
{'train/learning_rate_real': 2.145308576701492e-06, 'epoch': 8.16}

 82%|████████▏ | 596/730 [52:53<12:05,  5.41s/it]
 82%|████████▏ | 597/730 [52:54<12:02,  5.43s/it]
                                                 
{'debug/num_tok_total': 3030.0, 'debug/num_tok_loss': 1756.0, 'debug/num_lat_total': 3030.0, 'debug/num_lat_loss': 1756.0, 'epoch': 8.18}

 82%|████████▏ | 597/730 [52:55<12:02,  5.43s/it]
                                                 
{'train/ce_loss': 1.640625, 'train/diffusion_loss': 0.3273889124393463, 'epoch': 8.18}

 82%|████████▏ | 597/730 [52:55<12:02,  5.43s/it]
                                                 
{'train/learning_rate_real': 2.1143400503701755e-06, 'epoch': 8.18}

 82%|████████▏ | 597/730 [52:55<12:02,  5.43s/it]
                                                 
{'debug/num_tok_total': 2381.0, 'debug/num_tok_loss': 1755.0, 'debug/num_lat_total': 2381.0, 'debug/num_lat_loss': 1755.0, 'epoch': 8.18}

 82%|████████▏ | 597/730 [52:56<12:02,  5.43s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.4576631486415863, 'epoch': 8.18}

 82%|████████▏ | 597/730 [52:56<12:02,  5.43s/it]
                                                 
{'train/learning_rate_real': 2.1143400503701755e-06, 'epoch': 8.18}

 82%|████████▏ | 597/730 [52:56<12:02,  5.43s/it]
                                                 
{'debug/num_tok_total': 2830.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2830.0, 'debug/num_lat_loss': 1763.0, 'epoch': 8.18}

 82%|████████▏ | 597/730 [52:58<12:02,  5.43s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.3359583020210266, 'epoch': 8.18}

 82%|████████▏ | 597/730 [52:58<12:02,  5.43s/it]
                                                 
{'train/learning_rate_real': 2.1143400503701755e-06, 'epoch': 8.18}

 82%|████████▏ | 597/730 [52:58<12:02,  5.43s/it]
                                                 
{'debug/num_tok_total': 3240.0, 'debug/num_tok_loss': 1762.0, 'debug/num_lat_total': 3240.0, 'debug/num_lat_loss': 1762.0, 'epoch': 8.18}

 82%|████████▏ | 597/730 [52:59<12:02,  5.43s/it]
                                                 
{'train/ce_loss': 1.8984375, 'train/diffusion_loss': 0.2875746488571167, 'epoch': 8.18}

 82%|████████▏ | 597/730 [52:59<12:02,  5.43s/it]
                                                 
{'train/learning_rate_real': 2.1143400503701755e-06, 'epoch': 8.18}

 82%|████████▏ | 597/730 [52:59<12:02,  5.43s/it]
 82%|████████▏ | 598/730 [52:59<12:00,  5.46s/it]
                                                 
{'debug/num_tok_total': 2874.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2874.0, 'debug/num_lat_loss': 1790.0, 'epoch': 8.19}

 82%|████████▏ | 598/730 [53:01<12:00,  5.46s/it]
                                                 
{'train/ce_loss': 1.890625, 'train/diffusion_loss': 0.3227774500846863, 'epoch': 8.19}

 82%|████████▏ | 598/730 [53:01<12:00,  5.46s/it]
                                                 
{'train/learning_rate_real': 2.083576011468562e-06, 'epoch': 8.19}

 82%|████████▏ | 598/730 [53:01<12:00,  5.46s/it]
                                                 
{'debug/num_tok_total': 2429.0, 'debug/num_tok_loss': 1673.0, 'debug/num_lat_total': 2429.0, 'debug/num_lat_loss': 1673.0, 'epoch': 8.19}

 82%|████████▏ | 598/730 [53:02<12:00,  5.46s/it]
                                                 
{'train/ce_loss': 1.578125, 'train/diffusion_loss': 0.3840366303920746, 'epoch': 8.19}

 82%|████████▏ | 598/730 [53:02<12:00,  5.46s/it]
                                                 
{'train/learning_rate_real': 2.083576011468562e-06, 'epoch': 8.19}

 82%|████████▏ | 598/730 [53:02<12:00,  5.46s/it]
                                                 
{'debug/num_tok_total': 2624.0, 'debug/num_tok_loss': 1760.0, 'debug/num_lat_total': 2624.0, 'debug/num_lat_loss': 1760.0, 'epoch': 8.19}

 82%|████████▏ | 598/730 [53:03<12:00,  5.46s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.4002417027950287, 'epoch': 8.19}

 82%|████████▏ | 598/730 [53:03<12:00,  5.46s/it]
                                                 
{'train/learning_rate_real': 2.083576011468562e-06, 'epoch': 8.19}

 82%|████████▏ | 598/730 [53:03<12:00,  5.46s/it]
                                                 
{'debug/num_tok_total': 2848.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2848.0, 'debug/num_lat_loss': 1774.0, 'epoch': 8.19}

 82%|████████▏ | 598/730 [53:04<12:00,  5.46s/it]
                                                 
{'train/ce_loss': 1.609375, 'train/diffusion_loss': 0.33459097146987915, 'epoch': 8.19}

 82%|████████▏ | 598/730 [53:04<12:00,  5.46s/it]
                                                 
{'train/learning_rate_real': 2.083576011468562e-06, 'epoch': 8.19}

 82%|████████▏ | 598/730 [53:04<12:00,  5.46s/it]
 82%|████████▏ | 599/730 [53:05<11:53,  5.45s/it]
                                                 
{'debug/num_tok_total': 1869.0, 'debug/num_tok_loss': 1655.0, 'debug/num_lat_total': 1869.0, 'debug/num_lat_loss': 1655.0, 'epoch': 8.21}

 82%|████████▏ | 599/730 [53:06<11:53,  5.45s/it]
                                                 
{'train/ce_loss': 1.8828125, 'train/diffusion_loss': 0.510683000087738, 'epoch': 8.21}

 82%|████████▏ | 599/730 [53:06<11:53,  5.45s/it]
                                                 
{'train/learning_rate_real': 2.053017065722171e-06, 'epoch': 8.21}

 82%|████████▏ | 599/730 [53:06<11:53,  5.45s/it]
                                                 
{'debug/num_tok_total': 2860.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2860.0, 'debug/num_lat_loss': 1777.0, 'epoch': 8.21}

 82%|████████▏ | 599/730 [53:07<11:53,  5.45s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.34388813376426697, 'epoch': 8.21}

 82%|████████▏ | 599/730 [53:07<11:53,  5.45s/it]
                                                 
{'train/learning_rate_real': 2.053017065722171e-06, 'epoch': 8.21}

 82%|████████▏ | 599/730 [53:07<11:53,  5.45s/it]
                                                 
{'debug/num_tok_total': 2636.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2636.0, 'debug/num_lat_loss': 1782.0, 'epoch': 8.21}

 82%|████████▏ | 599/730 [53:08<11:53,  5.45s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.4122960865497589, 'epoch': 8.21}

 82%|████████▏ | 599/730 [53:08<11:53,  5.45s/it]
                                                 
{'train/learning_rate_real': 2.053017065722171e-06, 'epoch': 8.21}

 82%|████████▏ | 599/730 [53:08<11:53,  5.45s/it]
                                                 
{'debug/num_tok_total': 2528.0, 'debug/num_tok_loss': 1677.0, 'debug/num_lat_total': 2528.0, 'debug/num_lat_loss': 1677.0, 'epoch': 8.21}

 82%|████████▏ | 599/730 [53:10<11:53,  5.45s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.36972352862358093, 'epoch': 8.21}

 82%|████████▏ | 599/730 [53:10<11:53,  5.45s/it]
                                                 
{'train/learning_rate_real': 2.053017065722171e-06, 'epoch': 8.21}

 82%|████████▏ | 599/730 [53:10<11:53,  5.45s/it]03/16/2026 07:40:42 - INFO - __main__ - LoRA debug step 600: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 82%|████████▏ | 600/730 [53:10<11:35,  5.35s/it]
                                                 
{'loss': 2.3518, 'grad_norm': 1.1867148876190186, 'learning_rate': 2.053017065722171e-06, 'epoch': 8.22}

 82%|████████▏ | 600/730 [53:10<11:35,  5.35s/it]/home/ubuntu/.local/lib/python3.10/site-packages/peft/utils/save_and_load.py:295: UserWarning: Could not find a config file in  - will assume that the vocabulary was not modified.
  warnings.warn(

                                                 
{'debug/num_tok_total': 2818.0, 'debug/num_tok_loss': 1758.0, 'debug/num_lat_total': 2818.0, 'debug/num_lat_loss': 1758.0, 'epoch': 8.22}

 82%|████████▏ | 600/730 [53:13<11:35,  5.35s/it]
                                                 
{'train/ce_loss': 1.6484375, 'train/diffusion_loss': 0.3471631705760956, 'epoch': 8.22}

 82%|████████▏ | 600/730 [53:13<11:35,  5.35s/it]
                                                 
{'train/learning_rate_real': 2.022663814818357e-06, 'epoch': 8.22}

 82%|████████▏ | 600/730 [53:13<11:35,  5.35s/it]
                                                 
{'debug/num_tok_total': 2592.0, 'debug/num_tok_loss': 1724.0, 'debug/num_lat_total': 2592.0, 'debug/num_lat_loss': 1724.0, 'epoch': 8.22}

 82%|████████▏ | 600/730 [53:14<11:35,  5.35s/it]
                                                 
{'train/ce_loss': 1.640625, 'train/diffusion_loss': 0.3929574191570282, 'epoch': 8.22}

 82%|████████▏ | 600/730 [53:14<11:35,  5.35s/it]
                                                 
{'train/learning_rate_real': 2.022663814818357e-06, 'epoch': 8.22}

 82%|████████▏ | 600/730 [53:14<11:35,  5.35s/it]
                                                 
{'debug/num_tok_total': 2258.0, 'debug/num_tok_loss': 1815.0, 'debug/num_lat_total': 2258.0, 'debug/num_lat_loss': 1815.0, 'epoch': 8.22}

 82%|████████▏ | 600/730 [53:15<11:35,  5.35s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.5017621517181396, 'epoch': 8.22}

 82%|████████▏ | 600/730 [53:15<11:35,  5.35s/it]
                                                 
{'train/learning_rate_real': 2.022663814818357e-06, 'epoch': 8.22}

 82%|████████▏ | 600/730 [53:15<11:35,  5.35s/it]
                                                 
{'debug/num_tok_total': 2234.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 2234.0, 'debug/num_lat_loss': 1797.0, 'epoch': 8.22}

 82%|████████▏ | 600/730 [53:16<11:35,  5.35s/it]
                                                 
{'train/ce_loss': 1.8828125, 'train/diffusion_loss': 0.49106383323669434, 'epoch': 8.22}

 82%|████████▏ | 600/730 [53:16<11:35,  5.35s/it]
                                                 
{'train/learning_rate_real': 2.022663814818357e-06, 'epoch': 8.22}

 82%|████████▏ | 600/730 [53:16<11:35,  5.35s/it]
 82%|████████▏ | 601/730 [53:17<12:21,  5.75s/it]
                                                 
{'debug/num_tok_total': 2222.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2222.0, 'debug/num_lat_loss': 1786.0, 'epoch': 8.23}

 82%|████████▏ | 601/730 [53:18<12:21,  5.75s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.48819246888160706, 'epoch': 8.23}

 82%|████████▏ | 601/730 [53:18<12:21,  5.75s/it]
                                                 
{'train/learning_rate_real': 1.992516856394462e-06, 'epoch': 8.23}

 82%|████████▏ | 601/730 [53:18<12:21,  5.75s/it]
                                                 
{'debug/num_tok_total': 2526.0, 'debug/num_tok_loss': 1725.0, 'debug/num_lat_total': 2526.0, 'debug/num_lat_loss': 1725.0, 'epoch': 8.23}

 82%|████████▏ | 601/730 [53:19<12:21,  5.75s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.3887117803096771, 'epoch': 8.23}

 82%|████████▏ | 601/730 [53:19<12:21,  5.75s/it]
                                                 
{'train/learning_rate_real': 1.992516856394462e-06, 'epoch': 8.23}

 82%|████████▏ | 601/730 [53:19<12:21,  5.75s/it]
                                                 
{'debug/num_tok_total': 2268.0, 'debug/num_tok_loss': 1628.0, 'debug/num_lat_total': 2268.0, 'debug/num_lat_loss': 1628.0, 'epoch': 8.23}

 82%|████████▏ | 601/730 [53:20<12:21,  5.75s/it]
                                                 
{'train/ce_loss': 1.625, 'train/diffusion_loss': 0.44391611218452454, 'epoch': 8.23}

 82%|████████▏ | 601/730 [53:20<12:21,  5.75s/it]
                                                 
{'train/learning_rate_real': 1.992516856394462e-06, 'epoch': 8.23}

 82%|████████▏ | 601/730 [53:20<12:21,  5.75s/it]
                                                 
{'debug/num_tok_total': 2581.0, 'debug/num_tok_loss': 1751.0, 'debug/num_lat_total': 2581.0, 'debug/num_lat_loss': 1751.0, 'epoch': 8.23}

 82%|████████▏ | 601/730 [53:21<12:21,  5.75s/it]
                                                 
{'train/ce_loss': 1.65625, 'train/diffusion_loss': 0.4091222286224365, 'epoch': 8.23}

 82%|████████▏ | 601/730 [53:21<12:21,  5.75s/it]
                                                 
{'train/learning_rate_real': 1.992516856394462e-06, 'epoch': 8.23}

 82%|████████▏ | 601/730 [53:21<12:21,  5.75s/it]
 82%|████████▏ | 602/730 [53:22<11:53,  5.57s/it]
                                                 
{'debug/num_tok_total': 2191.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2191.0, 'debug/num_lat_loss': 1771.0, 'epoch': 8.25}

 82%|████████▏ | 602/730 [53:23<11:53,  5.57s/it]
                                                 
{'train/ce_loss': 1.609375, 'train/diffusion_loss': 0.4911876916885376, 'epoch': 8.25}

 82%|████████▏ | 602/730 [53:23<11:53,  5.57s/it]
                                                 
{'train/learning_rate_real': 1.9625767840260616e-06, 'epoch': 8.25}

 82%|████████▏ | 602/730 [53:23<11:53,  5.57s/it]
                                                 
{'debug/num_tok_total': 2698.0, 'debug/num_tok_loss': 1823.0, 'debug/num_lat_total': 2698.0, 'debug/num_lat_loss': 1823.0, 'epoch': 8.25}

 82%|████████▏ | 602/730 [53:24<11:53,  5.57s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.4191938042640686, 'epoch': 8.25}

 82%|████████▏ | 602/730 [53:24<11:53,  5.57s/it]
                                                 
{'train/learning_rate_real': 1.9625767840260616e-06, 'epoch': 8.25}

 82%|████████▏ | 602/730 [53:24<11:53,  5.57s/it]
                                                 
{'debug/num_tok_total': 3040.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 3040.0, 'debug/num_lat_loss': 1763.0, 'epoch': 8.25}

 82%|████████▏ | 602/730 [53:26<11:53,  5.57s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.3437688946723938, 'epoch': 8.25}

 82%|████████▏ | 602/730 [53:26<11:53,  5.57s/it]
                                                 
{'train/learning_rate_real': 1.9625767840260616e-06, 'epoch': 8.25}

 82%|████████▏ | 602/730 [53:26<11:53,  5.57s/it]
                                                 
{'debug/num_tok_total': 2296.0, 'debug/num_tok_loss': 1636.0, 'debug/num_lat_total': 2296.0, 'debug/num_lat_loss': 1636.0, 'epoch': 8.25}

 82%|████████▏ | 602/730 [53:27<11:53,  5.57s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.4422857463359833, 'epoch': 8.25}

 82%|████████▏ | 602/730 [53:27<11:53,  5.57s/it]
                                                 
{'train/learning_rate_real': 1.9625767840260616e-06, 'epoch': 8.25}

 82%|████████▏ | 602/730 [53:27<11:53,  5.57s/it]
 83%|████████▎ | 603/730 [53:27<11:46,  5.56s/it]
                                                 
{'debug/num_tok_total': 2414.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2414.0, 'debug/num_lat_loss': 1776.0, 'epoch': 8.26}

 83%|████████▎ | 603/730 [53:28<11:46,  5.56s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.42678284645080566, 'epoch': 8.26}

 83%|████████▎ | 603/730 [53:28<11:46,  5.56s/it]
                                                 
{'train/learning_rate_real': 1.9328441872152673e-06, 'epoch': 8.26}

 83%|████████▎ | 603/730 [53:28<11:46,  5.56s/it]
                                                 
{'debug/num_tok_total': 2616.0, 'debug/num_tok_loss': 1558.0, 'debug/num_lat_total': 2616.0, 'debug/num_lat_loss': 1558.0, 'epoch': 8.26}

 83%|████████▎ | 603/730 [53:30<11:46,  5.56s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.32167738676071167, 'epoch': 8.26}

 83%|████████▎ | 603/730 [53:30<11:46,  5.56s/it]
                                                 
{'train/learning_rate_real': 1.9328441872152673e-06, 'epoch': 8.26}

 83%|████████▎ | 603/730 [53:30<11:46,  5.56s/it]
                                                 
{'debug/num_tok_total': 2420.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2420.0, 'debug/num_lat_loss': 1772.0, 'epoch': 8.26}

 83%|████████▎ | 603/730 [53:31<11:46,  5.56s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.4474724531173706, 'epoch': 8.26}

 83%|████████▎ | 603/730 [53:31<11:46,  5.56s/it]
                                                 
{'train/learning_rate_real': 1.9328441872152673e-06, 'epoch': 8.26}

 83%|████████▎ | 603/730 [53:31<11:46,  5.56s/it]
                                                 
{'debug/num_tok_total': 2433.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2433.0, 'debug/num_lat_loss': 1785.0, 'epoch': 8.26}

 83%|████████▎ | 603/730 [53:32<11:46,  5.56s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.4415070116519928, 'epoch': 8.26}

 83%|████████▎ | 603/730 [53:32<11:46,  5.56s/it]
                                                 
{'train/learning_rate_real': 1.9328441872152673e-06, 'epoch': 8.26}

 83%|████████▎ | 603/730 [53:32<11:46,  5.56s/it]
 83%|████████▎ | 604/730 [53:33<11:36,  5.52s/it]
                                                 
{'debug/num_tok_total': 2889.0, 'debug/num_tok_loss': 1804.0, 'debug/num_lat_total': 2889.0, 'debug/num_lat_loss': 1804.0, 'epoch': 8.27}

 83%|████████▎ | 604/730 [53:34<11:36,  5.52s/it]
                                                 
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.3669939637184143, 'epoch': 8.27}

 83%|████████▎ | 604/730 [53:34<11:36,  5.52s/it]
                                                 
{'train/learning_rate_real': 1.9033196513791138e-06, 'epoch': 8.27}

 83%|████████▎ | 604/730 [53:34<11:36,  5.52s/it]
                                                 
{'debug/num_tok_total': 2622.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2622.0, 'debug/num_lat_loss': 1770.0, 'epoch': 8.27}

 83%|████████▎ | 604/730 [53:35<11:36,  5.52s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.425268292427063, 'epoch': 8.27}

 83%|████████▎ | 604/730 [53:35<11:36,  5.52s/it]
                                                 
{'train/learning_rate_real': 1.9033196513791138e-06, 'epoch': 8.27}

 83%|████████▎ | 604/730 [53:35<11:36,  5.52s/it]
                                                 
{'debug/num_tok_total': 2887.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2887.0, 'debug/num_lat_loss': 1799.0, 'epoch': 8.27}

 83%|████████▎ | 604/730 [53:37<11:36,  5.52s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.36688926815986633, 'epoch': 8.27}

 83%|████████▎ | 604/730 [53:37<11:36,  5.52s/it]
                                                 
{'train/learning_rate_real': 1.9033196513791138e-06, 'epoch': 8.27}

 83%|████████▎ | 604/730 [53:37<11:36,  5.52s/it]
                                                 
{'debug/num_tok_total': 2273.0, 'debug/num_tok_loss': 1699.0, 'debug/num_lat_total': 2273.0, 'debug/num_lat_loss': 1699.0, 'epoch': 8.27}

 83%|████████▎ | 604/730 [53:38<11:36,  5.52s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.45896315574645996, 'epoch': 8.27}

 83%|████████▎ | 604/730 [53:38<11:36,  5.52s/it]
                                                 
{'train/learning_rate_real': 1.9033196513791138e-06, 'epoch': 8.27}

 83%|████████▎ | 604/730 [53:38<11:36,  5.52s/it]
 83%|████████▎ | 605/730 [53:38<11:26,  5.49s/it]
                                                 
{'debug/num_tok_total': 2623.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2623.0, 'debug/num_lat_loss': 1772.0, 'epoch': 8.29}

 83%|████████▎ | 605/730 [53:39<11:26,  5.49s/it]
                                                 
{'train/ce_loss': 2.15625, 'train/diffusion_loss': 0.4327562749385834, 'epoch': 8.29}

 83%|████████▎ | 605/730 [53:39<11:26,  5.49s/it]
                                                 
{'train/learning_rate_real': 1.8740037578380499e-06, 'epoch': 8.29}

 83%|████████▎ | 605/730 [53:39<11:26,  5.49s/it]
                                                 
{'debug/num_tok_total': 2548.0, 'debug/num_tok_loss': 1742.0, 'debug/num_lat_total': 2548.0, 'debug/num_lat_loss': 1742.0, 'epoch': 8.29}

 83%|████████▎ | 605/730 [53:40<11:26,  5.49s/it]
                                                 
{'train/ce_loss': 1.9609375, 'train/diffusion_loss': 0.4155462682247162, 'epoch': 8.29}

 83%|████████▎ | 605/730 [53:40<11:26,  5.49s/it]
                                                 
{'train/learning_rate_real': 1.8740037578380499e-06, 'epoch': 8.29}

 83%|████████▎ | 605/730 [53:40<11:26,  5.49s/it]
                                                 
{'debug/num_tok_total': 2690.0, 'debug/num_tok_loss': 1645.0, 'debug/num_lat_total': 2690.0, 'debug/num_lat_loss': 1645.0, 'epoch': 8.29}

 83%|████████▎ | 605/730 [53:42<11:26,  5.49s/it]
                                                 
{'train/ce_loss': 1.6328125, 'train/diffusion_loss': 0.33981332182884216, 'epoch': 8.29}

 83%|████████▎ | 605/730 [53:42<11:26,  5.49s/it]
                                                 
{'train/learning_rate_real': 1.8740037578380499e-06, 'epoch': 8.29}

 83%|████████▎ | 605/730 [53:42<11:26,  5.49s/it]
                                                 
{'debug/num_tok_total': 2405.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2405.0, 'debug/num_lat_loss': 1763.0, 'epoch': 8.29}

 83%|████████▎ | 605/730 [53:43<11:26,  5.49s/it]
                                                 
{'train/ce_loss': 1.609375, 'train/diffusion_loss': 0.45871564745903015, 'epoch': 8.29}

 83%|████████▎ | 605/730 [53:43<11:26,  5.49s/it]
                                                 
{'train/learning_rate_real': 1.8740037578380499e-06, 'epoch': 8.29}

 83%|████████▎ | 605/730 [53:43<11:26,  5.49s/it]
 83%|████████▎ | 606/730 [53:43<11:07,  5.38s/it]
                                                 
{'debug/num_tok_total': 2448.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2448.0, 'debug/num_lat_loss': 1781.0, 'epoch': 8.3}

 83%|████████▎ | 606/730 [53:44<11:07,  5.38s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.45271581411361694, 'epoch': 8.3}

 83%|████████▎ | 606/730 [53:44<11:07,  5.38s/it]
                                                 
{'train/learning_rate_real': 1.8448970838044685e-06, 'epoch': 8.3}

 83%|████████▎ | 606/730 [53:44<11:07,  5.38s/it]
                                                 
{'debug/num_tok_total': 1954.0, 'debug/num_tok_loss': 1511.0, 'debug/num_lat_total': 1954.0, 'debug/num_lat_loss': 1511.0, 'epoch': 8.3}

 83%|████████▎ | 606/730 [53:45<11:07,  5.38s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.5332106947898865, 'epoch': 8.3}

 83%|████████▎ | 606/730 [53:45<11:07,  5.38s/it]
                                                 
{'train/learning_rate_real': 1.8448970838044685e-06, 'epoch': 8.3}

 83%|████████▎ | 606/730 [53:45<11:07,  5.38s/it]
                                                 
{'debug/num_tok_total': 2236.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2236.0, 'debug/num_lat_loss': 1789.0, 'epoch': 8.3}

 83%|████████▎ | 606/730 [53:47<11:07,  5.38s/it]
                                                 
{'train/ce_loss': 1.8828125, 'train/diffusion_loss': 0.5037495493888855, 'epoch': 8.3}

 83%|████████▎ | 606/730 [53:47<11:07,  5.38s/it]
                                                 
{'train/learning_rate_real': 1.8448970838044685e-06, 'epoch': 8.3}

 83%|████████▎ | 606/730 [53:47<11:07,  5.38s/it]
                                                 
{'debug/num_tok_total': 2195.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 2195.0, 'debug/num_lat_loss': 1765.0, 'epoch': 8.3}

 83%|████████▎ | 606/730 [53:48<11:07,  5.38s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.5024689435958862, 'epoch': 8.3}

 83%|████████▎ | 606/730 [53:48<11:07,  5.38s/it]
                                                 
{'train/learning_rate_real': 1.8448970838044685e-06, 'epoch': 8.3}

 83%|████████▎ | 606/730 [53:48<11:07,  5.38s/it]
 83%|████████▎ | 607/730 [53:48<10:43,  5.23s/it]
                                                 
{'debug/num_tok_total': 2842.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 2842.0, 'debug/num_lat_loss': 1768.0, 'epoch': 8.32}

 83%|████████▎ | 607/730 [53:49<10:43,  5.23s/it]
                                                 
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.3599364161491394, 'epoch': 8.32}

 83%|████████▎ | 607/730 [53:49<10:43,  5.23s/it]
                                                 
{'train/learning_rate_real': 1.8160002023713707e-06, 'epoch': 8.32}

 83%|████████▎ | 607/730 [53:49<10:43,  5.23s/it]
                                                 
{'debug/num_tok_total': 2886.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 2886.0, 'debug/num_lat_loss': 1797.0, 'epoch': 8.32}

 83%|████████▎ | 607/730 [53:51<10:43,  5.23s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.35360434651374817, 'epoch': 8.32}

 83%|████████▎ | 607/730 [53:51<10:43,  5.23s/it]
                                                 
{'train/learning_rate_real': 1.8160002023713707e-06, 'epoch': 8.32}

 83%|████████▎ | 607/730 [53:51<10:43,  5.23s/it]
                                                 
{'debug/num_tok_total': 3283.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 3283.0, 'debug/num_lat_loss': 1777.0, 'epoch': 8.32}

 83%|████████▎ | 607/730 [53:52<10:43,  5.23s/it]
                                                 
{'train/ce_loss': 1.6484375, 'train/diffusion_loss': 0.2652038335800171, 'epoch': 8.32}

 83%|████████▎ | 607/730 [53:52<10:43,  5.23s/it]
                                                 
{'train/learning_rate_real': 1.8160002023713707e-06, 'epoch': 8.32}

 83%|████████▎ | 607/730 [53:52<10:43,  5.23s/it]
                                                 
{'debug/num_tok_total': 2879.0, 'debug/num_tok_loss': 1681.0, 'debug/num_lat_total': 2879.0, 'debug/num_lat_loss': 1681.0, 'epoch': 8.32}

 83%|████████▎ | 607/730 [53:53<10:43,  5.23s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.31583186984062195, 'epoch': 8.32}

 83%|████████▎ | 607/730 [53:53<10:43,  5.23s/it]
                                                 
{'train/learning_rate_real': 1.8160002023713707e-06, 'epoch': 8.32}

 83%|████████▎ | 607/730 [53:53<10:43,  5.23s/it]
 83%|████████▎ | 608/730 [53:54<10:53,  5.36s/it]
                                                 
{'debug/num_tok_total': 2701.0, 'debug/num_tok_loss': 1816.0, 'debug/num_lat_total': 2701.0, 'debug/num_lat_loss': 1816.0, 'epoch': 8.33}

 83%|████████▎ | 608/730 [53:55<10:53,  5.36s/it]
                                                 
{'train/ce_loss': 2.109375, 'train/diffusion_loss': 0.4536134600639343, 'epoch': 8.33}

 83%|████████▎ | 608/730 [53:55<10:53,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.7873136825010614e-06, 'epoch': 8.33}

 83%|████████▎ | 608/730 [53:55<10:53,  5.36s/it]
                                                 
{'debug/num_tok_total': 2857.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2857.0, 'debug/num_lat_loss': 1786.0, 'epoch': 8.33}

 83%|████████▎ | 608/730 [53:56<10:53,  5.36s/it]
                                                 
{'train/ce_loss': 1.6328125, 'train/diffusion_loss': 0.3665321469306946, 'epoch': 8.33}

 83%|████████▎ | 608/730 [53:56<10:53,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.7873136825010614e-06, 'epoch': 8.33}

 83%|████████▎ | 608/730 [53:56<10:53,  5.36s/it]
                                                 
{'debug/num_tok_total': 2394.0, 'debug/num_tok_loss': 1750.0, 'debug/num_lat_total': 2394.0, 'debug/num_lat_loss': 1750.0, 'epoch': 8.33}

 83%|████████▎ | 608/730 [53:57<10:53,  5.36s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.4663736820220947, 'epoch': 8.33}

 83%|████████▎ | 608/730 [53:58<10:53,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.7873136825010614e-06, 'epoch': 8.33}

 83%|████████▎ | 608/730 [53:58<10:53,  5.36s/it]
                                                 
{'debug/num_tok_total': 2855.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2855.0, 'debug/num_lat_loss': 1784.0, 'epoch': 8.33}

 83%|████████▎ | 608/730 [53:59<10:53,  5.36s/it]
                                                 
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.35612615942955017, 'epoch': 8.33}

 83%|████████▎ | 608/730 [53:59<10:53,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.7873136825010614e-06, 'epoch': 8.33}

 83%|████████▎ | 608/730 [53:59<10:53,  5.36s/it]
 83%|████████▎ | 609/730 [53:59<10:51,  5.38s/it]
                                                 
{'debug/num_tok_total': 2031.0, 'debug/num_tok_loss': 1807.0, 'debug/num_lat_total': 2031.0, 'debug/num_lat_loss': 1807.0, 'epoch': 8.34}

 83%|████████▎ | 609/730 [54:00<10:51,  5.38s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.5232307314872742, 'epoch': 8.34}

 83%|████████▎ | 609/730 [54:00<10:51,  5.38s/it]
                                                 
{'train/learning_rate_real': 1.7588380890139464e-06, 'epoch': 8.34}

 83%|████████▎ | 609/730 [54:00<10:51,  5.38s/it]
                                                 
{'debug/num_tok_total': 2459.0, 'debug/num_tok_loss': 1680.0, 'debug/num_lat_total': 2459.0, 'debug/num_lat_loss': 1680.0, 'epoch': 8.34}

 83%|████████▎ | 609/730 [54:02<10:51,  5.38s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.4094069004058838, 'epoch': 8.34}

 83%|████████▎ | 609/730 [54:02<10:51,  5.38s/it]
                                                 
{'train/learning_rate_real': 1.7588380890139464e-06, 'epoch': 8.34}

 83%|████████▎ | 609/730 [54:02<10:51,  5.38s/it]
                                                 
{'debug/num_tok_total': 2827.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2827.0, 'debug/num_lat_loss': 1763.0, 'epoch': 8.34}

 83%|████████▎ | 609/730 [54:03<10:51,  5.38s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.3593791425228119, 'epoch': 8.34}

 83%|████████▎ | 609/730 [54:03<10:51,  5.38s/it]
                                                 
{'train/learning_rate_real': 1.7588380890139464e-06, 'epoch': 8.34}

 83%|████████▎ | 609/730 [54:03<10:51,  5.38s/it]
                                                 
{'debug/num_tok_total': 3330.0, 'debug/num_tok_loss': 1800.0, 'debug/num_lat_total': 3330.0, 'debug/num_lat_loss': 1800.0, 'epoch': 8.34}

 83%|████████▎ | 609/730 [54:04<10:51,  5.38s/it]
                                                 
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.2814733684062958, 'epoch': 8.34}

 83%|████████▎ | 609/730 [54:04<10:51,  5.38s/it]
                                                 
{'train/learning_rate_real': 1.7588380890139464e-06, 'epoch': 8.34}

 83%|████████▎ | 609/730 [54:04<10:51,  5.38s/it]03/16/2026 07:41:37 - INFO - __main__ - LoRA debug step 610: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 84%|████████▎ | 610/730 [54:05<10:46,  5.38s/it]
                                                 
{'loss': 2.6013, 'grad_norm': 3.262697219848633, 'learning_rate': 1.7588380890139464e-06, 'epoch': 8.36}

 84%|████████▎ | 610/730 [54:05<10:46,  5.38s/it]
                                                 
{'debug/num_tok_total': 2403.0, 'debug/num_tok_loss': 1754.0, 'debug/num_lat_total': 2403.0, 'debug/num_lat_loss': 1754.0, 'epoch': 8.36}

 84%|████████▎ | 610/730 [54:06<10:46,  5.38s/it]
                                                 
{'train/ce_loss': 1.6328125, 'train/diffusion_loss': 0.44287240505218506, 'epoch': 8.36}

 84%|████████▎ | 610/730 [54:06<10:46,  5.38s/it]
                                                 
{'train/learning_rate_real': 1.7305739825774228e-06, 'epoch': 8.36}

 84%|████████▎ | 610/730 [54:06<10:46,  5.38s/it]
                                                 
{'debug/num_tok_total': 2636.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2636.0, 'debug/num_lat_loss': 1788.0, 'epoch': 8.36}

 84%|████████▎ | 610/730 [54:07<10:46,  5.38s/it]
                                                 
{'train/ce_loss': 1.640625, 'train/diffusion_loss': 0.4203449487686157, 'epoch': 8.36}

 84%|████████▎ | 610/730 [54:07<10:46,  5.38s/it]
                                                 
{'train/learning_rate_real': 1.7305739825774228e-06, 'epoch': 8.36}

 84%|████████▎ | 610/730 [54:07<10:46,  5.38s/it]
                                                 
{'debug/num_tok_total': 2399.0, 'debug/num_tok_loss': 1756.0, 'debug/num_lat_total': 2399.0, 'debug/num_lat_loss': 1756.0, 'epoch': 8.36}

 84%|████████▎ | 610/730 [54:08<10:46,  5.38s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.4536251127719879, 'epoch': 8.36}

 84%|████████▎ | 610/730 [54:08<10:46,  5.38s/it]
                                                 
{'train/learning_rate_real': 1.7305739825774228e-06, 'epoch': 8.36}

 84%|████████▎ | 610/730 [54:08<10:46,  5.38s/it]
                                                 
{'debug/num_tok_total': 2991.0, 'debug/num_tok_loss': 1737.0, 'debug/num_lat_total': 2991.0, 'debug/num_lat_loss': 1737.0, 'epoch': 8.36}

 84%|████████▎ | 610/730 [54:09<10:46,  5.38s/it]
                                                 
{'train/ce_loss': 1.96875, 'train/diffusion_loss': 0.3182593584060669, 'epoch': 8.36}

 84%|████████▎ | 610/730 [54:09<10:46,  5.38s/it]
                                                 
{'train/learning_rate_real': 1.7305739825774228e-06, 'epoch': 8.36}

 84%|████████▎ | 610/730 [54:09<10:46,  5.38s/it]
 84%|████████▎ | 611/730 [54:10<10:37,  5.36s/it]
                                                 
{'debug/num_tok_total': 2029.0, 'debug/num_tok_loss': 1609.0, 'debug/num_lat_total': 2029.0, 'debug/num_lat_loss': 1609.0, 'epoch': 8.37}

 84%|████████▎ | 611/730 [54:11<10:37,  5.36s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.4866446256637573, 'epoch': 8.37}

 84%|████████▎ | 611/730 [54:11<10:37,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.7025219196948383e-06, 'epoch': 8.37}

 84%|████████▎ | 611/730 [54:11<10:37,  5.36s/it]
                                                 
{'debug/num_tok_total': 2647.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2647.0, 'debug/num_lat_loss': 1788.0, 'epoch': 8.37}

 84%|████████▎ | 611/730 [54:12<10:37,  5.36s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.429523766040802, 'epoch': 8.37}

 84%|████████▎ | 611/730 [54:12<10:37,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.7025219196948383e-06, 'epoch': 8.37}

 84%|████████▎ | 611/730 [54:12<10:37,  5.36s/it]
                                                 
{'debug/num_tok_total': 2430.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2430.0, 'debug/num_lat_loss': 1780.0, 'epoch': 8.37}

 84%|████████▎ | 611/730 [54:14<10:37,  5.36s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.44242265820503235, 'epoch': 8.37}

 84%|████████▎ | 611/730 [54:14<10:37,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.7025219196948383e-06, 'epoch': 8.37}

 84%|████████▎ | 611/730 [54:14<10:37,  5.36s/it]
                                                 
{'debug/num_tok_total': 2364.0, 'debug/num_tok_loss': 1738.0, 'debug/num_lat_total': 2364.0, 'debug/num_lat_loss': 1738.0, 'epoch': 8.37}

 84%|████████▎ | 611/730 [54:15<10:37,  5.36s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.44703209400177, 'epoch': 8.37}

 84%|████████▎ | 611/730 [54:15<10:37,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.7025219196948383e-06, 'epoch': 8.37}

 84%|████████▎ | 611/730 [54:15<10:37,  5.36s/it]
 84%|████████▍ | 612/730 [54:15<10:26,  5.31s/it]
                                                 
{'debug/num_tok_total': 3110.0, 'debug/num_tok_loss': 1808.0, 'debug/num_lat_total': 3110.0, 'debug/num_lat_loss': 1808.0, 'epoch': 8.38}

 84%|████████▍ | 612/730 [54:16<10:26,  5.31s/it]
                                                 
{'train/ce_loss': 1.671875, 'train/diffusion_loss': 0.30769646167755127, 'epoch': 8.38}

 84%|████████▍ | 612/730 [54:16<10:26,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.6746824526945163e-06, 'epoch': 8.38}

 84%|████████▍ | 612/730 [54:16<10:26,  5.31s/it]
                                                 
{'debug/num_tok_total': 2433.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2433.0, 'debug/num_lat_loss': 1788.0, 'epoch': 8.38}

 84%|████████▍ | 612/730 [54:18<10:26,  5.31s/it]
                                                 
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.42125776410102844, 'epoch': 8.38}

 84%|████████▍ | 612/730 [54:18<10:26,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.6746824526945163e-06, 'epoch': 8.38}

 84%|████████▍ | 612/730 [54:18<10:26,  5.31s/it]
                                                 
{'debug/num_tok_total': 2652.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2652.0, 'debug/num_lat_loss': 1786.0, 'epoch': 8.38}

 84%|████████▍ | 612/730 [54:19<10:26,  5.31s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.40377336740493774, 'epoch': 8.38}

 84%|████████▍ | 612/730 [54:19<10:26,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.6746824526945163e-06, 'epoch': 8.38}

 84%|████████▍ | 612/730 [54:19<10:26,  5.31s/it]
                                                 
{'debug/num_tok_total': 2025.0, 'debug/num_tok_loss': 1807.0, 'debug/num_lat_total': 2025.0, 'debug/num_lat_loss': 1807.0, 'epoch': 8.38}

 84%|████████▍ | 612/730 [54:20<10:26,  5.31s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.5707200765609741, 'epoch': 8.38}

 84%|████████▍ | 612/730 [54:20<10:26,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.6746824526945163e-06, 'epoch': 8.38}

 84%|████████▍ | 612/730 [54:20<10:26,  5.31s/it]
 84%|████████▍ | 613/730 [54:20<10:16,  5.27s/it]
                                                 
{'debug/num_tok_total': 2871.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2871.0, 'debug/num_lat_loss': 1791.0, 'epoch': 8.4}

 84%|████████▍ | 613/730 [54:22<10:16,  5.27s/it]
                                                 
{'train/ce_loss': 1.65625, 'train/diffusion_loss': 0.3667501211166382, 'epoch': 8.4}

 84%|████████▍ | 613/730 [54:22<10:16,  5.27s/it]
                                                 
{'train/learning_rate_real': 1.6470561297189117e-06, 'epoch': 8.4}

 84%|████████▍ | 613/730 [54:22<10:16,  5.27s/it]
                                                 
{'debug/num_tok_total': 2680.0, 'debug/num_tok_loss': 1808.0, 'debug/num_lat_total': 2680.0, 'debug/num_lat_loss': 1808.0, 'epoch': 8.4}

 84%|████████▍ | 613/730 [54:23<10:16,  5.27s/it]
                                                 
{'train/ce_loss': 2.125, 'train/diffusion_loss': 0.3944864571094513, 'epoch': 8.4}

 84%|████████▍ | 613/730 [54:23<10:16,  5.27s/it]
                                                 
{'train/learning_rate_real': 1.6470561297189117e-06, 'epoch': 8.4}

 84%|████████▍ | 613/730 [54:23<10:16,  5.27s/it]
                                                 
{'debug/num_tok_total': 2838.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2838.0, 'debug/num_lat_loss': 1764.0, 'epoch': 8.4}

 84%|████████▍ | 613/730 [54:24<10:16,  5.27s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.32839787006378174, 'epoch': 8.4}

 84%|████████▍ | 613/730 [54:24<10:16,  5.27s/it]
                                                 
{'train/learning_rate_real': 1.6470561297189117e-06, 'epoch': 8.4}

 84%|████████▍ | 613/730 [54:24<10:16,  5.27s/it]
                                                 
{'debug/num_tok_total': 2712.0, 'debug/num_tok_loss': 1604.0, 'debug/num_lat_total': 2712.0, 'debug/num_lat_loss': 1604.0, 'epoch': 8.4}

 84%|████████▍ | 613/730 [54:25<10:16,  5.27s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.31409865617752075, 'epoch': 8.4}

 84%|████████▍ | 613/730 [54:25<10:16,  5.27s/it]
                                                 
{'train/learning_rate_real': 1.6470561297189117e-06, 'epoch': 8.4}

 84%|████████▍ | 613/730 [54:25<10:16,  5.27s/it]
 84%|████████▍ | 614/730 [54:26<10:15,  5.31s/it]
                                                 
{'debug/num_tok_total': 2281.0, 'debug/num_tok_loss': 1717.0, 'debug/num_lat_total': 2281.0, 'debug/num_lat_loss': 1717.0, 'epoch': 8.41}

 84%|████████▍ | 614/730 [54:27<10:15,  5.31s/it]
                                                 
{'train/ce_loss': 1.59375, 'train/diffusion_loss': 0.45770084857940674, 'epoch': 8.41}

 84%|████████▍ | 614/730 [54:27<10:15,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.6196434947137865e-06, 'epoch': 8.41}

 84%|████████▍ | 614/730 [54:27<10:15,  5.31s/it]
                                                 
{'debug/num_tok_total': 2407.0, 'debug/num_tok_loss': 1761.0, 'debug/num_lat_total': 2407.0, 'debug/num_lat_loss': 1761.0, 'epoch': 8.41}

 84%|████████▍ | 614/730 [54:28<10:15,  5.31s/it]
                                                 
{'train/ce_loss': 1.625, 'train/diffusion_loss': 0.43111199140548706, 'epoch': 8.41}

 84%|████████▍ | 614/730 [54:28<10:15,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.6196434947137865e-06, 'epoch': 8.41}

 84%|████████▍ | 614/730 [54:28<10:15,  5.31s/it]
                                                 
{'debug/num_tok_total': 2242.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2242.0, 'debug/num_lat_loss': 1796.0, 'epoch': 8.41}

 84%|████████▍ | 614/730 [54:29<10:15,  5.31s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.5470199584960938, 'epoch': 8.41}

 84%|████████▍ | 614/730 [54:29<10:15,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.6196434947137865e-06, 'epoch': 8.41}

 84%|████████▍ | 614/730 [54:29<10:15,  5.31s/it]
                                                 
{'debug/num_tok_total': 2430.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2430.0, 'debug/num_lat_loss': 1784.0, 'epoch': 8.41}

 84%|████████▍ | 614/730 [54:30<10:15,  5.31s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.45620134472846985, 'epoch': 8.41}

 84%|████████▍ | 614/730 [54:30<10:15,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.6196434947137865e-06, 'epoch': 8.41}

 84%|████████▍ | 614/730 [54:30<10:15,  5.31s/it]
 84%|████████▍ | 615/730 [54:31<09:59,  5.22s/it]
                                                 
{'debug/num_tok_total': 2840.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2840.0, 'debug/num_lat_loss': 1774.0, 'epoch': 8.42}

 84%|████████▍ | 615/730 [54:32<09:59,  5.22s/it]
                                                 
{'train/ce_loss': 1.625, 'train/diffusion_loss': 0.34811699390411377, 'epoch': 8.42}

 84%|████████▍ | 615/730 [54:32<09:59,  5.22s/it]
                                                 
{'train/learning_rate_real': 1.5924450874175211e-06, 'epoch': 8.42}

 84%|████████▍ | 615/730 [54:32<09:59,  5.22s/it]
                                                 
{'debug/num_tok_total': 2670.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2670.0, 'debug/num_lat_loss': 1794.0, 'epoch': 8.42}

 84%|████████▍ | 615/730 [54:33<09:59,  5.22s/it]
                                                 
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.4134746193885803, 'epoch': 8.42}

 84%|████████▍ | 615/730 [54:33<09:59,  5.22s/it]
                                                 
{'train/learning_rate_real': 1.5924450874175211e-06, 'epoch': 8.42}

 84%|████████▍ | 615/730 [54:33<09:59,  5.22s/it]
                                                 
{'debug/num_tok_total': 3294.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 3294.0, 'debug/num_lat_loss': 1790.0, 'epoch': 8.42}

 84%|████████▍ | 615/730 [54:35<09:59,  5.22s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.29358041286468506, 'epoch': 8.42}

 84%|████████▍ | 615/730 [54:35<09:59,  5.22s/it]
                                                 
{'train/learning_rate_real': 1.5924450874175211e-06, 'epoch': 8.42}

 84%|████████▍ | 615/730 [54:35<09:59,  5.22s/it]
                                                 
{'debug/num_tok_total': 2648.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2648.0, 'debug/num_lat_loss': 1791.0, 'epoch': 8.42}

 84%|████████▍ | 615/730 [54:36<09:59,  5.22s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.428499311208725, 'epoch': 8.42}

 84%|████████▍ | 615/730 [54:36<09:59,  5.22s/it]
                                                 
{'train/learning_rate_real': 1.5924450874175211e-06, 'epoch': 8.42}

 84%|████████▍ | 615/730 [54:36<09:59,  5.22s/it]
 84%|████████▍ | 616/730 [54:36<10:06,  5.32s/it]
                                                 
{'debug/num_tok_total': 2615.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2615.0, 'debug/num_lat_loss': 1770.0, 'epoch': 8.44}

 84%|████████▍ | 616/730 [54:37<10:06,  5.32s/it]
                                                 
{'train/ce_loss': 1.6875, 'train/diffusion_loss': 0.42929351329803467, 'epoch': 8.44}

 84%|████████▍ | 616/730 [54:37<10:06,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.5654614433504841e-06, 'epoch': 8.44}

 84%|████████▍ | 616/730 [54:37<10:06,  5.32s/it]
                                                 
{'debug/num_tok_total': 2464.0, 'debug/num_tok_loss': 1808.0, 'debug/num_lat_total': 2464.0, 'debug/num_lat_loss': 1808.0, 'epoch': 8.44}

 84%|████████▍ | 616/730 [54:39<10:06,  5.32s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.438287615776062, 'epoch': 8.44}

 84%|████████▍ | 616/730 [54:39<10:06,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.5654614433504841e-06, 'epoch': 8.44}

 84%|████████▍ | 616/730 [54:39<10:06,  5.32s/it]
                                                 
{'debug/num_tok_total': 2645.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2645.0, 'debug/num_lat_loss': 1788.0, 'epoch': 8.44}

 84%|████████▍ | 616/730 [54:40<10:06,  5.32s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.43230709433555603, 'epoch': 8.44}

 84%|████████▍ | 616/730 [54:40<10:06,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.5654614433504841e-06, 'epoch': 8.44}

 84%|████████▍ | 616/730 [54:40<10:06,  5.32s/it]
                                                 
{'debug/num_tok_total': 2512.0, 'debug/num_tok_loss': 1648.0, 'debug/num_lat_total': 2512.0, 'debug/num_lat_loss': 1648.0, 'epoch': 8.44}

 84%|████████▍ | 616/730 [54:41<10:06,  5.32s/it]
                                                 
{'train/ce_loss': 1.640625, 'train/diffusion_loss': 0.36854854226112366, 'epoch': 8.44}

 84%|████████▍ | 616/730 [54:41<10:06,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.5654614433504841e-06, 'epoch': 8.44}

 84%|████████▍ | 616/730 [54:41<10:06,  5.32s/it]
 85%|████████▍ | 617/730 [54:42<10:05,  5.36s/it]
                                                 
{'debug/num_tok_total': 2327.0, 'debug/num_tok_loss': 1619.0, 'debug/num_lat_total': 2327.0, 'debug/num_lat_loss': 1619.0, 'epoch': 8.45}

 85%|████████▍ | 617/730 [54:43<10:05,  5.36s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.411994993686676, 'epoch': 8.45}

 85%|████████▍ | 617/730 [54:43<10:05,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.5386930938044761e-06, 'epoch': 8.45}

 85%|████████▍ | 617/730 [54:43<10:05,  5.36s/it]
                                                 
{'debug/num_tok_total': 2690.0, 'debug/num_tok_loss': 1815.0, 'debug/num_lat_total': 2690.0, 'debug/num_lat_loss': 1815.0, 'epoch': 8.45}

 85%|████████▍ | 617/730 [54:44<10:05,  5.36s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.41209056973457336, 'epoch': 8.45}

 85%|████████▍ | 617/730 [54:44<10:05,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.5386930938044761e-06, 'epoch': 8.45}

 85%|████████▍ | 617/730 [54:44<10:05,  5.36s/it]
                                                 
{'debug/num_tok_total': 3069.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 3069.0, 'debug/num_lat_loss': 1781.0, 'epoch': 8.45}

 85%|████████▍ | 617/730 [54:45<10:05,  5.36s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.3358912765979767, 'epoch': 8.45}

 85%|████████▍ | 617/730 [54:45<10:05,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.5386930938044761e-06, 'epoch': 8.45}

 85%|████████▍ | 617/730 [54:45<10:05,  5.36s/it]
                                                 
{'debug/num_tok_total': 2377.0, 'debug/num_tok_loss': 1569.0, 'debug/num_lat_total': 2377.0, 'debug/num_lat_loss': 1569.0, 'epoch': 8.45}

 85%|████████▍ | 617/730 [54:47<10:05,  5.36s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.3772900700569153, 'epoch': 8.45}

 85%|████████▍ | 617/730 [54:47<10:05,  5.36s/it]
                                                 
{'train/learning_rate_real': 1.5386930938044761e-06, 'epoch': 8.45}

 85%|████████▍ | 617/730 [54:47<10:05,  5.36s/it]
 85%|████████▍ | 618/730 [54:47<10:06,  5.41s/it]
                                                 
{'debug/num_tok_total': 2631.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2631.0, 'debug/num_lat_loss': 1783.0, 'epoch': 8.47}

 85%|████████▍ | 618/730 [54:48<10:06,  5.41s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.4003202021121979, 'epoch': 8.47}

 85%|████████▍ | 618/730 [54:48<10:06,  5.41s/it]
                                                 
{'train/learning_rate_real': 1.5121405658322846e-06, 'epoch': 8.47}

 85%|████████▍ | 618/730 [54:48<10:06,  5.41s/it]
                                                 
{'debug/num_tok_total': 2906.0, 'debug/num_tok_loss': 1809.0, 'debug/num_lat_total': 2906.0, 'debug/num_lat_loss': 1809.0, 'epoch': 8.47}

 85%|████████▍ | 618/730 [54:50<10:06,  5.41s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.3741415739059448, 'epoch': 8.47}

 85%|████████▍ | 618/730 [54:50<10:06,  5.41s/it]
                                                 
{'train/learning_rate_real': 1.5121405658322846e-06, 'epoch': 8.47}

 85%|████████▍ | 618/730 [54:50<10:06,  5.41s/it]
                                                 
{'debug/num_tok_total': 2653.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 2653.0, 'debug/num_lat_loss': 1797.0, 'epoch': 8.47}

 85%|████████▍ | 618/730 [54:51<10:06,  5.41s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.4078856408596039, 'epoch': 8.47}

 85%|████████▍ | 618/730 [54:51<10:06,  5.41s/it]
                                                 
{'train/learning_rate_real': 1.5121405658322846e-06, 'epoch': 8.47}

 85%|████████▍ | 618/730 [54:51<10:06,  5.41s/it]
                                                 
{'debug/num_tok_total': 2670.0, 'debug/num_tok_loss': 1606.0, 'debug/num_lat_total': 2670.0, 'debug/num_lat_loss': 1606.0, 'epoch': 8.47}

 85%|████████▍ | 618/730 [54:52<10:06,  5.41s/it]
                                                 
{'train/ce_loss': 1.640625, 'train/diffusion_loss': 0.33983469009399414, 'epoch': 8.47}

 85%|████████▍ | 618/730 [54:52<10:06,  5.41s/it]
                                                 
{'train/learning_rate_real': 1.5121405658322846e-06, 'epoch': 8.47}

 85%|████████▍ | 618/730 [54:52<10:06,  5.41s/it]
 85%|████████▍ | 619/730 [54:53<10:07,  5.47s/it]
                                                 
{'debug/num_tok_total': 2169.0, 'debug/num_tok_loss': 1750.0, 'debug/num_lat_total': 2169.0, 'debug/num_lat_loss': 1750.0, 'epoch': 8.48}

 85%|████████▍ | 619/730 [54:54<10:07,  5.47s/it]
                                                 
{'train/ce_loss': 1.9921875, 'train/diffusion_loss': 0.4759816527366638, 'epoch': 8.48}

 85%|████████▍ | 619/730 [54:54<10:07,  5.47s/it]
                                                 
{'train/learning_rate_real': 1.485804382237292e-06, 'epoch': 8.48}

 85%|████████▍ | 619/730 [54:54<10:07,  5.47s/it]
                                                 
{'debug/num_tok_total': 2837.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2837.0, 'debug/num_lat_loss': 1775.0, 'epoch': 8.48}

 85%|████████▍ | 619/730 [54:55<10:07,  5.47s/it]
                                                 
{'train/ce_loss': 1.59375, 'train/diffusion_loss': 0.3642684519290924, 'epoch': 8.48}

 85%|████████▍ | 619/730 [54:55<10:07,  5.47s/it]
                                                 
{'train/learning_rate_real': 1.485804382237292e-06, 'epoch': 8.48}

 85%|████████▍ | 619/730 [54:55<10:07,  5.47s/it]
                                                 
{'debug/num_tok_total': 2848.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2848.0, 'debug/num_lat_loss': 1781.0, 'epoch': 8.48}

 85%|████████▍ | 619/730 [54:56<10:07,  5.47s/it]
                                                 
{'train/ce_loss': 1.8828125, 'train/diffusion_loss': 0.3782137334346771, 'epoch': 8.48}

 85%|████████▍ | 619/730 [54:56<10:07,  5.47s/it]
                                                 
{'train/learning_rate_real': 1.485804382237292e-06, 'epoch': 8.48}

 85%|████████▍ | 619/730 [54:56<10:07,  5.47s/it]
                                                 
{'debug/num_tok_total': 2906.0, 'debug/num_tok_loss': 1813.0, 'debug/num_lat_total': 2906.0, 'debug/num_lat_loss': 1813.0, 'epoch': 8.48}

 85%|████████▍ | 619/730 [54:58<10:07,  5.47s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.3582521677017212, 'epoch': 8.48}

 85%|████████▍ | 619/730 [54:58<10:07,  5.47s/it]
                                                 
{'train/learning_rate_real': 1.485804382237292e-06, 'epoch': 8.48}

 85%|████████▍ | 619/730 [54:58<10:07,  5.47s/it]03/16/2026 07:42:30 - INFO - __main__ - LoRA debug step 620: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 85%|████████▍ | 620/730 [54:58<09:56,  5.42s/it]
                                                 
{'loss': 2.5538, 'grad_norm': 1.6053745746612549, 'learning_rate': 1.485804382237292e-06, 'epoch': 8.49}

 85%|████████▍ | 620/730 [54:58<09:56,  5.42s/it]
                                                 
{'debug/num_tok_total': 2669.0, 'debug/num_tok_loss': 1606.0, 'debug/num_lat_total': 2669.0, 'debug/num_lat_loss': 1606.0, 'epoch': 8.49}

 85%|████████▍ | 620/730 [54:59<09:56,  5.42s/it]
                                                 
{'train/ce_loss': 1.59375, 'train/diffusion_loss': 0.3526284396648407, 'epoch': 8.49}

 85%|████████▍ | 620/730 [54:59<09:56,  5.42s/it]
                                                 
{'train/learning_rate_real': 1.4596850615631984e-06, 'epoch': 8.49}

 85%|████████▍ | 620/730 [54:59<09:56,  5.42s/it]
                                                 
{'debug/num_tok_total': 2795.0, 'debug/num_tok_loss': 1744.0, 'debug/num_lat_total': 2795.0, 'debug/num_lat_loss': 1744.0, 'epoch': 8.49}

 85%|████████▍ | 620/730 [55:01<09:56,  5.42s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.3609633445739746, 'epoch': 8.49}

 85%|████████▍ | 620/730 [55:01<09:56,  5.42s/it]
                                                 
{'train/learning_rate_real': 1.4596850615631984e-06, 'epoch': 8.49}

 85%|████████▍ | 620/730 [55:01<09:56,  5.42s/it]
                                                 
{'debug/num_tok_total': 2677.0, 'debug/num_tok_loss': 1810.0, 'debug/num_lat_total': 2677.0, 'debug/num_lat_loss': 1810.0, 'epoch': 8.49}

 85%|████████▍ | 620/730 [55:02<09:56,  5.42s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.40720343589782715, 'epoch': 8.49}

 85%|████████▍ | 620/730 [55:02<09:56,  5.42s/it]
                                                 
{'train/learning_rate_real': 1.4596850615631984e-06, 'epoch': 8.49}

 85%|████████▍ | 620/730 [55:02<09:56,  5.42s/it]
                                                 
{'debug/num_tok_total': 3073.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 3073.0, 'debug/num_lat_loss': 1783.0, 'epoch': 8.49}

 85%|████████▍ | 620/730 [55:03<09:56,  5.42s/it]
                                                 
{'train/ce_loss': 1.6015625, 'train/diffusion_loss': 0.32336628437042236, 'epoch': 8.49}

 85%|████████▍ | 620/730 [55:03<09:56,  5.42s/it]
                                                 
{'train/learning_rate_real': 1.4596850615631984e-06, 'epoch': 8.49}

 85%|████████▍ | 620/730 [55:03<09:56,  5.42s/it]
 85%|████████▌ | 621/730 [55:04<09:53,  5.45s/it]
                                                 
{'debug/num_tok_total': 3037.0, 'debug/num_tok_loss': 1755.0, 'debug/num_lat_total': 3037.0, 'debug/num_lat_loss': 1755.0, 'epoch': 8.51}

 85%|████████▌ | 621/730 [55:05<09:53,  5.45s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.30525854229927063, 'epoch': 8.51}

 85%|████████▌ | 621/730 [55:05<09:53,  5.45s/it]
                                                 
{'train/learning_rate_real': 1.433783118083798e-06, 'epoch': 8.51}

 85%|████████▌ | 621/730 [55:05<09:53,  5.45s/it]
                                                 
{'debug/num_tok_total': 2766.0, 'debug/num_tok_loss': 1622.0, 'debug/num_lat_total': 2766.0, 'debug/num_lat_loss': 1622.0, 'epoch': 8.51}

 85%|████████▌ | 621/730 [55:06<09:53,  5.45s/it]
                                                 
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.3523417115211487, 'epoch': 8.51}

 85%|████████▌ | 621/730 [55:06<09:53,  5.45s/it]
                                                 
{'train/learning_rate_real': 1.433783118083798e-06, 'epoch': 8.51}

 85%|████████▌ | 621/730 [55:06<09:53,  5.45s/it]
                                                 
{'debug/num_tok_total': 2011.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2011.0, 'debug/num_lat_loss': 1786.0, 'epoch': 8.51}

 85%|████████▌ | 621/730 [55:07<09:53,  5.45s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.5052626729011536, 'epoch': 8.51}

 85%|████████▌ | 621/730 [55:07<09:53,  5.45s/it]
                                                 
{'train/learning_rate_real': 1.433783118083798e-06, 'epoch': 8.51}

 85%|████████▌ | 621/730 [55:07<09:53,  5.45s/it]
                                                 
{'debug/num_tok_total': 2862.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2862.0, 'debug/num_lat_loss': 1788.0, 'epoch': 8.51}

 85%|████████▌ | 621/730 [55:09<09:53,  5.45s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.35968250036239624, 'epoch': 8.51}

 85%|████████▌ | 621/730 [55:09<09:53,  5.45s/it]
                                                 
{'train/learning_rate_real': 1.433783118083798e-06, 'epoch': 8.51}

 85%|████████▌ | 621/730 [55:09<09:53,  5.45s/it]
 85%|████████▌ | 622/730 [55:09<09:41,  5.39s/it]
                                                 
{'debug/num_tok_total': 2624.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2624.0, 'debug/num_lat_loss': 1780.0, 'epoch': 8.52}

 85%|████████▌ | 622/730 [55:10<09:41,  5.39s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.4193119704723358, 'epoch': 8.52}

 85%|████████▌ | 622/730 [55:10<09:41,  5.39s/it]
                                                 
{'train/learning_rate_real': 1.4080990617928571e-06, 'epoch': 8.52}

 85%|████████▌ | 622/730 [55:10<09:41,  5.39s/it]
                                                 
{'debug/num_tok_total': 2616.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2616.0, 'debug/num_lat_loss': 1775.0, 'epoch': 8.52}

 85%|████████▌ | 622/730 [55:11<09:41,  5.39s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.4078620374202728, 'epoch': 8.52}

 85%|████████▌ | 622/730 [55:11<09:41,  5.39s/it]
                                                 
{'train/learning_rate_real': 1.4080990617928571e-06, 'epoch': 8.52}

 85%|████████▌ | 622/730 [55:11<09:41,  5.39s/it]
                                                 
{'debug/num_tok_total': 2433.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2433.0, 'debug/num_lat_loss': 1789.0, 'epoch': 8.52}

 85%|████████▌ | 622/730 [55:13<09:41,  5.39s/it]
                                                 
{'train/ce_loss': 1.8984375, 'train/diffusion_loss': 0.46615880727767944, 'epoch': 8.52}

 85%|████████▌ | 622/730 [55:13<09:41,  5.39s/it]
                                                 
{'train/learning_rate_real': 1.4080990617928571e-06, 'epoch': 8.52}

 85%|████████▌ | 622/730 [55:13<09:41,  5.39s/it]
                                                 
{'debug/num_tok_total': 2408.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 2408.0, 'debug/num_lat_loss': 1768.0, 'epoch': 8.52}

 85%|████████▌ | 622/730 [55:14<09:41,  5.39s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.4513102173805237, 'epoch': 8.52}

 85%|████████▌ | 622/730 [55:14<09:41,  5.39s/it]
                                                 
{'train/learning_rate_real': 1.4080990617928571e-06, 'epoch': 8.52}

 85%|████████▌ | 622/730 [55:14<09:41,  5.39s/it]
 85%|████████▌ | 623/730 [55:14<09:32,  5.35s/it]
                                                 
{'debug/num_tok_total': 2672.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2672.0, 'debug/num_lat_loss': 1795.0, 'epoch': 8.53}

 85%|████████▌ | 623/730 [55:15<09:32,  5.35s/it]
                                                 
{'train/ce_loss': 1.578125, 'train/diffusion_loss': 0.3767714500427246, 'epoch': 8.53}

 85%|████████▌ | 623/730 [55:15<09:32,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.3826333983940731e-06, 'epoch': 8.53}

 85%|████████▌ | 623/730 [55:15<09:32,  5.35s/it]
                                                 
{'debug/num_tok_total': 2482.0, 'debug/num_tok_loss': 1658.0, 'debug/num_lat_total': 2482.0, 'debug/num_lat_loss': 1658.0, 'epoch': 8.53}

 85%|████████▌ | 623/730 [55:17<09:32,  5.35s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.3981763422489166, 'epoch': 8.53}

 85%|████████▌ | 623/730 [55:17<09:32,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.3826333983940731e-06, 'epoch': 8.53}

 85%|████████▌ | 623/730 [55:17<09:32,  5.35s/it]
                                                 
{'debug/num_tok_total': 2045.0, 'debug/num_tok_loss': 1821.0, 'debug/num_lat_total': 2045.0, 'debug/num_lat_loss': 1821.0, 'epoch': 8.53}

 85%|████████▌ | 623/730 [55:18<09:32,  5.35s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.4929209053516388, 'epoch': 8.53}

 85%|████████▌ | 623/730 [55:18<09:32,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.3826333983940731e-06, 'epoch': 8.53}

 85%|████████▌ | 623/730 [55:18<09:32,  5.35s/it]
                                                 
{'debug/num_tok_total': 2858.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2858.0, 'debug/num_lat_loss': 1787.0, 'epoch': 8.53}

 85%|████████▌ | 623/730 [55:19<09:32,  5.35s/it]
                                                 
{'train/ce_loss': 1.921875, 'train/diffusion_loss': 0.4135373830795288, 'epoch': 8.53}

 85%|████████▌ | 623/730 [55:19<09:32,  5.35s/it]
                                                 
{'train/learning_rate_real': 1.3826333983940731e-06, 'epoch': 8.53}

 85%|████████▌ | 623/730 [55:19<09:32,  5.35s/it]
 85%|████████▌ | 624/730 [55:19<09:21,  5.30s/it]
                                                 
{'debug/num_tok_total': 1985.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 1985.0, 'debug/num_lat_loss': 1777.0, 'epoch': 8.55}

 85%|████████▌ | 624/730 [55:20<09:21,  5.30s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.522341787815094, 'epoch': 8.55}

 85%|████████▌ | 624/730 [55:20<09:21,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.3573866292911214e-06, 'epoch': 8.55}

 85%|████████▌ | 624/730 [55:20<09:21,  5.30s/it]
                                                 
{'debug/num_tok_total': 2470.0, 'debug/num_tok_loss': 1812.0, 'debug/num_lat_total': 2470.0, 'debug/num_lat_loss': 1812.0, 'epoch': 8.55}

 85%|████████▌ | 624/730 [55:22<09:21,  5.30s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.4510863721370697, 'epoch': 8.55}

 85%|████████▌ | 624/730 [55:22<09:21,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.3573866292911214e-06, 'epoch': 8.55}

 85%|████████▌ | 624/730 [55:22<09:21,  5.30s/it]
                                                 
{'debug/num_tok_total': 2818.0, 'debug/num_tok_loss': 1753.0, 'debug/num_lat_total': 2818.0, 'debug/num_lat_loss': 1753.0, 'epoch': 8.55}

 85%|████████▌ | 624/730 [55:23<09:21,  5.30s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.3420361578464508, 'epoch': 8.55}

 85%|████████▌ | 624/730 [55:23<09:21,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.3573866292911214e-06, 'epoch': 8.55}

 85%|████████▌ | 624/730 [55:23<09:21,  5.30s/it]
                                                 
{'debug/num_tok_total': 2002.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2002.0, 'debug/num_lat_loss': 1779.0, 'epoch': 8.55}

 85%|████████▌ | 624/730 [55:24<09:21,  5.30s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.5534858703613281, 'epoch': 8.55}

 85%|████████▌ | 624/730 [55:24<09:21,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.3573866292911214e-06, 'epoch': 8.55}

 85%|████████▌ | 624/730 [55:24<09:21,  5.30s/it]
 86%|████████▌ | 625/730 [55:25<09:13,  5.27s/it]
                                                 
{'debug/num_tok_total': 2477.0, 'debug/num_tok_loss': 1808.0, 'debug/num_lat_total': 2477.0, 'debug/num_lat_loss': 1808.0, 'epoch': 8.56}

 86%|████████▌ | 625/730 [55:26<09:13,  5.27s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.44980549812316895, 'epoch': 8.56}

 86%|████████▌ | 625/730 [55:26<09:13,  5.27s/it]
                                                 
{'train/learning_rate_real': 1.3323592515777805e-06, 'epoch': 8.56}

 86%|████████▌ | 625/730 [55:26<09:13,  5.27s/it]
                                                 
{'debug/num_tok_total': 2709.0, 'debug/num_tok_loss': 1705.0, 'debug/num_lat_total': 2709.0, 'debug/num_lat_loss': 1705.0, 'epoch': 8.56}

 86%|████████▌ | 625/730 [55:27<09:13,  5.27s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.37328094244003296, 'epoch': 8.56}

 86%|████████▌ | 625/730 [55:27<09:13,  5.27s/it]
                                                 
{'train/learning_rate_real': 1.3323592515777805e-06, 'epoch': 8.56}

 86%|████████▌ | 625/730 [55:27<09:13,  5.27s/it]
                                                 
{'debug/num_tok_total': 2601.0, 'debug/num_tok_loss': 1757.0, 'debug/num_lat_total': 2601.0, 'debug/num_lat_loss': 1757.0, 'epoch': 8.56}

 86%|████████▌ | 625/730 [55:28<09:13,  5.27s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.40748217701911926, 'epoch': 8.56}

 86%|████████▌ | 625/730 [55:28<09:13,  5.27s/it]
                                                 
{'train/learning_rate_real': 1.3323592515777805e-06, 'epoch': 8.56}

 86%|████████▌ | 625/730 [55:28<09:13,  5.27s/it]
                                                 
{'debug/num_tok_total': 2848.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2848.0, 'debug/num_lat_loss': 1771.0, 'epoch': 8.56}

 86%|████████▌ | 625/730 [55:30<09:13,  5.27s/it]
                                                 
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.3469734489917755, 'epoch': 8.56}

 86%|████████▌ | 625/730 [55:30<09:13,  5.27s/it]
                                                 
{'train/learning_rate_real': 1.3323592515777805e-06, 'epoch': 8.56}

 86%|████████▌ | 625/730 [55:30<09:13,  5.27s/it]
 86%|████████▌ | 626/730 [55:30<09:12,  5.31s/it]
                                                 
{'debug/num_tok_total': 2864.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2864.0, 'debug/num_lat_loss': 1781.0, 'epoch': 8.58}

 86%|████████▌ | 626/730 [55:31<09:12,  5.31s/it]
                                                 
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.3811911642551422, 'epoch': 8.58}

 86%|████████▌ | 626/730 [55:31<09:12,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.3075517580281375e-06, 'epoch': 8.58}

 86%|████████▌ | 626/730 [55:31<09:12,  5.31s/it]
                                                 
{'debug/num_tok_total': 2435.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2435.0, 'debug/num_lat_loss': 1778.0, 'epoch': 8.58}

 86%|████████▌ | 626/730 [55:32<09:12,  5.31s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.42199403047561646, 'epoch': 8.58}

 86%|████████▌ | 626/730 [55:32<09:12,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.3075517580281375e-06, 'epoch': 8.58}

 86%|████████▌ | 626/730 [55:32<09:12,  5.31s/it]
                                                 
{'debug/num_tok_total': 2211.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2211.0, 'debug/num_lat_loss': 1781.0, 'epoch': 8.58}

 86%|████████▌ | 626/730 [55:34<09:12,  5.31s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.4914396107196808, 'epoch': 8.58}

 86%|████████▌ | 626/730 [55:34<09:12,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.3075517580281375e-06, 'epoch': 8.58}

 86%|████████▌ | 626/730 [55:34<09:12,  5.31s/it]
                                                 
{'debug/num_tok_total': 2407.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2407.0, 'debug/num_lat_loss': 1766.0, 'epoch': 8.58}

 86%|████████▌ | 626/730 [55:35<09:12,  5.31s/it]
                                                 
{'train/ce_loss': 1.6015625, 'train/diffusion_loss': 0.44562891125679016, 'epoch': 8.58}

 86%|████████▌ | 626/730 [55:35<09:12,  5.31s/it]
                                                 
{'train/learning_rate_real': 1.3075517580281375e-06, 'epoch': 8.58}

 86%|████████▌ | 626/730 [55:35<09:12,  5.31s/it]
 86%|████████▌ | 627/730 [55:35<09:02,  5.27s/it]
                                                 
{'debug/num_tok_total': 2462.0, 'debug/num_tok_loss': 1807.0, 'debug/num_lat_total': 2462.0, 'debug/num_lat_loss': 1807.0, 'epoch': 8.59}

 86%|████████▌ | 627/730 [55:36<09:02,  5.27s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.4585142731666565, 'epoch': 8.59}

 86%|████████▌ | 627/730 [55:36<09:02,  5.27s/it]
                                                 
{'train/learning_rate_real': 1.282964637086899e-06, 'epoch': 8.59}

 86%|████████▌ | 627/730 [55:36<09:02,  5.27s/it]
                                                 
{'debug/num_tok_total': 2846.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2846.0, 'debug/num_lat_loss': 1776.0, 'epoch': 8.59}

 86%|████████▌ | 627/730 [55:38<09:02,  5.27s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.35692599415779114, 'epoch': 8.59}

 86%|████████▌ | 627/730 [55:38<09:02,  5.27s/it]
                                                 
{'train/learning_rate_real': 1.282964637086899e-06, 'epoch': 8.59}

 86%|████████▌ | 627/730 [55:38<09:02,  5.27s/it]
                                                 
{'debug/num_tok_total': 2441.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2441.0, 'debug/num_lat_loss': 1782.0, 'epoch': 8.59}

 86%|████████▌ | 627/730 [55:39<09:02,  5.27s/it]
                                                 
{'train/ce_loss': 1.9609375, 'train/diffusion_loss': 0.42437732219696045, 'epoch': 8.59}

 86%|████████▌ | 627/730 [55:39<09:02,  5.27s/it]
                                                 
{'train/learning_rate_real': 1.282964637086899e-06, 'epoch': 8.59}

 86%|████████▌ | 627/730 [55:39<09:02,  5.27s/it]
                                                 
{'debug/num_tok_total': 2047.0, 'debug/num_tok_loss': 1623.0, 'debug/num_lat_total': 2047.0, 'debug/num_lat_loss': 1623.0, 'epoch': 8.59}

 86%|████████▌ | 627/730 [55:40<09:02,  5.27s/it]
                                                 
{'train/ce_loss': 1.5390625, 'train/diffusion_loss': 0.4605744779109955, 'epoch': 8.59}

 86%|████████▌ | 627/730 [55:40<09:02,  5.27s/it]
                                                 
{'train/learning_rate_real': 1.282964637086899e-06, 'epoch': 8.59}

 86%|████████▌ | 627/730 [55:40<09:02,  5.27s/it]
 86%|████████▌ | 628/730 [55:40<08:53,  5.23s/it]
                                                 
{'debug/num_tok_total': 2661.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2661.0, 'debug/num_lat_loss': 1795.0, 'epoch': 8.6}

 86%|████████▌ | 628/730 [55:41<08:53,  5.23s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.3910449743270874, 'epoch': 8.6}

 86%|████████▌ | 628/730 [55:41<08:53,  5.23s/it]
                                                 
{'train/learning_rate_real': 1.2585983728597608e-06, 'epoch': 8.6}

 86%|████████▌ | 628/730 [55:41<08:53,  5.23s/it]
                                                 
{'debug/num_tok_total': 2812.0, 'debug/num_tok_loss': 1762.0, 'debug/num_lat_total': 2812.0, 'debug/num_lat_loss': 1762.0, 'epoch': 8.6}

 86%|████████▌ | 628/730 [55:43<08:53,  5.23s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.3531990945339203, 'epoch': 8.6}

 86%|████████▌ | 628/730 [55:43<08:53,  5.23s/it]
                                                 
{'train/learning_rate_real': 1.2585983728597608e-06, 'epoch': 8.6}

 86%|████████▌ | 628/730 [55:43<08:53,  5.23s/it]
                                                 
{'debug/num_tok_total': 3114.0, 'debug/num_tok_loss': 1810.0, 'debug/num_lat_total': 3114.0, 'debug/num_lat_loss': 1810.0, 'epoch': 8.6}

 86%|████████▌ | 628/730 [55:44<08:53,  5.23s/it]
                                                 
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.2955913543701172, 'epoch': 8.6}

 86%|████████▌ | 628/730 [55:44<08:53,  5.23s/it]
                                                 
{'train/learning_rate_real': 1.2585983728597608e-06, 'epoch': 8.6}

 86%|████████▌ | 628/730 [55:44<08:53,  5.23s/it]
                                                 
{'debug/num_tok_total': 2610.0, 'debug/num_tok_loss': 1760.0, 'debug/num_lat_total': 2610.0, 'debug/num_lat_loss': 1760.0, 'epoch': 8.6}

 86%|████████▌ | 628/730 [55:45<08:53,  5.23s/it]
                                                 
{'train/ce_loss': 1.59375, 'train/diffusion_loss': 0.38715896010398865, 'epoch': 8.6}

 86%|████████▌ | 628/730 [55:45<08:53,  5.23s/it]
                                                 
{'train/learning_rate_real': 1.2585983728597608e-06, 'epoch': 8.6}

 86%|████████▌ | 628/730 [55:45<08:53,  5.23s/it]
 86%|████████▌ | 629/730 [55:46<08:57,  5.32s/it]
                                                 
{'debug/num_tok_total': 2204.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2204.0, 'debug/num_lat_loss': 1770.0, 'epoch': 8.62}

 86%|████████▌ | 629/730 [55:47<08:57,  5.32s/it]
                                                 
{'train/ce_loss': 1.9140625, 'train/diffusion_loss': 0.4739186465740204, 'epoch': 8.62}

 86%|████████▌ | 629/730 [55:47<08:57,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.234453445103885e-06, 'epoch': 8.62}

 86%|████████▌ | 629/730 [55:47<08:57,  5.32s/it]
                                                 
{'debug/num_tok_total': 2435.0, 'debug/num_tok_loss': 1599.0, 'debug/num_lat_total': 2435.0, 'debug/num_lat_loss': 1599.0, 'epoch': 8.62}

 86%|████████▌ | 629/730 [55:48<08:57,  5.32s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.380092978477478, 'epoch': 8.62}

 86%|████████▌ | 629/730 [55:48<08:57,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.234453445103885e-06, 'epoch': 8.62}

 86%|████████▌ | 629/730 [55:48<08:57,  5.32s/it]
                                                 
{'debug/num_tok_total': 3127.0, 'debug/num_tok_loss': 1804.0, 'debug/num_lat_total': 3127.0, 'debug/num_lat_loss': 1804.0, 'epoch': 8.62}

 86%|████████▌ | 629/730 [55:49<08:57,  5.32s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.3197096884250641, 'epoch': 8.62}

 86%|████████▌ | 629/730 [55:49<08:57,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.234453445103885e-06, 'epoch': 8.62}

 86%|████████▌ | 629/730 [55:49<08:57,  5.32s/it]
                                                 
{'debug/num_tok_total': 2652.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2652.0, 'debug/num_lat_loss': 1780.0, 'epoch': 8.62}

 86%|████████▌ | 629/730 [55:51<08:57,  5.32s/it]
                                                 
{'train/ce_loss': 1.6875, 'train/diffusion_loss': 0.3955904543399811, 'epoch': 8.62}

 86%|████████▌ | 629/730 [55:51<08:57,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.234453445103885e-06, 'epoch': 8.62}

 86%|████████▌ | 629/730 [55:51<08:57,  5.32s/it]03/16/2026 07:43:23 - INFO - __main__ - LoRA debug step 630: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 86%|████████▋ | 630/730 [55:51<08:49,  5.29s/it]
                                                 
{'loss': 2.559, 'grad_norm': 1.4953088760375977, 'learning_rate': 1.234453445103885e-06, 'epoch': 8.63}

 86%|████████▋ | 630/730 [55:51<08:49,  5.29s/it]
                                                 
{'debug/num_tok_total': 2206.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2206.0, 'debug/num_lat_loss': 1776.0, 'epoch': 8.63}

 86%|████████▋ | 630/730 [55:52<08:49,  5.29s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.4772806465625763, 'epoch': 8.63}

 86%|████████▋ | 630/730 [55:52<08:49,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.2105303292184552e-06, 'epoch': 8.63}

 86%|████████▋ | 630/730 [55:52<08:49,  5.29s/it]
                                                 
{'debug/num_tok_total': 2422.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2422.0, 'debug/num_lat_loss': 1778.0, 'epoch': 8.63}

 86%|████████▋ | 630/730 [55:53<08:49,  5.29s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.4359516203403473, 'epoch': 8.63}

 86%|████████▋ | 630/730 [55:53<08:49,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.2105303292184552e-06, 'epoch': 8.63}

 86%|████████▋ | 630/730 [55:53<08:49,  5.29s/it]
                                                 
{'debug/num_tok_total': 3125.0, 'debug/num_tok_loss': 1804.0, 'debug/num_lat_total': 3125.0, 'debug/num_lat_loss': 1804.0, 'epoch': 8.63}

 86%|████████▋ | 630/730 [55:55<08:49,  5.29s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.3467935621738434, 'epoch': 8.63}

 86%|████████▋ | 630/730 [55:55<08:49,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.2105303292184552e-06, 'epoch': 8.63}

 86%|████████▋ | 630/730 [55:55<08:49,  5.29s/it]
                                                 
{'debug/num_tok_total': 2602.0, 'debug/num_tok_loss': 1756.0, 'debug/num_lat_total': 2602.0, 'debug/num_lat_loss': 1756.0, 'epoch': 8.63}

 86%|████████▋ | 630/730 [55:56<08:49,  5.29s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.39922434091567993, 'epoch': 8.63}

 86%|████████▋ | 630/730 [55:56<08:49,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.2105303292184552e-06, 'epoch': 8.63}

 86%|████████▋ | 630/730 [55:56<08:49,  5.29s/it]
 86%|████████▋ | 631/730 [55:56<08:44,  5.30s/it]
                                                 
{'debug/num_tok_total': 2869.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2869.0, 'debug/num_lat_loss': 1789.0, 'epoch': 8.64}

 86%|████████▋ | 631/730 [55:58<08:44,  5.30s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.36225059628486633, 'epoch': 8.64}

 86%|████████▋ | 631/730 [55:58<08:44,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.1868294962353033e-06, 'epoch': 8.64}

 86%|████████▋ | 631/730 [55:58<08:44,  5.30s/it]
                                                 
{'debug/num_tok_total': 2531.0, 'debug/num_tok_loss': 1623.0, 'debug/num_lat_total': 2531.0, 'debug/num_lat_loss': 1623.0, 'epoch': 8.64}

 86%|████████▋ | 631/730 [55:59<08:44,  5.30s/it]
                                                 
{'train/ce_loss': 1.890625, 'train/diffusion_loss': 0.3591125011444092, 'epoch': 8.64}

 86%|████████▋ | 631/730 [55:59<08:44,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.1868294962353033e-06, 'epoch': 8.64}

 86%|████████▋ | 631/730 [55:59<08:44,  5.30s/it]
                                                 
{'debug/num_tok_total': 2018.0, 'debug/num_tok_loss': 1809.0, 'debug/num_lat_total': 2018.0, 'debug/num_lat_loss': 1809.0, 'epoch': 8.64}

 86%|████████▋ | 631/730 [56:00<08:44,  5.30s/it]
                                                 
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.5320108532905579, 'epoch': 8.64}

 86%|████████▋ | 631/730 [56:00<08:44,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.1868294962353033e-06, 'epoch': 8.64}

 86%|████████▋ | 631/730 [56:00<08:44,  5.30s/it]
                                                 
{'debug/num_tok_total': 2601.0, 'debug/num_tok_loss': 1752.0, 'debug/num_lat_total': 2601.0, 'debug/num_lat_loss': 1752.0, 'epoch': 8.64}

 86%|████████▋ | 631/730 [56:01<08:44,  5.30s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.4067121148109436, 'epoch': 8.64}

 86%|████████▋ | 631/730 [56:01<08:44,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.1868294962353033e-06, 'epoch': 8.64}

 86%|████████▋ | 631/730 [56:01<08:44,  5.30s/it]
 87%|████████▋ | 632/730 [56:02<08:38,  5.29s/it]
                                                 
{'debug/num_tok_total': 2388.0, 'debug/num_tok_loss': 1760.0, 'debug/num_lat_total': 2388.0, 'debug/num_lat_loss': 1760.0, 'epoch': 8.66}

 87%|████████▋ | 632/730 [56:03<08:38,  5.29s/it]
                                                 
{'train/ce_loss': 1.640625, 'train/diffusion_loss': 0.44997096061706543, 'epoch': 8.66}

 87%|████████▋ | 632/730 [56:03<08:38,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.1633514128096518e-06, 'epoch': 8.66}

 87%|████████▋ | 632/730 [56:03<08:38,  5.29s/it]
                                                 
{'debug/num_tok_total': 2870.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2870.0, 'debug/num_lat_loss': 1786.0, 'epoch': 8.66}

 87%|████████▋ | 632/730 [56:04<08:38,  5.29s/it]
                                                 
{'train/ce_loss': 1.6640625, 'train/diffusion_loss': 0.35176968574523926, 'epoch': 8.66}

 87%|████████▋ | 632/730 [56:04<08:38,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.1633514128096518e-06, 'epoch': 8.66}

 87%|████████▋ | 632/730 [56:04<08:38,  5.29s/it]
                                                 
{'debug/num_tok_total': 2215.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2215.0, 'debug/num_lat_loss': 1784.0, 'epoch': 8.66}

 87%|████████▋ | 632/730 [56:05<08:38,  5.29s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.48681339621543884, 'epoch': 8.66}

 87%|████████▋ | 632/730 [56:05<08:38,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.1633514128096518e-06, 'epoch': 8.66}

 87%|████████▋ | 632/730 [56:05<08:38,  5.29s/it]
                                                 
{'debug/num_tok_total': 3089.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 3089.0, 'debug/num_lat_loss': 1790.0, 'epoch': 8.66}

 87%|████████▋ | 632/730 [56:06<08:38,  5.29s/it]
                                                 
{'train/ce_loss': 1.59375, 'train/diffusion_loss': 0.30530035495758057, 'epoch': 8.66}

 87%|████████▋ | 632/730 [56:06<08:38,  5.29s/it]
                                                 
{'train/learning_rate_real': 1.1633514128096518e-06, 'epoch': 8.66}

 87%|████████▋ | 632/730 [56:06<08:38,  5.29s/it]
 87%|████████▋ | 633/730 [56:07<08:29,  5.26s/it]
                                                 
{'debug/num_tok_total': 2728.0, 'debug/num_tok_loss': 1611.0, 'debug/num_lat_total': 2728.0, 'debug/num_lat_loss': 1611.0, 'epoch': 8.67}

 87%|████████▋ | 633/730 [56:08<08:29,  5.26s/it]
                                                 
{'train/ce_loss': 1.640625, 'train/diffusion_loss': 0.29804953932762146, 'epoch': 8.67}

 87%|████████▋ | 633/730 [56:08<08:29,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.1400965412109097e-06, 'epoch': 8.67}

 87%|████████▋ | 633/730 [56:08<08:29,  5.26s/it]
                                                 
{'debug/num_tok_total': 1633.0, 'debug/num_tok_loss': 1597.0, 'debug/num_lat_total': 1633.0, 'debug/num_lat_loss': 1597.0, 'epoch': 8.67}

 87%|████████▋ | 633/730 [56:09<08:29,  5.26s/it]
                                                 
{'train/ce_loss': 1.9296875, 'train/diffusion_loss': 0.5920611619949341, 'epoch': 8.67}

 87%|████████▋ | 633/730 [56:09<08:29,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.1400965412109097e-06, 'epoch': 8.67}

 87%|████████▋ | 633/730 [56:09<08:29,  5.26s/it]
                                                 
{'debug/num_tok_total': 3129.0, 'debug/num_tok_loss': 1815.0, 'debug/num_lat_total': 3129.0, 'debug/num_lat_loss': 1815.0, 'epoch': 8.67}

 87%|████████▋ | 633/730 [56:10<08:29,  5.26s/it]
                                                 
{'train/ce_loss': 1.671875, 'train/diffusion_loss': 0.29410862922668457, 'epoch': 8.67}

 87%|████████▋ | 633/730 [56:10<08:29,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.1400965412109097e-06, 'epoch': 8.67}

 87%|████████▋ | 633/730 [56:10<08:29,  5.26s/it]
                                                 
{'debug/num_tok_total': 1989.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 1989.0, 'debug/num_lat_loss': 1782.0, 'epoch': 8.67}

 87%|████████▋ | 633/730 [56:12<08:29,  5.26s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.5416147112846375, 'epoch': 8.67}

 87%|████████▋ | 633/730 [56:12<08:29,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.1400965412109097e-06, 'epoch': 8.67}

 87%|████████▋ | 633/730 [56:12<08:29,  5.26s/it]
 87%|████████▋ | 634/730 [56:12<08:19,  5.20s/it]
                                                 
{'debug/num_tok_total': 2212.0, 'debug/num_tok_loss': 1452.0, 'debug/num_lat_total': 2212.0, 'debug/num_lat_loss': 1452.0, 'epoch': 8.68}

 87%|████████▋ | 634/730 [56:13<08:19,  5.20s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.372467041015625, 'epoch': 8.68}

 87%|████████▋ | 634/730 [56:13<08:19,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.1170653393135847e-06, 'epoch': 8.68}

 87%|████████▋ | 634/730 [56:13<08:19,  5.20s/it]
                                                 
{'debug/num_tok_total': 2468.0, 'debug/num_tok_loss': 1608.0, 'debug/num_lat_total': 2468.0, 'debug/num_lat_loss': 1608.0, 'epoch': 8.68}

 87%|████████▋ | 634/730 [56:14<08:19,  5.20s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.4156140983104706, 'epoch': 8.68}

 87%|████████▋ | 634/730 [56:14<08:19,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.1170653393135847e-06, 'epoch': 8.68}

 87%|████████▋ | 634/730 [56:14<08:19,  5.20s/it]
                                                 
{'debug/num_tok_total': 2888.0, 'debug/num_tok_loss': 1808.0, 'debug/num_lat_total': 2888.0, 'debug/num_lat_loss': 1808.0, 'epoch': 8.68}

 87%|████████▋ | 634/730 [56:16<08:19,  5.20s/it]
                                                 
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.37275758385658264, 'epoch': 8.68}

 87%|████████▋ | 634/730 [56:16<08:19,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.1170653393135847e-06, 'epoch': 8.68}

 87%|████████▋ | 634/730 [56:16<08:19,  5.20s/it]
                                                 
{'debug/num_tok_total': 2382.0, 'debug/num_tok_loss': 1749.0, 'debug/num_lat_total': 2382.0, 'debug/num_lat_loss': 1749.0, 'epoch': 8.68}

 87%|████████▋ | 634/730 [56:17<08:19,  5.20s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.45669183135032654, 'epoch': 8.68}

 87%|████████▋ | 634/730 [56:17<08:19,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.1170653393135847e-06, 'epoch': 8.68}

 87%|████████▋ | 634/730 [56:17<08:19,  5.20s/it]
 87%|████████▋ | 635/730 [56:17<08:14,  5.20s/it]
                                                 
{'debug/num_tok_total': 2638.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2638.0, 'debug/num_lat_loss': 1780.0, 'epoch': 8.7}

 87%|████████▋ | 635/730 [56:18<08:14,  5.20s/it]
                                                 
{'train/ce_loss': 2.015625, 'train/diffusion_loss': 0.3974115550518036, 'epoch': 8.7}

 87%|████████▋ | 635/730 [56:18<08:14,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.0942582605882657e-06, 'epoch': 8.7}

 87%|████████▋ | 635/730 [56:18<08:14,  5.20s/it]
                                                 
{'debug/num_tok_total': 2450.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2450.0, 'debug/num_lat_loss': 1799.0, 'epoch': 8.7}

 87%|████████▋ | 635/730 [56:19<08:14,  5.20s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.4809345602989197, 'epoch': 8.7}

 87%|████████▋ | 635/730 [56:19<08:14,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.0942582605882657e-06, 'epoch': 8.7}

 87%|████████▋ | 635/730 [56:19<08:14,  5.20s/it]
                                                 
{'debug/num_tok_total': 2622.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2622.0, 'debug/num_lat_loss': 1781.0, 'epoch': 8.7}

 87%|████████▋ | 635/730 [56:21<08:14,  5.20s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.40356576442718506, 'epoch': 8.7}

 87%|████████▋ | 635/730 [56:21<08:14,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.0942582605882657e-06, 'epoch': 8.7}

 87%|████████▋ | 635/730 [56:21<08:14,  5.20s/it]
                                                 
{'debug/num_tok_total': 3082.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 3082.0, 'debug/num_lat_loss': 1781.0, 'epoch': 8.7}

 87%|████████▋ | 635/730 [56:22<08:14,  5.20s/it]
                                                 
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.29570019245147705, 'epoch': 8.7}

 87%|████████▋ | 635/730 [56:22<08:14,  5.20s/it]
                                                 
{'train/learning_rate_real': 1.0942582605882657e-06, 'epoch': 8.7}

 87%|████████▋ | 635/730 [56:22<08:14,  5.20s/it]
 87%|████████▋ | 636/730 [56:23<08:14,  5.26s/it]
                                                 
{'debug/num_tok_total': 2378.0, 'debug/num_tok_loss': 1751.0, 'debug/num_lat_total': 2378.0, 'debug/num_lat_loss': 1751.0, 'epoch': 8.71}

 87%|████████▋ | 636/730 [56:24<08:14,  5.26s/it]
                                                 
{'train/ce_loss': 1.6484375, 'train/diffusion_loss': 0.46681028604507446, 'epoch': 8.71}

 87%|████████▋ | 636/730 [56:24<08:14,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.0716757540926798e-06, 'epoch': 8.71}

 87%|████████▋ | 636/730 [56:24<08:14,  5.26s/it]
                                                 
{'debug/num_tok_total': 2658.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2658.0, 'debug/num_lat_loss': 1790.0, 'epoch': 8.71}

 87%|████████▋ | 636/730 [56:25<08:14,  5.26s/it]
                                                 
{'train/ce_loss': 1.65625, 'train/diffusion_loss': 0.4088594913482666, 'epoch': 8.71}

 87%|████████▋ | 636/730 [56:25<08:14,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.0716757540926798e-06, 'epoch': 8.71}

 87%|████████▋ | 636/730 [56:25<08:14,  5.26s/it]
                                                 
{'debug/num_tok_total': 3068.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 3068.0, 'debug/num_lat_loss': 1774.0, 'epoch': 8.71}

 87%|████████▋ | 636/730 [56:26<08:14,  5.26s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.31941258907318115, 'epoch': 8.71}

 87%|████████▋ | 636/730 [56:26<08:14,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.0716757540926798e-06, 'epoch': 8.71}

 87%|████████▋ | 636/730 [56:26<08:14,  5.26s/it]
                                                 
{'debug/num_tok_total': 2846.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2846.0, 'debug/num_lat_loss': 1777.0, 'epoch': 8.71}

 87%|████████▋ | 636/730 [56:27<08:14,  5.26s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.3816400170326233, 'epoch': 8.71}

 87%|████████▋ | 636/730 [56:28<08:14,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.0716757540926798e-06, 'epoch': 8.71}

 87%|████████▋ | 636/730 [56:28<08:14,  5.26s/it]
 87%|████████▋ | 637/730 [56:28<08:12,  5.30s/it]
                                                 
{'debug/num_tok_total': 2192.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 2192.0, 'debug/num_lat_loss': 1769.0, 'epoch': 8.73}

 87%|████████▋ | 637/730 [56:29<08:12,  5.30s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.5032124519348145, 'epoch': 8.73}

 87%|████████▋ | 637/730 [56:29<08:12,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.049318264462866e-06, 'epoch': 8.73}

 87%|████████▋ | 637/730 [56:29<08:12,  5.30s/it]
                                                 
{'debug/num_tok_total': 2615.0, 'debug/num_tok_loss': 1761.0, 'debug/num_lat_total': 2615.0, 'debug/num_lat_loss': 1761.0, 'epoch': 8.73}

 87%|████████▋ | 637/730 [56:30<08:12,  5.30s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.39580604434013367, 'epoch': 8.73}

 87%|████████▋ | 637/730 [56:30<08:12,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.049318264462866e-06, 'epoch': 8.73}

 87%|████████▋ | 637/730 [56:30<08:12,  5.30s/it]
                                                 
{'debug/num_tok_total': 2849.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2849.0, 'debug/num_lat_loss': 1776.0, 'epoch': 8.73}

 87%|████████▋ | 637/730 [56:31<08:12,  5.30s/it]
                                                 
{'train/ce_loss': 1.6484375, 'train/diffusion_loss': 0.3767808973789215, 'epoch': 8.73}

 87%|████████▋ | 637/730 [56:31<08:12,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.049318264462866e-06, 'epoch': 8.73}

 87%|████████▋ | 637/730 [56:31<08:12,  5.30s/it]
                                                 
{'debug/num_tok_total': 2607.0, 'debug/num_tok_loss': 1660.0, 'debug/num_lat_total': 2607.0, 'debug/num_lat_loss': 1660.0, 'epoch': 8.73}

 87%|████████▋ | 637/730 [56:33<08:12,  5.30s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.3772682249546051, 'epoch': 8.73}

 87%|████████▋ | 637/730 [56:33<08:12,  5.30s/it]
                                                 
{'train/learning_rate_real': 1.049318264462866e-06, 'epoch': 8.73}

 87%|████████▋ | 637/730 [56:33<08:12,  5.30s/it]
 87%|████████▋ | 638/730 [56:33<08:09,  5.32s/it]
                                                 
{'debug/num_tok_total': 2901.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2901.0, 'debug/num_lat_loss': 1802.0, 'epoch': 8.74}

 87%|████████▋ | 638/730 [56:34<08:09,  5.32s/it]
                                                 
{'train/ce_loss': 2.03125, 'train/diffusion_loss': 0.34852996468544006, 'epoch': 8.74}

 87%|████████▋ | 638/730 [56:34<08:09,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.0271862319044173e-06, 'epoch': 8.74}

 87%|████████▋ | 638/730 [56:34<08:09,  5.32s/it]
                                                 
{'debug/num_tok_total': 2418.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2418.0, 'debug/num_lat_loss': 1770.0, 'epoch': 8.74}

 87%|████████▋ | 638/730 [56:36<08:09,  5.32s/it]
                                                 
{'train/ce_loss': 1.5859375, 'train/diffusion_loss': 0.43168318271636963, 'epoch': 8.74}

 87%|████████▋ | 638/730 [56:36<08:09,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.0271862319044173e-06, 'epoch': 8.74}

 87%|████████▋ | 638/730 [56:36<08:09,  5.32s/it]
                                                 
{'debug/num_tok_total': 2637.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2637.0, 'debug/num_lat_loss': 1784.0, 'epoch': 8.74}

 87%|████████▋ | 638/730 [56:37<08:09,  5.32s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.42986050248146057, 'epoch': 8.74}

 87%|████████▋ | 638/730 [56:37<08:09,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.0271862319044173e-06, 'epoch': 8.74}

 87%|████████▋ | 638/730 [56:37<08:09,  5.32s/it]
                                                 
{'debug/num_tok_total': 2211.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2211.0, 'debug/num_lat_loss': 1770.0, 'epoch': 8.74}

 87%|████████▋ | 638/730 [56:38<08:09,  5.32s/it]
                                                 
{'train/ce_loss': 1.96875, 'train/diffusion_loss': 0.5003586411476135, 'epoch': 8.74}

 87%|████████▋ | 638/730 [56:38<08:09,  5.32s/it]
                                                 
{'train/learning_rate_real': 1.0271862319044173e-06, 'epoch': 8.74}

 87%|████████▋ | 638/730 [56:38<08:09,  5.32s/it]
 88%|████████▊ | 639/730 [56:38<07:59,  5.26s/it]
                                                 
{'debug/num_tok_total': 2202.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2202.0, 'debug/num_lat_loss': 1775.0, 'epoch': 8.75}

 88%|████████▊ | 639/730 [56:40<07:59,  5.26s/it]
                                                 
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.4796827435493469, 'epoch': 8.75}

 88%|████████▊ | 639/730 [56:40<07:59,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.0052800921838135e-06, 'epoch': 8.75}

 88%|████████▊ | 639/730 [56:40<07:59,  5.26s/it]
                                                 
{'debug/num_tok_total': 2186.0, 'debug/num_tok_loss': 1760.0, 'debug/num_lat_total': 2186.0, 'debug/num_lat_loss': 1760.0, 'epoch': 8.75}

 88%|████████▊ | 639/730 [56:41<07:59,  5.26s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.4834350347518921, 'epoch': 8.75}

 88%|████████▊ | 639/730 [56:41<07:59,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.0052800921838135e-06, 'epoch': 8.75}

 88%|████████▊ | 639/730 [56:41<07:59,  5.26s/it]
                                                 
{'debug/num_tok_total': 2845.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2845.0, 'debug/num_lat_loss': 1772.0, 'epoch': 8.75}

 88%|████████▊ | 639/730 [56:42<07:59,  5.26s/it]
                                                 
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.3429874777793884, 'epoch': 8.75}

 88%|████████▊ | 639/730 [56:42<07:59,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.0052800921838135e-06, 'epoch': 8.75}

 88%|████████▊ | 639/730 [56:42<07:59,  5.26s/it]
                                                 
{'debug/num_tok_total': 2374.0, 'debug/num_tok_loss': 1732.0, 'debug/num_lat_total': 2374.0, 'debug/num_lat_loss': 1732.0, 'epoch': 8.75}

 88%|████████▊ | 639/730 [56:43<07:59,  5.26s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.4393598139286041, 'epoch': 8.75}

 88%|████████▊ | 639/730 [56:43<07:59,  5.26s/it]
                                                 
{'train/learning_rate_real': 1.0052800921838135e-06, 'epoch': 8.75}

 88%|████████▊ | 639/730 [56:43<07:59,  5.26s/it]03/16/2026 07:44:16 - INFO - __main__ - LoRA debug step 640: changed A 196/196, changed B 195/196, lora_B_zero_now=0.

 88%|████████▊ | 640/730 [56:44<07:53,  5.26s/it]
                                                 
{'loss': 2.5936, 'grad_norm': 1.2918602228164673, 'learning_rate': 1.0052800921838135e-06, 'epoch': 8.77}

 88%|████████▊ | 640/730 [56:44<07:53,  5.26s/it]
                                                 
{'debug/num_tok_total': 2868.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2868.0, 'debug/num_lat_loss': 1780.0, 'epoch': 8.77}

 88%|████████▊ | 640/730 [56:45<07:53,  5.26s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.37705162167549133, 'epoch': 8.77}

 88%|████████▊ | 640/730 [56:45<07:53,  5.26s/it]
                                                 
{'train/learning_rate_real': 9.836002766198379e-07, 'epoch': 8.77}

 88%|████████▊ | 640/730 [56:45<07:53,  5.26s/it]
                                                 
{'debug/num_tok_total': 2213.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2213.0, 'debug/num_lat_loss': 1779.0, 'epoch': 8.77}

 88%|████████▊ | 640/730 [56:46<07:53,  5.26s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.4920317828655243, 'epoch': 8.77}

 88%|████████▊ | 640/730 [56:46<07:53,  5.26s/it]
                                                 
{'train/learning_rate_real': 9.836002766198379e-07, 'epoch': 8.77}

 88%|████████▊ | 640/730 [56:46<07:53,  5.26s/it]
                                                 
{'debug/num_tok_total': 2218.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2218.0, 'debug/num_lat_loss': 1792.0, 'epoch': 8.77}

 88%|████████▊ | 640/730 [56:47<07:53,  5.26s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.47286900877952576, 'epoch': 8.77}

 88%|████████▊ | 640/730 [56:47<07:53,  5.26s/it]
                                                 
{'train/learning_rate_real': 9.836002766198379e-07, 'epoch': 8.77}

 88%|████████▊ | 640/730 [56:47<07:53,  5.26s/it]
                                                 
{'debug/num_tok_total': 2667.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2667.0, 'debug/num_lat_loss': 1796.0, 'epoch': 8.77}

 88%|████████▊ | 640/730 [56:48<07:53,  5.26s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.4107883870601654, 'epoch': 8.77}

 88%|████████▊ | 640/730 [56:48<07:53,  5.26s/it]
                                                 
{'train/learning_rate_real': 9.836002766198379e-07, 'epoch': 8.77}

 88%|████████▊ | 640/730 [56:48<07:53,  5.26s/it]
 88%|████████▊ | 641/730 [56:49<07:44,  5.22s/it]
                                                 
{'debug/num_tok_total': 1986.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 1986.0, 'debug/num_lat_loss': 1775.0, 'epoch': 8.78}

 88%|████████▊ | 641/730 [56:50<07:44,  5.22s/it]
                                                 
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.5198059678077698, 'epoch': 8.78}

 88%|████████▊ | 641/730 [56:50<07:44,  5.22s/it]
                                                 
{'train/learning_rate_real': 9.621472120750846e-07, 'epoch': 8.78}

 88%|████████▊ | 641/730 [56:50<07:44,  5.22s/it]
                                                 
{'debug/num_tok_total': 2650.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2650.0, 'debug/num_lat_loss': 1783.0, 'epoch': 8.78}

 88%|████████▊ | 641/730 [56:51<07:44,  5.22s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.40174102783203125, 'epoch': 8.78}

 88%|████████▊ | 641/730 [56:51<07:44,  5.22s/it]
                                                 
{'train/learning_rate_real': 9.621472120750846e-07, 'epoch': 8.78}

 88%|████████▊ | 641/730 [56:51<07:44,  5.22s/it]
                                                 
{'debug/num_tok_total': 2231.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2231.0, 'debug/num_lat_loss': 1790.0, 'epoch': 8.78}

 88%|████████▊ | 641/730 [56:52<07:44,  5.22s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.47102487087249756, 'epoch': 8.78}

 88%|████████▊ | 641/730 [56:52<07:44,  5.22s/it]
                                                 
{'train/learning_rate_real': 9.621472120750846e-07, 'epoch': 8.78}

 88%|████████▊ | 641/730 [56:52<07:44,  5.22s/it]
                                                 
{'debug/num_tok_total': 2480.0, 'debug/num_tok_loss': 1815.0, 'debug/num_lat_total': 2480.0, 'debug/num_lat_loss': 1815.0, 'epoch': 8.78}

 88%|████████▊ | 641/730 [56:53<07:44,  5.22s/it]
                                                 
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.4528915584087372, 'epoch': 8.78}

 88%|████████▊ | 641/730 [56:53<07:44,  5.22s/it]
                                                 
{'train/learning_rate_real': 9.621472120750846e-07, 'epoch': 8.78}

 88%|████████▊ | 641/730 [56:53<07:44,  5.22s/it]
 88%|████████▊ | 642/730 [56:54<07:34,  5.16s/it]
                                                 
{'debug/num_tok_total': 2661.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2661.0, 'debug/num_lat_loss': 1796.0, 'epoch': 8.79}

 88%|████████▊ | 642/730 [56:55<07:34,  5.16s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.3909485936164856, 'epoch': 8.79}

 88%|████████▊ | 642/730 [56:55<07:34,  5.16s/it]
                                                 
{'train/learning_rate_real': 9.40921320947552e-07, 'epoch': 8.79}

 88%|████████▊ | 642/730 [56:55<07:34,  5.16s/it]
                                                 
{'debug/num_tok_total': 2838.0, 'debug/num_tok_loss': 1735.0, 'debug/num_lat_total': 2838.0, 'debug/num_lat_loss': 1735.0, 'epoch': 8.79}

 88%|████████▊ | 642/730 [56:56<07:34,  5.16s/it]
                                                 
{'train/ce_loss': 1.8828125, 'train/diffusion_loss': 0.35042327642440796, 'epoch': 8.79}

 88%|████████▊ | 642/730 [56:56<07:34,  5.16s/it]
                                                 
{'train/learning_rate_real': 9.40921320947552e-07, 'epoch': 8.79}

 88%|████████▊ | 642/730 [56:56<07:34,  5.16s/it]
                                                 
{'debug/num_tok_total': 2637.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2637.0, 'debug/num_lat_loss': 1784.0, 'epoch': 8.79}

 88%|████████▊ | 642/730 [56:58<07:34,  5.16s/it]
                                                 
{'train/ce_loss': 1.6875, 'train/diffusion_loss': 0.4298247694969177, 'epoch': 8.79}

 88%|████████▊ | 642/730 [56:58<07:34,  5.16s/it]
                                                 
{'train/learning_rate_real': 9.40921320947552e-07, 'epoch': 8.79}

 88%|████████▊ | 642/730 [56:58<07:34,  5.16s/it]
                                                 
{'debug/num_tok_total': 2199.0, 'debug/num_tok_loss': 1759.0, 'debug/num_lat_total': 2199.0, 'debug/num_lat_loss': 1759.0, 'epoch': 8.79}

 88%|████████▊ | 642/730 [56:59<07:34,  5.16s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.46796005964279175, 'epoch': 8.79}

 88%|████████▊ | 642/730 [56:59<07:34,  5.16s/it]
                                                 
{'train/learning_rate_real': 9.40921320947552e-07, 'epoch': 8.79}

 88%|████████▊ | 642/730 [56:59<07:34,  5.16s/it]
 88%|████████▊ | 643/730 [56:59<07:34,  5.22s/it]
                                                 
{'debug/num_tok_total': 2534.0, 'debug/num_tok_loss': 1667.0, 'debug/num_lat_total': 2534.0, 'debug/num_lat_loss': 1667.0, 'epoch': 8.81}

 88%|████████▊ | 643/730 [57:00<07:34,  5.22s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.3840373754501343, 'epoch': 8.81}

 88%|████████▊ | 643/730 [57:00<07:34,  5.22s/it]
                                                 
{'train/learning_rate_real': 9.19923021162343e-07, 'epoch': 8.81}

 88%|████████▊ | 643/730 [57:00<07:34,  5.22s/it]
                                                 
{'debug/num_tok_total': 2459.0, 'debug/num_tok_loss': 1598.0, 'debug/num_lat_total': 2459.0, 'debug/num_lat_loss': 1598.0, 'epoch': 8.81}

 88%|████████▊ | 643/730 [57:02<07:34,  5.22s/it]
                                                 
{'train/ce_loss': 1.6640625, 'train/diffusion_loss': 0.38765934109687805, 'epoch': 8.81}

 88%|████████▊ | 643/730 [57:02<07:34,  5.22s/it]
                                                 
{'train/learning_rate_real': 9.19923021162343e-07, 'epoch': 8.81}

 88%|████████▊ | 643/730 [57:02<07:34,  5.22s/it]
                                                 
{'debug/num_tok_total': 2642.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2642.0, 'debug/num_lat_loss': 1787.0, 'epoch': 8.81}

 88%|████████▊ | 643/730 [57:03<07:34,  5.22s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.40039485692977905, 'epoch': 8.81}

 88%|████████▊ | 643/730 [57:03<07:34,  5.22s/it]
                                                 
{'train/learning_rate_real': 9.19923021162343e-07, 'epoch': 8.81}

 88%|████████▊ | 643/730 [57:03<07:34,  5.22s/it]
                                                 
{'debug/num_tok_total': 2630.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2630.0, 'debug/num_lat_loss': 1781.0, 'epoch': 8.81}

 88%|████████▊ | 643/730 [57:04<07:34,  5.22s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.4165422022342682, 'epoch': 8.81}

 88%|████████▊ | 643/730 [57:04<07:34,  5.22s/it]
                                                 
{'train/learning_rate_real': 9.19923021162343e-07, 'epoch': 8.81}

 88%|████████▊ | 643/730 [57:04<07:34,  5.22s/it]
 88%|████████▊ | 644/730 [57:04<07:29,  5.23s/it]
                                                 
{'debug/num_tok_total': 2694.0, 'debug/num_tok_loss': 1818.0, 'debug/num_lat_total': 2694.0, 'debug/num_lat_loss': 1818.0, 'epoch': 8.82}

 88%|████████▊ | 644/730 [57:06<07:29,  5.23s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.37862253189086914, 'epoch': 8.82}

 88%|████████▊ | 644/730 [57:06<07:29,  5.23s/it]
                                                 
{'train/learning_rate_real': 8.99152726163413e-07, 'epoch': 8.82}

 88%|████████▊ | 644/730 [57:06<07:29,  5.23s/it]
                                                 
{'debug/num_tok_total': 2402.0, 'debug/num_tok_loss': 1762.0, 'debug/num_lat_total': 2402.0, 'debug/num_lat_loss': 1762.0, 'epoch': 8.82}

 88%|████████▊ | 644/730 [57:07<07:29,  5.23s/it]
                                                 
{'train/ce_loss': 2.0625, 'train/diffusion_loss': 0.4396543800830841, 'epoch': 8.82}

 88%|████████▊ | 644/730 [57:07<07:29,  5.23s/it]
                                                 
{'train/learning_rate_real': 8.99152726163413e-07, 'epoch': 8.82}

 88%|████████▊ | 644/730 [57:07<07:29,  5.23s/it]
                                                 
{'debug/num_tok_total': 2626.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2626.0, 'debug/num_lat_loss': 1770.0, 'epoch': 8.82}

 88%|████████▊ | 644/730 [57:08<07:29,  5.23s/it]
                                                 
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.4153808057308197, 'epoch': 8.82}

 88%|████████▊ | 644/730 [57:08<07:29,  5.23s/it]
                                                 
{'train/learning_rate_real': 8.99152726163413e-07, 'epoch': 8.82}

 88%|████████▊ | 644/730 [57:08<07:29,  5.23s/it]
                                                 
{'debug/num_tok_total': 2185.0, 'debug/num_tok_loss': 1754.0, 'debug/num_lat_total': 2185.0, 'debug/num_lat_loss': 1754.0, 'epoch': 8.82}

 88%|████████▊ | 644/730 [57:09<07:29,  5.23s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.47416505217552185, 'epoch': 8.82}

 88%|████████▊ | 644/730 [57:09<07:29,  5.23s/it]
                                                 
{'train/learning_rate_real': 8.99152726163413e-07, 'epoch': 8.82}

 88%|████████▊ | 644/730 [57:09<07:29,  5.23s/it]
 88%|████████▊ | 645/730 [57:10<07:23,  5.22s/it]
                                                 
{'debug/num_tok_total': 2686.0, 'debug/num_tok_loss': 1812.0, 'debug/num_lat_total': 2686.0, 'debug/num_lat_loss': 1812.0, 'epoch': 8.84}

 88%|████████▊ | 645/730 [57:11<07:23,  5.22s/it]
                                                 
{'train/ce_loss': 1.8828125, 'train/diffusion_loss': 0.4176783561706543, 'epoch': 8.84}

 88%|████████▊ | 645/730 [57:11<07:23,  5.22s/it]
                                                 
{'train/learning_rate_real': 8.786108449054379e-07, 'epoch': 8.84}

 88%|████████▊ | 645/730 [57:11<07:23,  5.22s/it]
                                                 
{'debug/num_tok_total': 2878.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2878.0, 'debug/num_lat_loss': 1802.0, 'epoch': 8.84}

 88%|████████▊ | 645/730 [57:12<07:23,  5.22s/it]
                                                 
{'train/ce_loss': 1.6171875, 'train/diffusion_loss': 0.36315006017684937, 'epoch': 8.84}

 88%|████████▊ | 645/730 [57:12<07:23,  5.22s/it]
                                                 
{'train/learning_rate_real': 8.786108449054379e-07, 'epoch': 8.84}

 88%|████████▊ | 645/730 [57:12<07:23,  5.22s/it]
                                                 
{'debug/num_tok_total': 2906.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2906.0, 'debug/num_lat_loss': 1799.0, 'epoch': 8.84}

 88%|████████▊ | 645/730 [57:13<07:23,  5.22s/it]
                                                 
{'train/ce_loss': 1.6328125, 'train/diffusion_loss': 0.37344083189964294, 'epoch': 8.84}

 88%|████████▊ | 645/730 [57:13<07:23,  5.22s/it]
                                                 
{'train/learning_rate_real': 8.786108449054379e-07, 'epoch': 8.84}

 88%|████████▊ | 645/730 [57:13<07:23,  5.22s/it]
                                                 
{'debug/num_tok_total': 2062.0, 'debug/num_tok_loss': 1554.0, 'debug/num_lat_total': 2062.0, 'debug/num_lat_loss': 1554.0, 'epoch': 8.84}

 88%|████████▊ | 645/730 [57:14<07:23,  5.22s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.4462355077266693, 'epoch': 8.84}

 88%|████████▊ | 645/730 [57:14<07:23,  5.22s/it]
                                                 
{'train/learning_rate_real': 8.786108449054379e-07, 'epoch': 8.84}

 88%|████████▊ | 645/730 [57:14<07:23,  5.22s/it]
 88%|████████▊ | 646/730 [57:15<07:23,  5.28s/it]
                                                 
{'debug/num_tok_total': 2863.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2863.0, 'debug/num_lat_loss': 1789.0, 'epoch': 8.85}

 88%|████████▊ | 646/730 [57:16<07:23,  5.28s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.3642660677433014, 'epoch': 8.85}

 88%|████████▊ | 646/730 [57:16<07:23,  5.28s/it]
                                                 
{'train/learning_rate_real': 8.582977818457696e-07, 'epoch': 8.85}

 88%|████████▊ | 646/730 [57:16<07:23,  5.28s/it]
                                                 
{'debug/num_tok_total': 2438.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2438.0, 'debug/num_lat_loss': 1791.0, 'epoch': 8.85}

 88%|████████▊ | 646/730 [57:18<07:23,  5.28s/it]
                                                 
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.46786195039749146, 'epoch': 8.85}

 88%|████████▊ | 646/730 [57:18<07:23,  5.28s/it]
                                                 
{'train/learning_rate_real': 8.582977818457696e-07, 'epoch': 8.85}

 88%|████████▊ | 646/730 [57:18<07:23,  5.28s/it]
                                                 
{'debug/num_tok_total': 2489.0, 'debug/num_tok_loss': 1496.0, 'debug/num_lat_total': 2489.0, 'debug/num_lat_loss': 1496.0, 'epoch': 8.85}

 88%|████████▊ | 646/730 [57:19<07:23,  5.28s/it]
                                                 
{'train/ce_loss': 1.578125, 'train/diffusion_loss': 0.33849436044692993, 'epoch': 8.85}

 88%|████████▊ | 646/730 [57:19<07:23,  5.28s/it]
                                                 
{'train/learning_rate_real': 8.582977818457696e-07, 'epoch': 8.85}

 88%|████████▊ | 646/730 [57:19<07:23,  5.28s/it]
                                                 
{'debug/num_tok_total': 2665.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 2665.0, 'debug/num_lat_loss': 1797.0, 'epoch': 8.85}

 88%|████████▊ | 646/730 [57:20<07:23,  5.28s/it]
                                                 
{'train/ce_loss': 1.6640625, 'train/diffusion_loss': 0.41925308108329773, 'epoch': 8.85}

 88%|████████▊ | 646/730 [57:20<07:23,  5.28s/it]
                                                 
{'train/learning_rate_real': 8.582977818457696e-07, 'epoch': 8.85}

 88%|████████▊ | 646/730 [57:20<07:23,  5.28s/it]
 89%|████████▊ | 647/730 [57:21<07:24,  5.36s/it]
                                                 
{'debug/num_tok_total': 3091.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 3091.0, 'debug/num_lat_loss': 1793.0, 'epoch': 8.86}

 89%|████████▊ | 647/730 [57:22<07:24,  5.36s/it]
                                                 
{'train/ce_loss': 1.59375, 'train/diffusion_loss': 0.29850855469703674, 'epoch': 8.86}

 89%|████████▊ | 647/730 [57:22<07:24,  5.36s/it]
                                                 
{'train/learning_rate_real': 8.382139369364647e-07, 'epoch': 8.86}

 89%|████████▊ | 647/730 [57:22<07:24,  5.36s/it]
                                                 
{'debug/num_tok_total': 2930.0, 'debug/num_tok_loss': 1823.0, 'debug/num_lat_total': 2930.0, 'debug/num_lat_loss': 1823.0, 'epoch': 8.86}

 89%|████████▊ | 647/730 [57:23<07:24,  5.36s/it]
                                                 
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.378746896982193, 'epoch': 8.86}

 89%|████████▊ | 647/730 [57:23<07:24,  5.36s/it]
                                                 
{'train/learning_rate_real': 8.382139369364647e-07, 'epoch': 8.86}

 89%|████████▊ | 647/730 [57:23<07:24,  5.36s/it]
                                                 
{'debug/num_tok_total': 2878.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 2878.0, 'debug/num_lat_loss': 1797.0, 'epoch': 8.86}

 89%|████████▊ | 647/730 [57:25<07:24,  5.36s/it]
                                                 
{'train/ce_loss': 1.6640625, 'train/diffusion_loss': 0.3531334698200226, 'epoch': 8.86}

 89%|████████▊ | 647/730 [57:25<07:24,  5.36s/it]
                                                 
{'train/learning_rate_real': 8.382139369364647e-07, 'epoch': 8.86}

 89%|████████▊ | 647/730 [57:25<07:24,  5.36s/it]
                                                 
{'debug/num_tok_total': 2523.0, 'debug/num_tok_loss': 1665.0, 'debug/num_lat_total': 2523.0, 'debug/num_lat_loss': 1665.0, 'epoch': 8.86}

 89%|████████▊ | 647/730 [57:26<07:24,  5.36s/it]
                                                 
{'train/ce_loss': 1.9296875, 'train/diffusion_loss': 0.36325815320014954, 'epoch': 8.86}

 89%|████████▊ | 647/730 [57:26<07:24,  5.36s/it]
                                                 
{'train/learning_rate_real': 8.382139369364647e-07, 'epoch': 8.86}

 89%|████████▊ | 647/730 [57:26<07:24,  5.36s/it]
 89%|████████▉ | 648/730 [57:26<07:25,  5.44s/it]
                                                 
{'debug/num_tok_total': 2912.0, 'debug/num_tok_loss': 1820.0, 'debug/num_lat_total': 2912.0, 'debug/num_lat_loss': 1820.0, 'epoch': 8.88}

 89%|████████▉ | 648/730 [57:27<07:25,  5.44s/it]
                                                 
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.34086906909942627, 'epoch': 8.88}

 89%|████████▉ | 648/730 [57:27<07:25,  5.44s/it]
                                                 
{'train/learning_rate_real': 8.183597056164144e-07, 'epoch': 8.88}

 89%|████████▉ | 648/730 [57:27<07:25,  5.44s/it]
                                                 
{'debug/num_tok_total': 2345.0, 'debug/num_tok_loss': 1737.0, 'debug/num_lat_total': 2345.0, 'debug/num_lat_loss': 1737.0, 'epoch': 8.88}

 89%|████████▉ | 648/730 [57:29<07:25,  5.44s/it]
                                                 
{'train/ce_loss': 1.671875, 'train/diffusion_loss': 0.4794709086418152, 'epoch': 8.88}

 89%|████████▉ | 648/730 [57:29<07:25,  5.44s/it]
                                                 
{'train/learning_rate_real': 8.183597056164144e-07, 'epoch': 8.88}

 89%|████████▉ | 648/730 [57:29<07:25,  5.44s/it]
                                                 
{'debug/num_tok_total': 2434.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2434.0, 'debug/num_lat_loss': 1784.0, 'epoch': 8.88}

 89%|████████▉ | 648/730 [57:30<07:25,  5.44s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.4393218457698822, 'epoch': 8.88}

 89%|████████▉ | 648/730 [57:30<07:25,  5.44s/it]
                                                 
{'train/learning_rate_real': 8.183597056164144e-07, 'epoch': 8.88}

 89%|████████▉ | 648/730 [57:30<07:25,  5.44s/it]
                                                 
{'debug/num_tok_total': 3251.0, 'debug/num_tok_loss': 1759.0, 'debug/num_lat_total': 3251.0, 'debug/num_lat_loss': 1759.0, 'epoch': 8.88}

 89%|████████▉ | 648/730 [57:31<07:25,  5.44s/it]
                                                 
{'train/ce_loss': 1.6875, 'train/diffusion_loss': 0.27202191948890686, 'epoch': 8.88}

 89%|████████▉ | 648/730 [57:31<07:25,  5.44s/it]
                                                 
{'train/learning_rate_real': 8.183597056164144e-07, 'epoch': 8.88}

 89%|████████▉ | 648/730 [57:31<07:25,  5.44s/it]
 89%|████████▉ | 649/730 [57:32<07:19,  5.42s/it]
                                                 
{'debug/num_tok_total': 2567.0, 'debug/num_tok_loss': 1704.0, 'debug/num_lat_total': 2567.0, 'debug/num_lat_loss': 1704.0, 'epoch': 8.89}

 89%|████████▉ | 649/730 [57:33<07:19,  5.42s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.44055119156837463, 'epoch': 8.89}

 89%|████████▉ | 649/730 [57:33<07:19,  5.42s/it]
                                                 
{'train/learning_rate_real': 7.9873547880355e-07, 'epoch': 8.89}

 89%|████████▉ | 649/730 [57:33<07:19,  5.42s/it]
                                                 
{'debug/num_tok_total': 2751.0, 'debug/num_tok_loss': 1690.0, 'debug/num_lat_total': 2751.0, 'debug/num_lat_loss': 1690.0, 'epoch': 8.89}

 89%|████████▉ | 649/730 [57:34<07:19,  5.42s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.3224574625492096, 'epoch': 8.89}

 89%|████████▉ | 649/730 [57:34<07:19,  5.42s/it]
                                                 
{'train/learning_rate_real': 7.9873547880355e-07, 'epoch': 8.89}

 89%|████████▉ | 649/730 [57:34<07:19,  5.42s/it]
                                                 
{'debug/num_tok_total': 2862.0, 'debug/num_tok_loss': 1682.0, 'debug/num_lat_total': 2862.0, 'debug/num_lat_loss': 1682.0, 'epoch': 8.89}

 89%|████████▉ | 649/730 [57:35<07:19,  5.42s/it]
                                                 
{'train/ce_loss': 1.671875, 'train/diffusion_loss': 0.3089441955089569, 'epoch': 8.89}

 89%|████████▉ | 649/730 [57:35<07:19,  5.42s/it]
                                                 
{'train/learning_rate_real': 7.9873547880355e-07, 'epoch': 8.89}

 89%|████████▉ | 649/730 [57:35<07:19,  5.42s/it]
                                                 
{'debug/num_tok_total': 2646.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2646.0, 'debug/num_lat_loss': 1792.0, 'epoch': 8.89}

 89%|████████▉ | 649/730 [57:37<07:19,  5.42s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.4025222361087799, 'epoch': 8.89}

 89%|████████▉ | 649/730 [57:37<07:19,  5.42s/it]
                                                 
{'train/learning_rate_real': 7.9873547880355e-07, 'epoch': 8.89}

 89%|████████▉ | 649/730 [57:37<07:19,  5.42s/it]03/16/2026 07:45:09 - INFO - __main__ - LoRA debug step 650: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 89%|████████▉ | 650/730 [57:37<07:12,  5.41s/it]
                                                 
{'loss': 2.5378, 'grad_norm': 1.286134958267212, 'learning_rate': 7.9873547880355e-07, 'epoch': 8.9}

 89%|████████▉ | 650/730 [57:37<07:12,  5.41s/it]
                                                 
{'debug/num_tok_total': 2849.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2849.0, 'debug/num_lat_loss': 1783.0, 'epoch': 8.9}

 89%|████████▉ | 650/730 [57:38<07:12,  5.41s/it]
                                                 
{'train/ce_loss': 1.8984375, 'train/diffusion_loss': 0.3567376732826233, 'epoch': 8.9}

 89%|████████▉ | 650/730 [57:38<07:12,  5.41s/it]
                                                 
{'train/learning_rate_real': 7.79341642887152e-07, 'epoch': 8.9}

 89%|████████▉ | 650/730 [57:38<07:12,  5.41s/it]
                                                 
{'debug/num_tok_total': 2589.0, 'debug/num_tok_loss': 1746.0, 'debug/num_lat_total': 2589.0, 'debug/num_lat_loss': 1746.0, 'epoch': 8.9}

 89%|████████▉ | 650/730 [57:39<07:12,  5.41s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.39021778106689453, 'epoch': 8.9}

 89%|████████▉ | 650/730 [57:40<07:12,  5.41s/it]
                                                 
{'train/learning_rate_real': 7.79341642887152e-07, 'epoch': 8.9}

 89%|████████▉ | 650/730 [57:40<07:12,  5.41s/it]
                                                 
{'debug/num_tok_total': 3129.0, 'debug/num_tok_loss': 1818.0, 'debug/num_lat_total': 3129.0, 'debug/num_lat_loss': 1818.0, 'epoch': 8.9}

 89%|████████▉ | 650/730 [57:41<07:12,  5.41s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.3153533339500427, 'epoch': 8.9}

 89%|████████▉ | 650/730 [57:41<07:12,  5.41s/it]
                                                 
{'train/learning_rate_real': 7.79341642887152e-07, 'epoch': 8.9}

 89%|████████▉ | 650/730 [57:41<07:12,  5.41s/it]
                                                 
{'debug/num_tok_total': 2669.0, 'debug/num_tok_loss': 1800.0, 'debug/num_lat_total': 2669.0, 'debug/num_lat_loss': 1800.0, 'epoch': 8.9}

 89%|████████▉ | 650/730 [57:42<07:12,  5.41s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.40469998121261597, 'epoch': 8.9}

 89%|████████▉ | 650/730 [57:42<07:12,  5.41s/it]
                                                 
{'train/learning_rate_real': 7.79341642887152e-07, 'epoch': 8.9}

 89%|████████▉ | 650/730 [57:42<07:12,  5.41s/it]
 89%|████████▉ | 651/730 [57:43<07:15,  5.52s/it]
                                                 
{'debug/num_tok_total': 2498.0, 'debug/num_tok_loss': 1599.0, 'debug/num_lat_total': 2498.0, 'debug/num_lat_loss': 1599.0, 'epoch': 8.92}

 89%|████████▉ | 651/730 [57:44<07:15,  5.52s/it]
                                                 
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.375074565410614, 'epoch': 8.92}

 89%|████████▉ | 651/730 [57:44<07:15,  5.52s/it]
                                                 
{'train/learning_rate_real': 7.601785797202471e-07, 'epoch': 8.92}

 89%|████████▉ | 651/730 [57:44<07:15,  5.52s/it]
                                                 
{'debug/num_tok_total': 1974.0, 'debug/num_tok_loss': 1762.0, 'debug/num_lat_total': 1974.0, 'debug/num_lat_loss': 1762.0, 'epoch': 8.92}

 89%|████████▉ | 651/730 [57:45<07:15,  5.52s/it]
                                                 
{'train/ce_loss': 1.6875, 'train/diffusion_loss': 0.5410798788070679, 'epoch': 8.92}

 89%|████████▉ | 651/730 [57:45<07:15,  5.52s/it]
                                                 
{'train/learning_rate_real': 7.601785797202471e-07, 'epoch': 8.92}

 89%|████████▉ | 651/730 [57:45<07:15,  5.52s/it]
                                                 
{'debug/num_tok_total': 2540.0, 'debug/num_tok_loss': 1469.0, 'debug/num_lat_total': 2540.0, 'debug/num_lat_loss': 1469.0, 'epoch': 8.92}

 89%|████████▉ | 651/730 [57:46<07:15,  5.52s/it]
                                                 
{'train/ce_loss': 1.65625, 'train/diffusion_loss': 0.29693081974983215, 'epoch': 8.92}

 89%|████████▉ | 651/730 [57:46<07:15,  5.52s/it]
                                                 
{'train/learning_rate_real': 7.601785797202471e-07, 'epoch': 8.92}

 89%|████████▉ | 651/730 [57:46<07:15,  5.52s/it]
                                                 
{'debug/num_tok_total': 2888.0, 'debug/num_tok_loss': 1800.0, 'debug/num_lat_total': 2888.0, 'debug/num_lat_loss': 1800.0, 'epoch': 8.92}

 89%|████████▉ | 651/730 [57:48<07:15,  5.52s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.3673584759235382, 'epoch': 8.92}

 89%|████████▉ | 651/730 [57:48<07:15,  5.52s/it]
                                                 
{'train/learning_rate_real': 7.601785797202471e-07, 'epoch': 8.92}

 89%|████████▉ | 651/730 [57:48<07:15,  5.52s/it]
 89%|████████▉ | 652/730 [57:48<07:04,  5.44s/it]
                                                 
{'debug/num_tok_total': 2441.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2441.0, 'debug/num_lat_loss': 1779.0, 'epoch': 8.93}

 89%|████████▉ | 652/730 [57:49<07:04,  5.44s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.44160377979278564, 'epoch': 8.93}

 89%|████████▉ | 652/730 [57:49<07:04,  5.44s/it]
                                                 
{'train/learning_rate_real': 7.412466666120846e-07, 'epoch': 8.93}

 89%|████████▉ | 652/730 [57:49<07:04,  5.44s/it]
                                                 
{'debug/num_tok_total': 2372.0, 'debug/num_tok_loss': 1738.0, 'debug/num_lat_total': 2372.0, 'debug/num_lat_loss': 1738.0, 'epoch': 8.93}

 89%|████████▉ | 652/730 [57:50<07:04,  5.44s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.43758684396743774, 'epoch': 8.93}

 89%|████████▉ | 652/730 [57:50<07:04,  5.44s/it]
                                                 
{'train/learning_rate_real': 7.412466666120846e-07, 'epoch': 8.93}

 89%|████████▉ | 652/730 [57:50<07:04,  5.44s/it]
                                                 
{'debug/num_tok_total': 2630.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2630.0, 'debug/num_lat_loss': 1771.0, 'epoch': 8.93}

 89%|████████▉ | 652/730 [57:52<07:04,  5.44s/it]
                                                 
{'train/ce_loss': 1.6640625, 'train/diffusion_loss': 0.3982909321784973, 'epoch': 8.93}

 89%|████████▉ | 652/730 [57:52<07:04,  5.44s/it]
                                                 
{'train/learning_rate_real': 7.412466666120846e-07, 'epoch': 8.93}

 89%|████████▉ | 652/730 [57:52<07:04,  5.44s/it]
                                                 
{'debug/num_tok_total': 2365.0, 'debug/num_tok_loss': 1733.0, 'debug/num_lat_total': 2365.0, 'debug/num_lat_loss': 1733.0, 'epoch': 8.93}

 89%|████████▉ | 652/730 [57:53<07:04,  5.44s/it]
                                                 
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.4682052731513977, 'epoch': 8.93}

 89%|████████▉ | 652/730 [57:53<07:04,  5.44s/it]
                                                 
{'train/learning_rate_real': 7.412466666120846e-07, 'epoch': 8.93}

 89%|████████▉ | 652/730 [57:53<07:04,  5.44s/it]
 89%|████████▉ | 653/730 [57:53<06:55,  5.40s/it]
                                                 
{'debug/num_tok_total': 2624.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2624.0, 'debug/num_lat_loss': 1772.0, 'epoch': 8.95}

 89%|████████▉ | 653/730 [57:55<06:55,  5.40s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.3882725238800049, 'epoch': 8.95}

 89%|████████▉ | 653/730 [57:55<06:55,  5.40s/it]
                                                 
{'train/learning_rate_real': 7.225462763207e-07, 'epoch': 8.95}

 89%|████████▉ | 653/730 [57:55<06:55,  5.40s/it]
                                                 
{'debug/num_tok_total': 2888.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 2888.0, 'debug/num_lat_loss': 1801.0, 'epoch': 8.95}

 89%|████████▉ | 653/730 [57:56<06:55,  5.40s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.3477257192134857, 'epoch': 8.95}

 89%|████████▉ | 653/730 [57:56<06:55,  5.40s/it]
                                                 
{'train/learning_rate_real': 7.225462763207e-07, 'epoch': 8.95}

 89%|████████▉ | 653/730 [57:56<06:55,  5.40s/it]
                                                 
{'debug/num_tok_total': 2684.0, 'debug/num_tok_loss': 1810.0, 'debug/num_lat_total': 2684.0, 'debug/num_lat_loss': 1810.0, 'epoch': 8.95}

 89%|████████▉ | 653/730 [57:57<06:55,  5.40s/it]
                                                 
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.3921751379966736, 'epoch': 8.95}

 89%|████████▉ | 653/730 [57:57<06:55,  5.40s/it]
                                                 
{'train/learning_rate_real': 7.225462763207e-07, 'epoch': 8.95}

 89%|████████▉ | 653/730 [57:57<06:55,  5.40s/it]
                                                 
{'debug/num_tok_total': 2416.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2416.0, 'debug/num_lat_loss': 1781.0, 'epoch': 8.95}

 89%|████████▉ | 653/730 [57:58<06:55,  5.40s/it]
                                                 
{'train/ce_loss': 1.65625, 'train/diffusion_loss': 0.42893296480178833, 'epoch': 8.95}

 89%|████████▉ | 653/730 [57:58<06:55,  5.40s/it]
                                                 
{'train/learning_rate_real': 7.225462763207e-07, 'epoch': 8.95}

 89%|████████▉ | 653/730 [57:58<06:55,  5.40s/it]
 90%|████████▉ | 654/730 [57:59<06:52,  5.42s/it]
                                                 
{'debug/num_tok_total': 2876.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2876.0, 'debug/num_lat_loss': 1786.0, 'epoch': 8.96}

 90%|████████▉ | 654/730 [58:00<06:52,  5.42s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.3606095314025879, 'epoch': 8.96}

 90%|████████▉ | 654/730 [58:00<06:52,  5.42s/it]
                                                 
{'train/learning_rate_real': 7.04077777045592e-07, 'epoch': 8.96}

 90%|████████▉ | 654/730 [58:00<06:52,  5.42s/it]
                                                 
{'debug/num_tok_total': 2676.0, 'debug/num_tok_loss': 1809.0, 'debug/num_lat_total': 2676.0, 'debug/num_lat_loss': 1809.0, 'epoch': 8.96}

 90%|████████▉ | 654/730 [58:01<06:52,  5.42s/it]
                                                 
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.3943428695201874, 'epoch': 8.96}

 90%|████████▉ | 654/730 [58:01<06:52,  5.42s/it]
                                                 
{'train/learning_rate_real': 7.04077777045592e-07, 'epoch': 8.96}

 90%|████████▉ | 654/730 [58:01<06:52,  5.42s/it]
                                                 
{'debug/num_tok_total': 2007.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2007.0, 'debug/num_lat_loss': 1791.0, 'epoch': 8.96}

 90%|████████▉ | 654/730 [58:02<06:52,  5.42s/it]
                                                 
{'train/ce_loss': 1.625, 'train/diffusion_loss': 0.5325796604156494, 'epoch': 8.96}

 90%|████████▉ | 654/730 [58:02<06:52,  5.42s/it]
                                                 
{'train/learning_rate_real': 7.04077777045592e-07, 'epoch': 8.96}

 90%|████████▉ | 654/730 [58:02<06:52,  5.42s/it]
                                                 
{'debug/num_tok_total': 3052.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 3052.0, 'debug/num_lat_loss': 1772.0, 'epoch': 8.96}

 90%|████████▉ | 654/730 [58:04<06:52,  5.42s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.28197547793388367, 'epoch': 8.96}

 90%|████████▉ | 654/730 [58:04<06:52,  5.42s/it]
                                                 
{'train/learning_rate_real': 7.04077777045592e-07, 'epoch': 8.96}

 90%|████████▉ | 654/730 [58:04<06:52,  5.42s/it]
 90%|████████▉ | 655/730 [58:04<06:46,  5.42s/it]
                                                 
{'debug/num_tok_total': 3004.0, 'debug/num_tok_loss': 1754.0, 'debug/num_lat_total': 3004.0, 'debug/num_lat_loss': 1754.0, 'epoch': 8.97}

 90%|████████▉ | 655/730 [58:06<06:46,  5.42s/it]
                                                 
{'train/ce_loss': 1.9375, 'train/diffusion_loss': 0.30705371499061584, 'epoch': 8.97}

 90%|████████▉ | 655/730 [58:06<06:46,  5.42s/it]
                                                 
{'train/learning_rate_real': 6.858415324204559e-07, 'epoch': 8.97}

 90%|████████▉ | 655/730 [58:06<06:46,  5.42s/it]
                                                 
{'debug/num_tok_total': 2831.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 2831.0, 'debug/num_lat_loss': 1765.0, 'epoch': 8.97}

 90%|████████▉ | 655/730 [58:07<06:46,  5.42s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.3633372187614441, 'epoch': 8.97}

 90%|████████▉ | 655/730 [58:07<06:46,  5.42s/it]
                                                 
{'train/learning_rate_real': 6.858415324204559e-07, 'epoch': 8.97}

 90%|████████▉ | 655/730 [58:07<06:46,  5.42s/it]
                                                 
{'debug/num_tok_total': 2630.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2630.0, 'debug/num_lat_loss': 1778.0, 'epoch': 8.97}

 90%|████████▉ | 655/730 [58:08<06:46,  5.42s/it]
                                                 
{'train/ce_loss': 1.9765625, 'train/diffusion_loss': 0.43571558594703674, 'epoch': 8.97}

 90%|████████▉ | 655/730 [58:08<06:46,  5.42s/it]
                                                 
{'train/learning_rate_real': 6.858415324204559e-07, 'epoch': 8.97}

 90%|████████▉ | 655/730 [58:08<06:46,  5.42s/it]
                                                 
{'debug/num_tok_total': 2674.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2674.0, 'debug/num_lat_loss': 1802.0, 'epoch': 8.97}

 90%|████████▉ | 655/730 [58:09<06:46,  5.42s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.4368835687637329, 'epoch': 8.97}

 90%|████████▉ | 655/730 [58:09<06:46,  5.42s/it]
                                                 
{'train/learning_rate_real': 6.858415324204559e-07, 'epoch': 8.97}

 90%|████████▉ | 655/730 [58:09<06:46,  5.42s/it]
 90%|████████▉ | 656/730 [58:10<06:45,  5.48s/it]
                                                 
{'debug/num_tok_total': 2635.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2635.0, 'debug/num_lat_loss': 1783.0, 'epoch': 8.99}

 90%|████████▉ | 656/730 [58:11<06:45,  5.48s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.4057542681694031, 'epoch': 8.99}

 90%|████████▉ | 656/730 [58:11<06:45,  5.48s/it]
                                                 
{'train/learning_rate_real': 6.678379015060382e-07, 'epoch': 8.99}

 90%|████████▉ | 656/730 [58:11<06:45,  5.48s/it]
                                                 
{'debug/num_tok_total': 3092.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 3092.0, 'debug/num_lat_loss': 1792.0, 'epoch': 8.99}

 90%|████████▉ | 656/730 [58:12<06:45,  5.48s/it]
                                                 
{'train/ce_loss': 1.671875, 'train/diffusion_loss': 0.2983284890651703, 'epoch': 8.99}

 90%|████████▉ | 656/730 [58:12<06:45,  5.48s/it]
                                                 
{'train/learning_rate_real': 6.678379015060382e-07, 'epoch': 8.99}

 90%|████████▉ | 656/730 [58:12<06:45,  5.48s/it]
                                                 
{'debug/num_tok_total': 2650.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2650.0, 'debug/num_lat_loss': 1794.0, 'epoch': 8.99}

 90%|████████▉ | 656/730 [58:14<06:45,  5.48s/it]
                                                 
{'train/ce_loss': 1.5546875, 'train/diffusion_loss': 0.3854256570339203, 'epoch': 8.99}

 90%|████████▉ | 656/730 [58:14<06:45,  5.48s/it]
                                                 
{'train/learning_rate_real': 6.678379015060382e-07, 'epoch': 8.99}

 90%|████████▉ | 656/730 [58:14<06:45,  5.48s/it]
                                                 
{'debug/num_tok_total': 633.0, 'debug/num_tok_loss': 428.0, 'debug/num_lat_total': 633.0, 'debug/num_lat_loss': 428.0, 'epoch': 8.99}

 90%|████████▉ | 656/730 [58:14<06:45,  5.48s/it]
                                                 
{'train/ce_loss': 1.6640625, 'train/diffusion_loss': 0.3700653314590454, 'epoch': 8.99}

 90%|████████▉ | 656/730 [58:14<06:45,  5.48s/it]
                                                 
{'train/learning_rate_real': 6.678379015060382e-07, 'epoch': 8.99}

 90%|████████▉ | 656/730 [58:14<06:45,  5.48s/it]
 90%|█████████ | 657/730 [58:15<06:25,  5.28s/it]
                                                 
{'debug/num_tok_total': 2862.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2862.0, 'debug/num_lat_loss': 1783.0, 'epoch': 9.0}

 90%|█████████ | 657/730 [58:16<06:25,  5.28s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.3782511353492737, 'epoch': 9.0}

 90%|█████████ | 657/730 [58:16<06:25,  5.28s/it]
                                                 
{'train/learning_rate_real': 6.5006723878306e-07, 'epoch': 9.0}

 90%|█████████ | 657/730 [58:16<06:25,  5.28s/it]
                                                 
{'debug/num_tok_total': 2833.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 2833.0, 'debug/num_lat_loss': 1767.0, 'epoch': 9.0}

 90%|█████████ | 657/730 [58:17<06:25,  5.28s/it]
                                                 
{'train/ce_loss': 1.5390625, 'train/diffusion_loss': 0.34828177094459534, 'epoch': 9.0}

 90%|█████████ | 657/730 [58:17<06:25,  5.28s/it]
                                                 
{'train/learning_rate_real': 6.5006723878306e-07, 'epoch': 9.0}

 90%|█████████ | 657/730 [58:17<06:25,  5.28s/it]
                                                 
{'debug/num_tok_total': 2283.0, 'debug/num_tok_loss': 1638.0, 'debug/num_lat_total': 2283.0, 'debug/num_lat_loss': 1638.0, 'epoch': 9.0}

 90%|█████████ | 657/730 [58:19<06:25,  5.28s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.4649447202682495, 'epoch': 9.0}

 90%|█████████ | 657/730 [58:19<06:25,  5.28s/it]
                                                 
{'train/learning_rate_real': 6.5006723878306e-07, 'epoch': 9.0}

 90%|█████████ | 657/730 [58:19<06:25,  5.28s/it]
                                                 
{'debug/num_tok_total': 3012.0, 'debug/num_tok_loss': 1752.0, 'debug/num_lat_total': 3012.0, 'debug/num_lat_loss': 1752.0, 'epoch': 9.0}

 90%|█████████ | 657/730 [58:20<06:25,  5.28s/it]
                                                 
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.33454886078834534, 'epoch': 9.0}

 90%|█████████ | 657/730 [58:20<06:25,  5.28s/it]
                                                 
{'train/learning_rate_real': 6.5006723878306e-07, 'epoch': 9.0}

 90%|█████████ | 657/730 [58:20<06:25,  5.28s/it]
 90%|█████████ | 658/730 [58:21<06:33,  5.47s/it]
                                                 
{'debug/num_tok_total': 2802.0, 'debug/num_tok_loss': 1751.0, 'debug/num_lat_total': 2802.0, 'debug/num_lat_loss': 1751.0, 'epoch': 9.01}

 90%|█████████ | 658/730 [58:22<06:33,  5.47s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.36447247862815857, 'epoch': 9.01}

 90%|█████████ | 658/730 [58:22<06:33,  5.47s/it]
                                                 
{'train/learning_rate_real': 6.32529894145234e-07, 'epoch': 9.01}

 90%|█████████ | 658/730 [58:22<06:33,  5.47s/it]
                                                 
{'debug/num_tok_total': 2613.0, 'debug/num_tok_loss': 1759.0, 'debug/num_lat_total': 2613.0, 'debug/num_lat_loss': 1759.0, 'epoch': 9.01}

 90%|█████████ | 658/730 [58:23<06:33,  5.47s/it]
                                                 
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.3852800130844116, 'epoch': 9.01}

 90%|█████████ | 658/730 [58:23<06:33,  5.47s/it]
                                                 
{'train/learning_rate_real': 6.32529894145234e-07, 'epoch': 9.01}

 90%|█████████ | 658/730 [58:23<06:33,  5.47s/it]
                                                 
{'debug/num_tok_total': 3141.0, 'debug/num_tok_loss': 1817.0, 'debug/num_lat_total': 3141.0, 'debug/num_lat_loss': 1817.0, 'epoch': 9.01}

 90%|█████████ | 658/730 [58:24<06:33,  5.47s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.31997185945510864, 'epoch': 9.01}

 90%|█████████ | 658/730 [58:24<06:33,  5.47s/it]
                                                 
{'train/learning_rate_real': 6.32529894145234e-07, 'epoch': 9.01}

 90%|█████████ | 658/730 [58:24<06:33,  5.47s/it]
                                                 
{'debug/num_tok_total': 1998.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 1998.0, 'debug/num_lat_loss': 1785.0, 'epoch': 9.01}

 90%|█████████ | 658/730 [58:25<06:33,  5.47s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.5343276858329773, 'epoch': 9.01}

 90%|█████████ | 658/730 [58:25<06:33,  5.47s/it]
                                                 
{'train/learning_rate_real': 6.32529894145234e-07, 'epoch': 9.01}

 90%|█████████ | 658/730 [58:25<06:33,  5.47s/it]
 90%|█████████ | 659/730 [58:26<06:26,  5.45s/it]
                                                 
{'debug/num_tok_total': 2206.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2206.0, 'debug/num_lat_loss': 1791.0, 'epoch': 9.03}

 90%|█████████ | 659/730 [58:27<06:26,  5.45s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.49765250086784363, 'epoch': 9.03}

 90%|█████████ | 659/730 [58:27<06:26,  5.45s/it]
                                                 
{'train/learning_rate_real': 6.152262128923836e-07, 'epoch': 9.03}

 90%|█████████ | 659/730 [58:27<06:26,  5.45s/it]
                                                 
{'debug/num_tok_total': 2663.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2663.0, 'debug/num_lat_loss': 1795.0, 'epoch': 9.03}

 90%|█████████ | 659/730 [58:28<06:26,  5.45s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.39735791087150574, 'epoch': 9.03}

 90%|█████████ | 659/730 [58:28<06:26,  5.45s/it]
                                                 
{'train/learning_rate_real': 6.152262128923836e-07, 'epoch': 9.03}

 90%|█████████ | 659/730 [58:28<06:26,  5.45s/it]
                                                 
{'debug/num_tok_total': 2471.0, 'debug/num_tok_loss': 1813.0, 'debug/num_lat_total': 2471.0, 'debug/num_lat_loss': 1813.0, 'epoch': 9.03}

 90%|█████████ | 659/730 [58:29<06:26,  5.45s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.4199342727661133, 'epoch': 9.03}

 90%|█████████ | 659/730 [58:29<06:26,  5.45s/it]
                                                 
{'train/learning_rate_real': 6.152262128923836e-07, 'epoch': 9.03}

 90%|█████████ | 659/730 [58:29<06:26,  5.45s/it]
                                                 
{'debug/num_tok_total': 2193.0, 'debug/num_tok_loss': 1762.0, 'debug/num_lat_total': 2193.0, 'debug/num_lat_loss': 1762.0, 'epoch': 9.03}

 90%|█████████ | 659/730 [58:31<06:26,  5.45s/it]
                                                 
{'train/ce_loss': 1.625, 'train/diffusion_loss': 0.48111966252326965, 'epoch': 9.03}

 90%|█████████ | 659/730 [58:31<06:26,  5.45s/it]
                                                 
{'train/learning_rate_real': 6.152262128923836e-07, 'epoch': 9.03}

 90%|█████████ | 659/730 [58:31<06:26,  5.45s/it]03/16/2026 07:46:03 - INFO - __main__ - LoRA debug step 660: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 90%|█████████ | 660/730 [58:31<06:15,  5.36s/it]
                                                 
{'loss': 2.4972, 'grad_norm': 1.2349767684936523, 'learning_rate': 6.152262128923836e-07, 'epoch': 9.04}

 90%|█████████ | 660/730 [58:31<06:15,  5.36s/it]
                                                 
{'debug/num_tok_total': 1921.0, 'debug/num_tok_loss': 1699.0, 'debug/num_lat_total': 1921.0, 'debug/num_lat_loss': 1699.0, 'epoch': 9.04}

 90%|█████████ | 660/730 [58:32<06:15,  5.36s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.5214577317237854, 'epoch': 9.04}

 90%|█████████ | 660/730 [58:32<06:15,  5.36s/it]
                                                 
{'train/learning_rate_real': 5.981565357236346e-07, 'epoch': 9.04}

 90%|█████████ | 660/730 [58:32<06:15,  5.36s/it]
                                                 
{'debug/num_tok_total': 3004.0, 'debug/num_tok_loss': 1726.0, 'debug/num_lat_total': 3004.0, 'debug/num_lat_loss': 1726.0, 'epoch': 9.04}

 90%|█████████ | 660/730 [58:34<06:15,  5.36s/it]
                                                 
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.2969726026058197, 'epoch': 9.04}

 90%|█████████ | 660/730 [58:34<06:15,  5.36s/it]
                                                 
{'train/learning_rate_real': 5.981565357236346e-07, 'epoch': 9.04}

 90%|█████████ | 660/730 [58:34<06:15,  5.36s/it]
                                                 
{'debug/num_tok_total': 3324.0, 'debug/num_tok_loss': 1804.0, 'debug/num_lat_total': 3324.0, 'debug/num_lat_loss': 1804.0, 'epoch': 9.04}

 90%|█████████ | 660/730 [58:35<06:15,  5.36s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.2666879892349243, 'epoch': 9.04}

 90%|█████████ | 660/730 [58:35<06:15,  5.36s/it]
                                                 
{'train/learning_rate_real': 5.981565357236346e-07, 'epoch': 9.04}

 90%|█████████ | 660/730 [58:35<06:15,  5.36s/it]
                                                 
{'debug/num_tok_total': 2428.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2428.0, 'debug/num_lat_loss': 1772.0, 'epoch': 9.04}

 90%|█████████ | 660/730 [58:36<06:15,  5.36s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.4430752992630005, 'epoch': 9.04}

 90%|█████████ | 660/730 [58:36<06:15,  5.36s/it]
                                                 
{'train/learning_rate_real': 5.981565357236346e-07, 'epoch': 9.04}

 90%|█████████ | 660/730 [58:36<06:15,  5.36s/it]
 91%|█████████ | 661/730 [58:37<06:13,  5.41s/it]
                                                 
{'debug/num_tok_total': 2697.0, 'debug/num_tok_loss': 1609.0, 'debug/num_lat_total': 2697.0, 'debug/num_lat_loss': 1609.0, 'epoch': 9.05}

 91%|█████████ | 661/730 [58:38<06:13,  5.41s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.37265437841415405, 'epoch': 9.05}

 91%|█████████ | 661/730 [58:38<06:13,  5.41s/it]
                                                 
{'train/learning_rate_real': 5.813211987307207e-07, 'epoch': 9.05}

 91%|█████████ | 661/730 [58:38<06:13,  5.41s/it]
                                                 
{'debug/num_tok_total': 2801.0, 'debug/num_tok_loss': 1745.0, 'debug/num_lat_total': 2801.0, 'debug/num_lat_loss': 1745.0, 'epoch': 9.05}

 91%|█████████ | 661/730 [58:39<06:13,  5.41s/it]
                                                 
{'train/ce_loss': 1.8984375, 'train/diffusion_loss': 0.3374194800853729, 'epoch': 9.05}

 91%|█████████ | 661/730 [58:39<06:13,  5.41s/it]
                                                 
{'train/learning_rate_real': 5.813211987307207e-07, 'epoch': 9.05}

 91%|█████████ | 661/730 [58:39<06:13,  5.41s/it]
                                                 
{'debug/num_tok_total': 2222.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2222.0, 'debug/num_lat_loss': 1780.0, 'epoch': 9.05}

 91%|█████████ | 661/730 [58:40<06:13,  5.41s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.4742825925350189, 'epoch': 9.05}

 91%|█████████ | 661/730 [58:40<06:13,  5.41s/it]
                                                 
{'train/learning_rate_real': 5.813211987307207e-07, 'epoch': 9.05}

 91%|█████████ | 661/730 [58:40<06:13,  5.41s/it]
                                                 
{'debug/num_tok_total': 2478.0, 'debug/num_tok_loss': 1815.0, 'debug/num_lat_total': 2478.0, 'debug/num_lat_loss': 1815.0, 'epoch': 9.05}

 91%|█████████ | 661/730 [58:42<06:13,  5.41s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.4612460136413574, 'epoch': 9.05}

 91%|█████████ | 661/730 [58:42<06:13,  5.41s/it]
                                                 
{'train/learning_rate_real': 5.813211987307207e-07, 'epoch': 9.05}

 91%|█████████ | 661/730 [58:42<06:13,  5.41s/it]
 91%|█████████ | 662/730 [58:42<06:08,  5.41s/it]
                                                 
{'debug/num_tok_total': 2709.0, 'debug/num_tok_loss': 1823.0, 'debug/num_lat_total': 2709.0, 'debug/num_lat_loss': 1823.0, 'epoch': 9.07}

 91%|█████████ | 662/730 [58:43<06:08,  5.41s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.3856673538684845, 'epoch': 9.07}

 91%|█████████ | 662/730 [58:43<06:08,  5.41s/it]
                                                 
{'train/learning_rate_real': 5.64720533391351e-07, 'epoch': 9.07}

 91%|█████████ | 662/730 [58:43<06:08,  5.41s/it]
                                                 
{'debug/num_tok_total': 3059.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 3059.0, 'debug/num_lat_loss': 1773.0, 'epoch': 9.07}

 91%|█████████ | 662/730 [58:45<06:08,  5.41s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.3110087215900421, 'epoch': 9.07}

 91%|█████████ | 662/730 [58:45<06:08,  5.41s/it]
                                                 
{'train/learning_rate_real': 5.64720533391351e-07, 'epoch': 9.07}

 91%|█████████ | 662/730 [58:45<06:08,  5.41s/it]
                                                 
{'debug/num_tok_total': 2575.0, 'debug/num_tok_loss': 1722.0, 'debug/num_lat_total': 2575.0, 'debug/num_lat_loss': 1722.0, 'epoch': 9.07}

 91%|█████████ | 662/730 [58:46<06:08,  5.41s/it]
                                                 
{'train/ce_loss': 1.578125, 'train/diffusion_loss': 0.3810831606388092, 'epoch': 9.07}

 91%|█████████ | 662/730 [58:46<06:08,  5.41s/it]
                                                 
{'train/learning_rate_real': 5.64720533391351e-07, 'epoch': 9.07}

 91%|█████████ | 662/730 [58:46<06:08,  5.41s/it]
                                                 
{'debug/num_tok_total': 2262.0, 'debug/num_tok_loss': 1695.0, 'debug/num_lat_total': 2262.0, 'debug/num_lat_loss': 1695.0, 'epoch': 9.07}

 91%|█████████ | 662/730 [58:47<06:08,  5.41s/it]
                                                 
{'train/ce_loss': 2.125, 'train/diffusion_loss': 0.4662960469722748, 'epoch': 9.07}

 91%|█████████ | 662/730 [58:47<06:08,  5.41s/it]
                                                 
{'train/learning_rate_real': 5.64720533391351e-07, 'epoch': 9.07}

 91%|█████████ | 662/730 [58:47<06:08,  5.41s/it]
 91%|█████████ | 663/730 [58:48<06:05,  5.45s/it]
                                                 
{'debug/num_tok_total': 2634.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2634.0, 'debug/num_lat_loss': 1775.0, 'epoch': 9.08}

 91%|█████████ | 663/730 [58:49<06:05,  5.45s/it]
                                                 
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.38980910181999207, 'epoch': 9.08}

 91%|█████████ | 663/730 [58:49<06:05,  5.45s/it]
                                                 
{'train/learning_rate_real': 5.483548665626934e-07, 'epoch': 9.08}

 91%|█████████ | 663/730 [58:49<06:05,  5.45s/it]
                                                 
{'debug/num_tok_total': 2672.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2672.0, 'debug/num_lat_loss': 1796.0, 'epoch': 9.08}

 91%|█████████ | 663/730 [58:50<06:05,  5.45s/it]
                                                 
{'train/ce_loss': 1.8671875, 'train/diffusion_loss': 0.40199923515319824, 'epoch': 9.08}

 91%|█████████ | 663/730 [58:50<06:05,  5.45s/it]
                                                 
{'train/learning_rate_real': 5.483548665626934e-07, 'epoch': 9.08}

 91%|█████████ | 663/730 [58:50<06:05,  5.45s/it]
                                                 
{'debug/num_tok_total': 2893.0, 'debug/num_tok_loss': 1806.0, 'debug/num_lat_total': 2893.0, 'debug/num_lat_loss': 1806.0, 'epoch': 9.08}

 91%|█████████ | 663/730 [58:51<06:05,  5.45s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.348300963640213, 'epoch': 9.08}

 91%|█████████ | 663/730 [58:52<06:05,  5.45s/it]
                                                 
{'train/learning_rate_real': 5.483548665626934e-07, 'epoch': 9.08}

 91%|█████████ | 663/730 [58:52<06:05,  5.45s/it]
                                                 
{'debug/num_tok_total': 2876.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2876.0, 'debug/num_lat_loss': 1786.0, 'epoch': 9.08}

 91%|█████████ | 663/730 [58:53<06:05,  5.45s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.3463915288448334, 'epoch': 9.08}

 91%|█████████ | 663/730 [58:53<06:05,  5.45s/it]
                                                 
{'train/learning_rate_real': 5.483548665626934e-07, 'epoch': 9.08}

 91%|█████████ | 663/730 [58:53<06:05,  5.45s/it]
 91%|█████████ | 664/730 [58:53<06:05,  5.53s/it]
                                                 
{'debug/num_tok_total': 1665.0, 'debug/num_tok_loss': 1665.0, 'debug/num_lat_total': 1665.0, 'debug/num_lat_loss': 1665.0, 'epoch': 9.1}

 91%|█████████ | 664/730 [58:54<06:05,  5.53s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.5857201218605042, 'epoch': 9.1}

 91%|█████████ | 664/730 [58:54<06:05,  5.53s/it]
                                                 
{'train/learning_rate_real': 5.322245204749319e-07, 'epoch': 9.1}

 91%|█████████ | 664/730 [58:54<06:05,  5.53s/it]
                                                 
{'debug/num_tok_total': 2240.0, 'debug/num_tok_loss': 1600.0, 'debug/num_lat_total': 2240.0, 'debug/num_lat_loss': 1600.0, 'epoch': 9.1}

 91%|█████████ | 664/730 [58:55<06:05,  5.53s/it]
                                                 
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.4344039857387543, 'epoch': 9.1}

 91%|█████████ | 664/730 [58:55<06:05,  5.53s/it]
                                                 
{'train/learning_rate_real': 5.322245204749319e-07, 'epoch': 9.1}

 91%|█████████ | 664/730 [58:55<06:05,  5.53s/it]
                                                 
{'debug/num_tok_total': 2140.0, 'debug/num_tok_loss': 1758.0, 'debug/num_lat_total': 2140.0, 'debug/num_lat_loss': 1758.0, 'epoch': 9.1}

 91%|█████████ | 664/730 [58:57<06:05,  5.53s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.49359017610549927, 'epoch': 9.1}

 91%|█████████ | 664/730 [58:57<06:05,  5.53s/it]
                                                 
{'train/learning_rate_real': 5.322245204749319e-07, 'epoch': 9.1}

 91%|█████████ | 664/730 [58:57<06:05,  5.53s/it]
                                                 
{'debug/num_tok_total': 3076.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 3076.0, 'debug/num_lat_loss': 1781.0, 'epoch': 9.1}

 91%|█████████ | 664/730 [58:58<06:05,  5.53s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.31927233934402466, 'epoch': 9.1}

 91%|█████████ | 664/730 [58:58<06:05,  5.53s/it]
                                                 
{'train/learning_rate_real': 5.322245204749319e-07, 'epoch': 9.1}

 91%|█████████ | 664/730 [58:58<06:05,  5.53s/it]
 91%|█████████ | 665/730 [58:58<05:49,  5.38s/it]
                                                 
{'debug/num_tok_total': 3051.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 3051.0, 'debug/num_lat_loss': 1769.0, 'epoch': 9.11}

 91%|█████████ | 665/730 [59:00<05:49,  5.38s/it]
                                                 
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.3266885280609131, 'epoch': 9.11}

 91%|█████████ | 665/730 [59:00<05:49,  5.38s/it]
                                                 
{'train/learning_rate_real': 5.163298127249306e-07, 'epoch': 9.11}

 91%|█████████ | 665/730 [59:00<05:49,  5.38s/it]
                                                 
{'debug/num_tok_total': 2452.0, 'debug/num_tok_loss': 1682.0, 'debug/num_lat_total': 2452.0, 'debug/num_lat_loss': 1682.0, 'epoch': 9.11}

 91%|█████████ | 665/730 [59:01<05:49,  5.38s/it]
                                                 
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.4262148141860962, 'epoch': 9.11}

 91%|█████████ | 665/730 [59:01<05:49,  5.38s/it]
                                                 
{'train/learning_rate_real': 5.163298127249306e-07, 'epoch': 9.11}

 91%|█████████ | 665/730 [59:01<05:49,  5.38s/it]
                                                 
{'debug/num_tok_total': 2835.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2835.0, 'debug/num_lat_loss': 1766.0, 'epoch': 9.11}

 91%|█████████ | 665/730 [59:02<05:49,  5.38s/it]
                                                 
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.3629128932952881, 'epoch': 9.11}

 91%|█████████ | 665/730 [59:02<05:49,  5.38s/it]
                                                 
{'train/learning_rate_real': 5.163298127249306e-07, 'epoch': 9.11}

 91%|█████████ | 665/730 [59:02<05:49,  5.38s/it]
                                                 
{'debug/num_tok_total': 1972.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 1972.0, 'debug/num_lat_loss': 1765.0, 'epoch': 9.11}

 91%|█████████ | 665/730 [59:03<05:49,  5.38s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.532158374786377, 'epoch': 9.11}

 91%|█████████ | 665/730 [59:03<05:49,  5.38s/it]
                                                 
{'train/learning_rate_real': 5.163298127249306e-07, 'epoch': 9.11}

 91%|█████████ | 665/730 [59:03<05:49,  5.38s/it]
 91%|█████████ | 666/730 [59:04<05:44,  5.39s/it]
                                                 
{'debug/num_tok_total': 1986.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 1986.0, 'debug/num_lat_loss': 1772.0, 'epoch': 9.12}

 91%|█████████ | 666/730 [59:05<05:44,  5.39s/it]
                                                 
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.537484884262085, 'epoch': 9.12}

 91%|█████████ | 666/730 [59:05<05:44,  5.39s/it]
                                                 
{'train/learning_rate_real': 5.006710562699759e-07, 'epoch': 9.12}

 91%|█████████ | 666/730 [59:05<05:44,  5.39s/it]
                                                 
{'debug/num_tok_total': 2629.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 2629.0, 'debug/num_lat_loss': 1773.0, 'epoch': 9.12}

 91%|█████████ | 666/730 [59:06<05:44,  5.39s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.39163169264793396, 'epoch': 9.12}

 91%|█████████ | 666/730 [59:06<05:44,  5.39s/it]
                                                 
{'train/learning_rate_real': 5.006710562699759e-07, 'epoch': 9.12}

 91%|█████████ | 666/730 [59:06<05:44,  5.39s/it]
                                                 
{'debug/num_tok_total': 3113.0, 'debug/num_tok_loss': 1809.0, 'debug/num_lat_total': 3113.0, 'debug/num_lat_loss': 1809.0, 'epoch': 9.12}

 91%|█████████ | 666/730 [59:07<05:44,  5.39s/it]
                                                 
{'train/ce_loss': 1.6484375, 'train/diffusion_loss': 0.3134002387523651, 'epoch': 9.12}

 91%|█████████ | 666/730 [59:07<05:44,  5.39s/it]
                                                 
{'train/learning_rate_real': 5.006710562699759e-07, 'epoch': 9.12}

 91%|█████████ | 666/730 [59:07<05:44,  5.39s/it]
                                                 
{'debug/num_tok_total': 2413.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 2413.0, 'debug/num_lat_loss': 1769.0, 'epoch': 9.12}

 91%|█████████ | 666/730 [59:09<05:44,  5.39s/it]
                                                 
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.4475841820240021, 'epoch': 9.12}

 91%|█████████ | 666/730 [59:09<05:44,  5.39s/it]
                                                 
{'train/learning_rate_real': 5.006710562699759e-07, 'epoch': 9.12}

 91%|█████████ | 666/730 [59:09<05:44,  5.39s/it]
 91%|█████████▏| 667/730 [59:09<05:38,  5.37s/it]
                                                 
{'debug/num_tok_total': 2443.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2443.0, 'debug/num_lat_loss': 1793.0, 'epoch': 9.14}

 91%|█████████▏| 667/730 [59:10<05:38,  5.37s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.44055360555648804, 'epoch': 9.14}

 91%|█████████▏| 667/730 [59:10<05:38,  5.37s/it]
                                                 
{'train/learning_rate_real': 4.852485594216083e-07, 'epoch': 9.14}

 91%|█████████▏| 667/730 [59:10<05:38,  5.37s/it]
                                                 
{'debug/num_tok_total': 2094.0, 'debug/num_tok_loss': 1676.0, 'debug/num_lat_total': 2094.0, 'debug/num_lat_loss': 1676.0, 'epoch': 9.14}

 91%|█████████▏| 667/730 [59:11<05:38,  5.37s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.47489672899246216, 'epoch': 9.14}

 91%|█████████▏| 667/730 [59:11<05:38,  5.37s/it]
                                                 
{'train/learning_rate_real': 4.852485594216083e-07, 'epoch': 9.14}

 91%|█████████▏| 667/730 [59:11<05:38,  5.37s/it]
                                                 
{'debug/num_tok_total': 2835.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2835.0, 'debug/num_lat_loss': 1764.0, 'epoch': 9.14}

 91%|█████████▏| 667/730 [59:13<05:38,  5.37s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.35188934206962585, 'epoch': 9.14}

 91%|█████████▏| 667/730 [59:13<05:38,  5.37s/it]
                                                 
{'train/learning_rate_real': 4.852485594216083e-07, 'epoch': 9.14}

 91%|█████████▏| 667/730 [59:13<05:38,  5.37s/it]
                                                 
{'debug/num_tok_total': 2615.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 2615.0, 'debug/num_lat_loss': 1768.0, 'epoch': 9.14}

 91%|█████████▏| 667/730 [59:14<05:38,  5.37s/it]
                                                 
{'train/ce_loss': 1.6171875, 'train/diffusion_loss': 0.38012808561325073, 'epoch': 9.14}

 91%|█████████▏| 667/730 [59:14<05:38,  5.37s/it]
                                                 
{'train/learning_rate_real': 4.852485594216083e-07, 'epoch': 9.14}

 91%|█████████▏| 667/730 [59:14<05:38,  5.37s/it]
 92%|█████████▏| 668/730 [59:14<05:31,  5.34s/it]
                                                 
{'debug/num_tok_total': 2427.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2427.0, 'debug/num_lat_loss': 1778.0, 'epoch': 9.15}

 92%|█████████▏| 668/730 [59:15<05:31,  5.34s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.41185978055000305, 'epoch': 9.15}

 92%|█████████▏| 668/730 [59:15<05:31,  5.34s/it]
                                                 
{'train/learning_rate_real': 4.700626258395652e-07, 'epoch': 9.15}

 92%|█████████▏| 668/730 [59:15<05:31,  5.34s/it]
                                                 
{'debug/num_tok_total': 2451.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 2451.0, 'debug/num_lat_loss': 1797.0, 'epoch': 9.15}

 92%|█████████▏| 668/730 [59:17<05:31,  5.34s/it]
                                                 
{'train/ce_loss': 2.03125, 'train/diffusion_loss': 0.4560587704181671, 'epoch': 9.15}

 92%|█████████▏| 668/730 [59:17<05:31,  5.34s/it]
                                                 
{'train/learning_rate_real': 4.700626258395652e-07, 'epoch': 9.15}

 92%|█████████▏| 668/730 [59:17<05:31,  5.34s/it]
                                                 
{'debug/num_tok_total': 2449.0, 'debug/num_tok_loss': 1807.0, 'debug/num_lat_total': 2449.0, 'debug/num_lat_loss': 1807.0, 'epoch': 9.15}

 92%|█████████▏| 668/730 [59:18<05:31,  5.34s/it]
                                                 
{'train/ce_loss': 1.640625, 'train/diffusion_loss': 0.4532594680786133, 'epoch': 9.15}

 92%|█████████▏| 668/730 [59:18<05:31,  5.34s/it]
                                                 
{'train/learning_rate_real': 4.700626258395652e-07, 'epoch': 9.15}

 92%|█████████▏| 668/730 [59:18<05:31,  5.34s/it]
                                                 
{'debug/num_tok_total': 2429.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2429.0, 'debug/num_lat_loss': 1785.0, 'epoch': 9.15}

 92%|█████████▏| 668/730 [59:19<05:31,  5.34s/it]
                                                 
{'train/ce_loss': 1.96875, 'train/diffusion_loss': 0.4588242471218109, 'epoch': 9.15}

 92%|█████████▏| 668/730 [59:19<05:31,  5.34s/it]
                                                 
{'train/learning_rate_real': 4.700626258395652e-07, 'epoch': 9.15}

 92%|█████████▏| 668/730 [59:19<05:31,  5.34s/it]
 92%|█████████▏| 669/730 [59:20<05:22,  5.29s/it]
                                                 
{'debug/num_tok_total': 2182.0, 'debug/num_tok_loss': 1757.0, 'debug/num_lat_total': 2182.0, 'debug/num_lat_loss': 1757.0, 'epoch': 9.16}

 92%|█████████▏| 669/730 [59:21<05:22,  5.29s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.4664234220981598, 'epoch': 9.16}

 92%|█████████▏| 669/730 [59:21<05:22,  5.29s/it]
                                                 
{'train/learning_rate_real': 4.551135545257892e-07, 'epoch': 9.16}

 92%|█████████▏| 669/730 [59:21<05:22,  5.29s/it]
                                                 
{'debug/num_tok_total': 1989.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 1989.0, 'debug/num_lat_loss': 1770.0, 'epoch': 9.16}

 92%|█████████▏| 669/730 [59:22<05:22,  5.29s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.5550342798233032, 'epoch': 9.16}

 92%|█████████▏| 669/730 [59:22<05:22,  5.29s/it]
                                                 
{'train/learning_rate_real': 4.551135545257892e-07, 'epoch': 9.16}

 92%|█████████▏| 669/730 [59:22<05:22,  5.29s/it]
                                                 
{'debug/num_tok_total': 2007.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2007.0, 'debug/num_lat_loss': 1791.0, 'epoch': 9.16}

 92%|█████████▏| 669/730 [59:23<05:22,  5.29s/it]
                                                 
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.5325809717178345, 'epoch': 9.16}

 92%|█████████▏| 669/730 [59:23<05:22,  5.29s/it]
                                                 
{'train/learning_rate_real': 4.551135545257892e-07, 'epoch': 9.16}

 92%|█████████▏| 669/730 [59:23<05:22,  5.29s/it]
                                                 
{'debug/num_tok_total': 2901.0, 'debug/num_tok_loss': 1804.0, 'debug/num_lat_total': 2901.0, 'debug/num_lat_loss': 1804.0, 'epoch': 9.16}

 92%|█████████▏| 669/730 [59:24<05:22,  5.29s/it]
                                                 
{'train/ce_loss': 1.8828125, 'train/diffusion_loss': 0.3618415892124176, 'epoch': 9.16}

 92%|█████████▏| 669/730 [59:24<05:22,  5.29s/it]
                                                 
{'train/learning_rate_real': 4.551135545257892e-07, 'epoch': 9.16}

 92%|█████████▏| 669/730 [59:24<05:22,  5.29s/it]03/16/2026 07:46:56 - INFO - __main__ - LoRA debug step 670: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 92%|█████████▏| 670/730 [59:24<05:09,  5.15s/it]
                                                 
{'loss': 2.624, 'grad_norm': 1.269409418106079, 'learning_rate': 4.551135545257892e-07, 'epoch': 9.18}

 92%|█████████▏| 670/730 [59:24<05:09,  5.15s/it]
                                                 
{'debug/num_tok_total': 2771.0, 'debug/num_tok_loss': 1739.0, 'debug/num_lat_total': 2771.0, 'debug/num_lat_loss': 1739.0, 'epoch': 9.18}

 92%|█████████▏| 670/730 [59:26<05:09,  5.15s/it]
                                                 
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.3815280497074127, 'epoch': 9.18}

 92%|█████████▏| 670/730 [59:26<05:09,  5.15s/it]
                                                 
{'train/learning_rate_real': 4.4040163981855095e-07, 'epoch': 9.18}

 92%|█████████▏| 670/730 [59:26<05:09,  5.15s/it]
                                                 
{'debug/num_tok_total': 2637.0, 'debug/num_tok_loss': 1778.0, 'debug/num_lat_total': 2637.0, 'debug/num_lat_loss': 1778.0, 'epoch': 9.18}

 92%|█████████▏| 670/730 [59:27<05:09,  5.15s/it]
                                                 
{'train/ce_loss': 1.625, 'train/diffusion_loss': 0.4038456380367279, 'epoch': 9.18}

 92%|█████████▏| 670/730 [59:27<05:09,  5.15s/it]
                                                 
{'train/learning_rate_real': 4.4040163981855095e-07, 'epoch': 9.18}

 92%|█████████▏| 670/730 [59:27<05:09,  5.15s/it]
                                                 
{'debug/num_tok_total': 2627.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2627.0, 'debug/num_lat_loss': 1777.0, 'epoch': 9.18}

 92%|█████████▏| 670/730 [59:28<05:09,  5.15s/it]
                                                 
{'train/ce_loss': 1.6484375, 'train/diffusion_loss': 0.4037295877933502, 'epoch': 9.18}

 92%|█████████▏| 670/730 [59:28<05:09,  5.15s/it]
                                                 
{'train/learning_rate_real': 4.4040163981855095e-07, 'epoch': 9.18}

 92%|█████████▏| 670/730 [59:28<05:09,  5.15s/it]
                                                 
{'debug/num_tok_total': 2157.0, 'debug/num_tok_loss': 1761.0, 'debug/num_lat_total': 2157.0, 'debug/num_lat_loss': 1761.0, 'epoch': 9.18}

 92%|█████████▏| 670/730 [59:29<05:09,  5.15s/it]
                                                 
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.49073365330696106, 'epoch': 9.18}

 92%|█████████▏| 670/730 [59:29<05:09,  5.15s/it]
                                                 
{'train/learning_rate_real': 4.4040163981855095e-07, 'epoch': 9.18}

 92%|█████████▏| 670/730 [59:29<05:09,  5.15s/it]
 92%|█████████▏| 671/730 [59:29<05:02,  5.13s/it]
                                                 
{'debug/num_tok_total': 2695.0, 'debug/num_tok_loss': 1707.0, 'debug/num_lat_total': 2695.0, 'debug/num_lat_loss': 1707.0, 'epoch': 9.19}

 92%|█████████▏| 671/730 [59:31<05:02,  5.13s/it]
                                                 
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.3586810231208801, 'epoch': 9.19}

 92%|█████████▏| 671/730 [59:31<05:02,  5.13s/it]
                                                 
{'train/learning_rate_real': 4.259271713866475e-07, 'epoch': 9.19}

 92%|█████████▏| 671/730 [59:31<05:02,  5.13s/it]
                                                 
{'debug/num_tok_total': 2601.0, 'debug/num_tok_loss': 1756.0, 'debug/num_lat_total': 2601.0, 'debug/num_lat_loss': 1756.0, 'epoch': 9.19}

 92%|█████████▏| 671/730 [59:32<05:02,  5.13s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.39324381947517395, 'epoch': 9.19}

 92%|█████████▏| 671/730 [59:32<05:02,  5.13s/it]
                                                 
{'train/learning_rate_real': 4.259271713866475e-07, 'epoch': 9.19}

 92%|█████████▏| 671/730 [59:32<05:02,  5.13s/it]
                                                 
{'debug/num_tok_total': 2565.0, 'debug/num_tok_loss': 1711.0, 'debug/num_lat_total': 2565.0, 'debug/num_lat_loss': 1711.0, 'epoch': 9.19}

 92%|█████████▏| 671/730 [59:33<05:02,  5.13s/it]
                                                 
{'train/ce_loss': 1.65625, 'train/diffusion_loss': 0.40417250990867615, 'epoch': 9.19}

 92%|█████████▏| 671/730 [59:33<05:02,  5.13s/it]
                                                 
{'train/learning_rate_real': 4.259271713866475e-07, 'epoch': 9.19}

 92%|█████████▏| 671/730 [59:33<05:02,  5.13s/it]
                                                 
{'debug/num_tok_total': 3093.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 3093.0, 'debug/num_lat_loss': 1797.0, 'epoch': 9.19}

 92%|█████████▏| 671/730 [59:34<05:02,  5.13s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.3194330930709839, 'epoch': 9.19}

 92%|█████████▏| 671/730 [59:34<05:02,  5.13s/it]
                                                 
{'train/learning_rate_real': 4.259271713866475e-07, 'epoch': 9.19}

 92%|█████████▏| 671/730 [59:34<05:02,  5.13s/it]
 92%|█████████▏| 672/730 [59:35<05:04,  5.24s/it]
                                                 
{'debug/num_tok_total': 2609.0, 'debug/num_tok_loss': 1759.0, 'debug/num_lat_total': 2609.0, 'debug/num_lat_loss': 1759.0, 'epoch': 9.21}

 92%|█████████▏| 672/730 [59:36<05:04,  5.24s/it]
                                                 
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.38384974002838135, 'epoch': 9.21}

 92%|█████████▏| 672/730 [59:36<05:04,  5.24s/it]
                                                 
{'train/learning_rate_real': 4.1169043422369795e-07, 'epoch': 9.21}

 92%|█████████▏| 672/730 [59:36<05:04,  5.24s/it]
                                                 
{'debug/num_tok_total': 2301.0, 'debug/num_tok_loss': 1656.0, 'debug/num_lat_total': 2301.0, 'debug/num_lat_loss': 1656.0, 'epoch': 9.21}

 92%|█████████▏| 672/730 [59:37<05:04,  5.24s/it]
                                                 
{'train/ce_loss': 2.15625, 'train/diffusion_loss': 0.43983203172683716, 'epoch': 9.21}

 92%|█████████▏| 672/730 [59:37<05:04,  5.24s/it]
                                                 
{'train/learning_rate_real': 4.1169043422369795e-07, 'epoch': 9.21}

 92%|█████████▏| 672/730 [59:37<05:04,  5.24s/it]
                                                 
{'debug/num_tok_total': 2545.0, 'debug/num_tok_loss': 1690.0, 'debug/num_lat_total': 2545.0, 'debug/num_lat_loss': 1690.0, 'epoch': 9.21}

 92%|█████████▏| 672/730 [59:39<05:04,  5.24s/it]
                                                 
{'train/ce_loss': 1.6640625, 'train/diffusion_loss': 0.4200958013534546, 'epoch': 9.21}

 92%|█████████▏| 672/730 [59:39<05:04,  5.24s/it]
                                                 
{'train/learning_rate_real': 4.1169043422369795e-07, 'epoch': 9.21}

 92%|█████████▏| 672/730 [59:39<05:04,  5.24s/it]
                                                 
{'debug/num_tok_total': 2568.0, 'debug/num_tok_loss': 1750.0, 'debug/num_lat_total': 2568.0, 'debug/num_lat_loss': 1750.0, 'epoch': 9.21}

 92%|█████████▏| 672/730 [59:40<05:04,  5.24s/it]
                                                 
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.3874298930168152, 'epoch': 9.21}

 92%|█████████▏| 672/730 [59:40<05:04,  5.24s/it]
                                                 
{'train/learning_rate_real': 4.1169043422369795e-07, 'epoch': 9.21}

 92%|█████████▏| 672/730 [59:40<05:04,  5.24s/it]
 92%|█████████▏| 673/730 [59:40<05:00,  5.26s/it]
                                                 
{'debug/num_tok_total': 2610.0, 'debug/num_tok_loss': 1762.0, 'debug/num_lat_total': 2610.0, 'debug/num_lat_loss': 1762.0, 'epoch': 9.22}

 92%|█████████▏| 673/730 [59:41<05:00,  5.26s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.38593876361846924, 'epoch': 9.22}

 92%|█████████▏| 673/730 [59:41<05:00,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.9769170864254327e-07, 'epoch': 9.22}

 92%|█████████▏| 673/730 [59:41<05:00,  5.26s/it]
                                                 
{'debug/num_tok_total': 2605.0, 'debug/num_tok_loss': 1756.0, 'debug/num_lat_total': 2605.0, 'debug/num_lat_loss': 1756.0, 'epoch': 9.22}

 92%|█████████▏| 673/730 [59:43<05:00,  5.26s/it]
                                                 
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.3874664008617401, 'epoch': 9.22}

 92%|█████████▏| 673/730 [59:43<05:00,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.9769170864254327e-07, 'epoch': 9.22}

 92%|█████████▏| 673/730 [59:43<05:00,  5.26s/it]
                                                 
{'debug/num_tok_total': 2657.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2657.0, 'debug/num_lat_loss': 1786.0, 'epoch': 9.22}

 92%|█████████▏| 673/730 [59:44<05:00,  5.26s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.3910128176212311, 'epoch': 9.22}

 92%|█████████▏| 673/730 [59:44<05:00,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.9769170864254327e-07, 'epoch': 9.22}

 92%|█████████▏| 673/730 [59:44<05:00,  5.26s/it]
                                                 
{'debug/num_tok_total': 2448.0, 'debug/num_tok_loss': 1670.0, 'debug/num_lat_total': 2448.0, 'debug/num_lat_loss': 1670.0, 'epoch': 9.22}

 92%|█████████▏| 673/730 [59:45<05:00,  5.26s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.4148980975151062, 'epoch': 9.22}

 92%|█████████▏| 673/730 [59:45<05:00,  5.26s/it]
                                                 
{'train/learning_rate_real': 3.9769170864254327e-07, 'epoch': 9.22}

 92%|█████████▏| 673/730 [59:45<05:00,  5.26s/it]
 92%|█████████▏| 674/730 [59:46<04:55,  5.27s/it]
                                                 
{'debug/num_tok_total': 2255.0, 'debug/num_tok_loss': 1696.0, 'debug/num_lat_total': 2255.0, 'debug/num_lat_loss': 1696.0, 'epoch': 9.23}

 92%|█████████▏| 674/730 [59:47<04:55,  5.27s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.4666903614997864, 'epoch': 9.23}

 92%|█████████▏| 674/730 [59:47<04:55,  5.27s/it]
                                                 
{'train/learning_rate_real': 3.8393127026970786e-07, 'epoch': 9.23}

 92%|█████████▏| 674/730 [59:47<04:55,  5.27s/it]
                                                 
{'debug/num_tok_total': 2450.0, 'debug/num_tok_loss': 1535.0, 'debug/num_lat_total': 2450.0, 'debug/num_lat_loss': 1535.0, 'epoch': 9.23}

 92%|█████████▏| 674/730 [59:48<04:55,  5.27s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.3781052827835083, 'epoch': 9.23}

 92%|█████████▏| 674/730 [59:48<04:55,  5.27s/it]
                                                 
{'train/learning_rate_real': 3.8393127026970786e-07, 'epoch': 9.23}

 92%|█████████▏| 674/730 [59:48<04:55,  5.27s/it]
                                                 
{'debug/num_tok_total': 2545.0, 'debug/num_tok_loss': 1683.0, 'debug/num_lat_total': 2545.0, 'debug/num_lat_loss': 1683.0, 'epoch': 9.23}

 92%|█████████▏| 674/730 [59:49<04:55,  5.27s/it]
                                                 
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.38854873180389404, 'epoch': 9.23}

 92%|█████████▏| 674/730 [59:49<04:55,  5.27s/it]
                                                 
{'train/learning_rate_real': 3.8393127026970786e-07, 'epoch': 9.23}

 92%|█████████▏| 674/730 [59:49<04:55,  5.27s/it]
                                                 
{'debug/num_tok_total': 2339.0, 'debug/num_tok_loss': 1624.0, 'debug/num_lat_total': 2339.0, 'debug/num_lat_loss': 1624.0, 'epoch': 9.23}

 92%|█████████▏| 674/730 [59:50<04:55,  5.27s/it]
                                                 
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.4245136082172394, 'epoch': 9.23}

 92%|█████████▏| 674/730 [59:50<04:55,  5.27s/it]
                                                 
{'train/learning_rate_real': 3.8393127026970786e-07, 'epoch': 9.23}

 92%|█████████▏| 674/730 [59:50<04:55,  5.27s/it]
 92%|█████████▏| 675/730 [59:51<04:49,  5.27s/it]
                                                 
{'debug/num_tok_total': 2642.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2642.0, 'debug/num_lat_loss': 1785.0, 'epoch': 9.25}

 92%|█████████▏| 675/730 [59:52<04:49,  5.27s/it]
                                                 
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.38903501629829407, 'epoch': 9.25}

 92%|█████████▏| 675/730 [59:52<04:49,  5.27s/it]
                                                 
{'train/learning_rate_real': 3.704093900399913e-07, 'epoch': 9.25}

 92%|█████████▏| 675/730 [59:52<04:49,  5.27s/it]
                                                 
{'debug/num_tok_total': 2447.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2447.0, 'debug/num_lat_loss': 1791.0, 'epoch': 9.25}

 92%|█████████▏| 675/730 [59:53<04:49,  5.27s/it]
                                                 
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.4481946527957916, 'epoch': 9.25}

 92%|█████████▏| 675/730 [59:53<04:49,  5.27s/it]
                                                 
{'train/learning_rate_real': 3.704093900399913e-07, 'epoch': 9.25}

 92%|█████████▏| 675/730 [59:53<04:49,  5.27s/it]
                                                 
{'debug/num_tok_total': 2946.0, 'debug/num_tok_loss': 1666.0, 'debug/num_lat_total': 2946.0, 'debug/num_lat_loss': 1666.0, 'epoch': 9.25}

 92%|█████████▏| 675/730 [59:54<04:49,  5.27s/it]
                                                 
{'train/ce_loss': 1.6484375, 'train/diffusion_loss': 0.2881959080696106, 'epoch': 9.25}

 92%|█████████▏| 675/730 [59:54<04:49,  5.27s/it]
                                                 
{'train/learning_rate_real': 3.704093900399913e-07, 'epoch': 9.25}

 92%|█████████▏| 675/730 [59:54<04:49,  5.27s/it]
                                                 
{'debug/num_tok_total': 3286.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 3286.0, 'debug/num_lat_loss': 1776.0, 'epoch': 9.25}

 92%|█████████▏| 675/730 [59:56<04:49,  5.27s/it]
                                                 
{'train/ce_loss': 1.5234375, 'train/diffusion_loss': 0.267887145280838, 'epoch': 9.25}

 92%|█████████▏| 675/730 [59:56<04:49,  5.27s/it]
                                                 
{'train/learning_rate_real': 3.704093900399913e-07, 'epoch': 9.25}

 92%|█████████▏| 675/730 [59:56<04:49,  5.27s/it]
 93%|█████████▎| 676/730 [59:56<04:48,  5.33s/it]
                                                 
{'debug/num_tok_total': 2846.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2846.0, 'debug/num_lat_loss': 1786.0, 'epoch': 9.26}

 93%|█████████▎| 676/730 [59:57<04:48,  5.33s/it]
                                                 
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.3766798675060272, 'epoch': 9.26}

 93%|█████████▎| 676/730 [59:57<04:48,  5.33s/it]
                                                 
{'train/learning_rate_real': 3.5712633419112686e-07, 'epoch': 9.26}

 93%|█████████▎| 676/730 [59:57<04:48,  5.33s/it]
                                                 
{'debug/num_tok_total': 2385.0, 'debug/num_tok_loss': 1753.0, 'debug/num_lat_total': 2385.0, 'debug/num_lat_loss': 1753.0, 'epoch': 9.26}

 93%|█████████▎| 676/730 [59:59<04:48,  5.33s/it]
                                                 
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.43088197708129883, 'epoch': 9.26}

 93%|█████████▎| 676/730 [59:59<04:48,  5.33s/it]
                                                 
{'train/learning_rate_real': 3.5712633419112686e-07, 'epoch': 9.26}

 93%|█████████▎| 676/730 [59:59<04:48,  5.33s/it]
                                                 
{'debug/num_tok_total': 2955.0, 'debug/num_tok_loss': 1721.0, 'debug/num_lat_total': 2955.0, 'debug/num_lat_loss': 1721.0, 'epoch': 9.26}

 93%|█████████▎| 676/730 [1:00:00<04:48,  5.33s/it]
                                                   
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.32843339443206787, 'epoch': 9.26}

 93%|█████████▎| 676/730 [1:00:00<04:48,  5.33s/it]
                                                   
{'train/learning_rate_real': 3.5712633419112686e-07, 'epoch': 9.26}

 93%|█████████▎| 676/730 [1:00:00<04:48,  5.33s/it]
                                                   
{'debug/num_tok_total': 2651.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2651.0, 'debug/num_lat_loss': 1782.0, 'epoch': 9.26}

 93%|█████████▎| 676/730 [1:00:01<04:48,  5.33s/it]
                                                   
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.4005345106124878, 'epoch': 9.26}

 93%|█████████▎| 676/730 [1:00:01<04:48,  5.33s/it]
                                                   
{'train/learning_rate_real': 3.5712633419112686e-07, 'epoch': 9.26}

 93%|█████████▎| 676/730 [1:00:01<04:48,  5.33s/it]
 93%|█████████▎| 677/730 [1:00:02<04:42,  5.32s/it]
                                                   
{'debug/num_tok_total': 2624.0, 'debug/num_tok_loss': 1768.0, 'debug/num_lat_total': 2624.0, 'debug/num_lat_loss': 1768.0, 'epoch': 9.27}

 93%|█████████▎| 677/730 [1:00:03<04:42,  5.32s/it]
                                                   
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.4205111265182495, 'epoch': 9.27}

 93%|█████████▎| 677/730 [1:00:03<04:42,  5.32s/it]
                                                   
{'train/learning_rate_real': 3.4408236425853e-07, 'epoch': 9.27}

 93%|█████████▎| 677/730 [1:00:03<04:42,  5.32s/it]
                                                   
{'debug/num_tok_total': 2879.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2879.0, 'debug/num_lat_loss': 1793.0, 'epoch': 9.27}

 93%|█████████▎| 677/730 [1:00:04<04:42,  5.32s/it]
                                                   
{'train/ce_loss': 1.6875, 'train/diffusion_loss': 0.3574899733066559, 'epoch': 9.27}

 93%|█████████▎| 677/730 [1:00:04<04:42,  5.32s/it]
                                                   
{'train/learning_rate_real': 3.4408236425853e-07, 'epoch': 9.27}

 93%|█████████▎| 677/730 [1:00:04<04:42,  5.32s/it]
                                                   
{'debug/num_tok_total': 2854.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2854.0, 'debug/num_lat_loss': 1779.0, 'epoch': 9.27}

 93%|█████████▎| 677/730 [1:00:05<04:42,  5.32s/it]
                                                   
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.33877453207969666, 'epoch': 9.27}

 93%|█████████▎| 677/730 [1:00:05<04:42,  5.32s/it]
                                                   
{'train/learning_rate_real': 3.4408236425853e-07, 'epoch': 9.27}

 93%|█████████▎| 677/730 [1:00:05<04:42,  5.32s/it]
                                                   
{'debug/num_tok_total': 2692.0, 'debug/num_tok_loss': 1813.0, 'debug/num_lat_total': 2692.0, 'debug/num_lat_loss': 1813.0, 'epoch': 9.27}

 93%|█████████▎| 677/730 [1:00:07<04:42,  5.32s/it]
                                                   
{'train/ce_loss': 1.6328125, 'train/diffusion_loss': 0.4021083414554596, 'epoch': 9.27}

 93%|█████████▎| 677/730 [1:00:07<04:42,  5.32s/it]
                                                   
{'train/learning_rate_real': 3.4408236425853e-07, 'epoch': 9.27}

 93%|█████████▎| 677/730 [1:00:07<04:42,  5.32s/it]
 93%|█████████▎| 678/730 [1:00:07<04:37,  5.34s/it]
                                                   
{'debug/num_tok_total': 2363.0, 'debug/num_tok_loss': 1640.0, 'debug/num_lat_total': 2363.0, 'debug/num_lat_loss': 1640.0, 'epoch': 9.29}

 93%|█████████▎| 678/730 [1:00:08<04:37,  5.34s/it]
                                                   
{'train/ce_loss': 1.9140625, 'train/diffusion_loss': 0.43787723779678345, 'epoch': 9.29}

 93%|█████████▎| 678/730 [1:00:08<04:37,  5.34s/it]
                                                   
{'train/learning_rate_real': 3.3127773707016666e-07, 'epoch': 9.29}

 93%|█████████▎| 678/730 [1:00:08<04:37,  5.34s/it]
                                                   
{'debug/num_tok_total': 3082.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 3082.0, 'debug/num_lat_loss': 1790.0, 'epoch': 9.29}

 93%|█████████▎| 678/730 [1:00:09<04:37,  5.34s/it]
                                                   
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.2998664677143097, 'epoch': 9.29}

 93%|█████████▎| 678/730 [1:00:09<04:37,  5.34s/it]
                                                   
{'train/learning_rate_real': 3.3127773707016666e-07, 'epoch': 9.29}

 93%|█████████▎| 678/730 [1:00:09<04:37,  5.34s/it]
                                                   
{'debug/num_tok_total': 2847.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2847.0, 'debug/num_lat_loss': 1783.0, 'epoch': 9.29}

 93%|█████████▎| 678/730 [1:00:11<04:37,  5.34s/it]
                                                   
{'train/ce_loss': 2.046875, 'train/diffusion_loss': 0.3605981767177582, 'epoch': 9.29}

 93%|█████████▎| 678/730 [1:00:11<04:37,  5.34s/it]
                                                   
{'train/learning_rate_real': 3.3127773707016666e-07, 'epoch': 9.29}

 93%|█████████▎| 678/730 [1:00:11<04:37,  5.34s/it]
                                                   
{'debug/num_tok_total': 2627.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 2627.0, 'debug/num_lat_loss': 1765.0, 'epoch': 9.29}

 93%|█████████▎| 678/730 [1:00:12<04:37,  5.34s/it]
                                                   
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.3911869525909424, 'epoch': 9.29}

 93%|█████████▎| 678/730 [1:00:12<04:37,  5.34s/it]
                                                   
{'train/learning_rate_real': 3.3127773707016666e-07, 'epoch': 9.29}

 93%|█████████▎| 678/730 [1:00:12<04:37,  5.34s/it]
 93%|█████████▎| 679/730 [1:00:12<04:31,  5.32s/it]
                                                   
{'debug/num_tok_total': 2387.0, 'debug/num_tok_loss': 1756.0, 'debug/num_lat_total': 2387.0, 'debug/num_lat_loss': 1756.0, 'epoch': 9.3}

 93%|█████████▎| 679/730 [1:00:13<04:31,  5.32s/it]
                                                   
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.40464258193969727, 'epoch': 9.3}

 93%|█████████▎| 679/730 [1:00:13<04:31,  5.32s/it]
                                                   
{'train/learning_rate_real': 3.1871270474148617e-07, 'epoch': 9.3}

 93%|█████████▎| 679/730 [1:00:13<04:31,  5.32s/it]
                                                   
{'debug/num_tok_total': 2396.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2396.0, 'debug/num_lat_loss': 1763.0, 'epoch': 9.3}

 93%|█████████▎| 679/730 [1:00:14<04:31,  5.32s/it]
                                                   
{'train/ce_loss': 1.671875, 'train/diffusion_loss': 0.4320482313632965, 'epoch': 9.3}

 93%|█████████▎| 679/730 [1:00:14<04:31,  5.32s/it]
                                                   
{'train/learning_rate_real': 3.1871270474148617e-07, 'epoch': 9.3}

 93%|█████████▎| 679/730 [1:00:14<04:31,  5.32s/it]
                                                   
{'debug/num_tok_total': 2060.0, 'debug/num_tok_loss': 1638.0, 'debug/num_lat_total': 2060.0, 'debug/num_lat_loss': 1638.0, 'epoch': 9.3}

 93%|█████████▎| 679/730 [1:00:16<04:31,  5.32s/it]
                                                   
{'train/ce_loss': 1.5859375, 'train/diffusion_loss': 0.5135572552680969, 'epoch': 9.3}

 93%|█████████▎| 679/730 [1:00:16<04:31,  5.32s/it]
                                                   
{'train/learning_rate_real': 3.1871270474148617e-07, 'epoch': 9.3}

 93%|█████████▎| 679/730 [1:00:16<04:31,  5.32s/it]
                                                   
{'debug/num_tok_total': 2350.0, 'debug/num_tok_loss': 1705.0, 'debug/num_lat_total': 2350.0, 'debug/num_lat_loss': 1705.0, 'epoch': 9.3}

 93%|█████████▎| 679/730 [1:00:17<04:31,  5.32s/it]
                                                   
{'train/ce_loss': 1.6484375, 'train/diffusion_loss': 0.43515193462371826, 'epoch': 9.3}

 93%|█████████▎| 679/730 [1:00:17<04:31,  5.32s/it]
                                                   
{'train/learning_rate_real': 3.1871270474148617e-07, 'epoch': 9.3}

 93%|█████████▎| 679/730 [1:00:17<04:31,  5.32s/it]03/16/2026 07:47:49 - INFO - __main__ - LoRA debug step 680: changed A 195/196, changed B 196/196, lora_B_zero_now=0.

 93%|█████████▎| 680/730 [1:00:17<04:22,  5.25s/it]
                                                   
{'loss': 2.4832, 'grad_norm': 1.2420822381973267, 'learning_rate': 3.1871270474148617e-07, 'epoch': 9.32}

 93%|█████████▎| 680/730 [1:00:17<04:22,  5.25s/it]
                                                   
{'debug/num_tok_total': 2468.0, 'debug/num_tok_loss': 1667.0, 'debug/num_lat_total': 2468.0, 'debug/num_lat_loss': 1667.0, 'epoch': 9.32}

 93%|█████████▎| 680/730 [1:00:18<04:22,  5.25s/it]
                                                   
{'train/ce_loss': 1.953125, 'train/diffusion_loss': 0.4030461311340332, 'epoch': 9.32}

 93%|█████████▎| 680/730 [1:00:18<04:22,  5.25s/it]
                                                   
{'train/learning_rate_real': 3.0638751467045036e-07, 'epoch': 9.32}

 93%|█████████▎| 680/730 [1:00:18<04:22,  5.25s/it]
                                                   
{'debug/num_tok_total': 2520.0, 'debug/num_tok_loss': 1649.0, 'debug/num_lat_total': 2520.0, 'debug/num_lat_loss': 1649.0, 'epoch': 9.32}

 93%|█████████▎| 680/730 [1:00:20<04:22,  5.25s/it]
                                                   
{'train/ce_loss': 1.6484375, 'train/diffusion_loss': 0.4082297682762146, 'epoch': 9.32}

 93%|█████████▎| 680/730 [1:00:20<04:22,  5.25s/it]
                                                   
{'train/learning_rate_real': 3.0638751467045036e-07, 'epoch': 9.32}

 93%|█████████▎| 680/730 [1:00:20<04:22,  5.25s/it]
                                                   
{'debug/num_tok_total': 3067.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 3067.0, 'debug/num_lat_loss': 1779.0, 'epoch': 9.32}

 93%|█████████▎| 680/730 [1:00:21<04:22,  5.25s/it]
                                                   
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.29411864280700684, 'epoch': 9.32}

 93%|█████████▎| 680/730 [1:00:21<04:22,  5.25s/it]
                                                   
{'train/learning_rate_real': 3.0638751467045036e-07, 'epoch': 9.32}

 93%|█████████▎| 680/730 [1:00:21<04:22,  5.25s/it]
                                                   
{'debug/num_tok_total': 2688.0, 'debug/num_tok_loss': 1818.0, 'debug/num_lat_total': 2688.0, 'debug/num_lat_loss': 1818.0, 'epoch': 9.32}

 93%|█████████▎| 680/730 [1:00:22<04:22,  5.25s/it]
                                                   
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.4232625961303711, 'epoch': 9.32}

 93%|█████████▎| 680/730 [1:00:22<04:22,  5.25s/it]
                                                   
{'train/learning_rate_real': 3.0638751467045036e-07, 'epoch': 9.32}

 93%|█████████▎| 680/730 [1:00:22<04:22,  5.25s/it]
 93%|█████████▎| 681/730 [1:00:23<04:19,  5.29s/it]
                                                   
{'debug/num_tok_total': 2505.0, 'debug/num_tok_loss': 1632.0, 'debug/num_lat_total': 2505.0, 'debug/num_lat_loss': 1632.0, 'epoch': 9.33}

 93%|█████████▎| 681/730 [1:00:24<04:19,  5.29s/it]
                                                   
{'train/ce_loss': 1.6875, 'train/diffusion_loss': 0.4237677752971649, 'epoch': 9.33}

 93%|█████████▎| 681/730 [1:00:24<04:19,  5.29s/it]
                                                   
{'train/learning_rate_real': 2.9430240953268323e-07, 'epoch': 9.33}

 93%|█████████▎| 681/730 [1:00:24<04:19,  5.29s/it]
                                                   
{'debug/num_tok_total': 3095.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 3095.0, 'debug/num_lat_loss': 1799.0, 'epoch': 9.33}

 93%|█████████▎| 681/730 [1:00:25<04:19,  5.29s/it]
                                                   
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.3027908504009247, 'epoch': 9.33}

 93%|█████████▎| 681/730 [1:00:25<04:19,  5.29s/it]
                                                   
{'train/learning_rate_real': 2.9430240953268323e-07, 'epoch': 9.33}

 93%|█████████▎| 681/730 [1:00:25<04:19,  5.29s/it]
                                                   
{'debug/num_tok_total': 2673.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 2673.0, 'debug/num_lat_loss': 1793.0, 'epoch': 9.33}

 93%|█████████▎| 681/730 [1:00:27<04:19,  5.29s/it]
                                                   
{'train/ce_loss': 1.625, 'train/diffusion_loss': 0.3994653522968292, 'epoch': 9.33}

 93%|█████████▎| 681/730 [1:00:27<04:19,  5.29s/it]
                                                   
{'train/learning_rate_real': 2.9430240953268323e-07, 'epoch': 9.33}

 93%|█████████▎| 681/730 [1:00:27<04:19,  5.29s/it]
                                                   
{'debug/num_tok_total': 1792.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 1792.0, 'debug/num_lat_loss': 1792.0, 'epoch': 9.33}

 93%|█████████▎| 681/730 [1:00:28<04:19,  5.29s/it]
                                                   
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.5659502148628235, 'epoch': 9.33}

 93%|█████████▎| 681/730 [1:00:28<04:19,  5.29s/it]
                                                   
{'train/learning_rate_real': 2.9430240953268323e-07, 'epoch': 9.33}

 93%|█████████▎| 681/730 [1:00:28<04:19,  5.29s/it]
 93%|█████████▎| 682/730 [1:00:28<04:12,  5.26s/it]
                                                   
{'debug/num_tok_total': 2468.0, 'debug/num_tok_loss': 1585.0, 'debug/num_lat_total': 2468.0, 'debug/num_lat_loss': 1585.0, 'epoch': 9.34}

 93%|█████████▎| 682/730 [1:00:29<04:12,  5.26s/it]
                                                   
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.37399232387542725, 'epoch': 9.34}

 93%|█████████▎| 682/730 [1:00:29<04:12,  5.26s/it]
                                                   
{'train/learning_rate_real': 2.824576272766666e-07, 'epoch': 9.34}

 93%|█████████▎| 682/730 [1:00:29<04:12,  5.26s/it]
                                                   
{'debug/num_tok_total': 2363.0, 'debug/num_tok_loss': 1737.0, 'debug/num_lat_total': 2363.0, 'debug/num_lat_loss': 1737.0, 'epoch': 9.34}

 93%|█████████▎| 682/730 [1:00:30<04:12,  5.26s/it]
                                                   
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.46190786361694336, 'epoch': 9.34}

 93%|█████████▎| 682/730 [1:00:30<04:12,  5.26s/it]
                                                   
{'train/learning_rate_real': 2.824576272766666e-07, 'epoch': 9.34}

 93%|█████████▎| 682/730 [1:00:30<04:12,  5.26s/it]
                                                   
{'debug/num_tok_total': 2628.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2628.0, 'debug/num_lat_loss': 1766.0, 'epoch': 9.34}

 93%|█████████▎| 682/730 [1:00:31<04:12,  5.26s/it]
                                                   
{'train/ce_loss': 1.8515625, 'train/diffusion_loss': 0.3943422734737396, 'epoch': 9.34}

 93%|█████████▎| 682/730 [1:00:31<04:12,  5.26s/it]
                                                   
{'train/learning_rate_real': 2.824576272766666e-07, 'epoch': 9.34}

 93%|█████████▎| 682/730 [1:00:31<04:12,  5.26s/it]
                                                   
{'debug/num_tok_total': 2407.0, 'debug/num_tok_loss': 1759.0, 'debug/num_lat_total': 2407.0, 'debug/num_lat_loss': 1759.0, 'epoch': 9.34}

 93%|█████████▎| 682/730 [1:00:33<04:12,  5.26s/it]
                                                   
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.4653802812099457, 'epoch': 9.34}

 93%|█████████▎| 682/730 [1:00:33<04:12,  5.26s/it]
                                                   
{'train/learning_rate_real': 2.824576272766666e-07, 'epoch': 9.34}

 93%|█████████▎| 682/730 [1:00:33<04:12,  5.26s/it]
 94%|█████████▎| 683/730 [1:00:33<04:05,  5.22s/it]
                                                   
{'debug/num_tok_total': 2246.0, 'debug/num_tok_loss': 1599.0, 'debug/num_lat_total': 2246.0, 'debug/num_lat_loss': 1599.0, 'epoch': 9.36}

 94%|█████████▎| 683/730 [1:00:34<04:05,  5.22s/it]
                                                   
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.42918717861175537, 'epoch': 9.36}

 94%|█████████▎| 683/730 [1:00:34<04:05,  5.22s/it]
                                                   
{'train/learning_rate_real': 2.7085340111907967e-07, 'epoch': 9.36}

 94%|█████████▎| 683/730 [1:00:34<04:05,  5.22s/it]
                                                   
{'debug/num_tok_total': 2634.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2634.0, 'debug/num_lat_loss': 1775.0, 'epoch': 9.36}

 94%|█████████▎| 683/730 [1:00:35<04:05,  5.22s/it]
                                                   
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.3940320312976837, 'epoch': 9.36}

 94%|█████████▎| 683/730 [1:00:35<04:05,  5.22s/it]
                                                   
{'train/learning_rate_real': 2.7085340111907967e-07, 'epoch': 9.36}

 94%|█████████▎| 683/730 [1:00:35<04:05,  5.22s/it]
                                                   
{'debug/num_tok_total': 2651.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2651.0, 'debug/num_lat_loss': 1790.0, 'epoch': 9.36}

 94%|█████████▎| 683/730 [1:00:37<04:05,  5.22s/it]
                                                   
{'train/ce_loss': 1.5859375, 'train/diffusion_loss': 0.39075005054473877, 'epoch': 9.36}

 94%|█████████▎| 683/730 [1:00:37<04:05,  5.22s/it]
                                                   
{'train/learning_rate_real': 2.7085340111907967e-07, 'epoch': 9.36}

 94%|█████████▎| 683/730 [1:00:37<04:05,  5.22s/it]
                                                   
{'debug/num_tok_total': 3089.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 3089.0, 'debug/num_lat_loss': 1789.0, 'epoch': 9.36}

 94%|█████████▎| 683/730 [1:00:38<04:05,  5.22s/it]
                                                   
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.32452723383903503, 'epoch': 9.36}

 94%|█████████▎| 683/730 [1:00:38<04:05,  5.22s/it]
                                                   
{'train/learning_rate_real': 2.7085340111907967e-07, 'epoch': 9.36}

 94%|█████████▎| 683/730 [1:00:38<04:05,  5.22s/it]
 94%|█████████▎| 684/730 [1:00:38<04:02,  5.28s/it]
                                                   
{'debug/num_tok_total': 2626.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2626.0, 'debug/num_lat_loss': 1771.0, 'epoch': 9.37}

 94%|█████████▎| 684/730 [1:00:40<04:02,  5.28s/it]
                                                   
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.36911189556121826, 'epoch': 9.37}

 94%|█████████▎| 684/730 [1:00:40<04:02,  5.28s/it]
                                                   
{'train/learning_rate_real': 2.59489959540192e-07, 'epoch': 9.37}

 94%|█████████▎| 684/730 [1:00:40<04:02,  5.28s/it]
                                                   
{'debug/num_tok_total': 2692.0, 'debug/num_tok_loss': 1812.0, 'debug/num_lat_total': 2692.0, 'debug/num_lat_loss': 1812.0, 'epoch': 9.37}

 94%|█████████▎| 684/730 [1:00:41<04:02,  5.28s/it]
                                                   
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.4018833637237549, 'epoch': 9.37}

 94%|█████████▎| 684/730 [1:00:41<04:02,  5.28s/it]
                                                   
{'train/learning_rate_real': 2.59489959540192e-07, 'epoch': 9.37}

 94%|█████████▎| 684/730 [1:00:41<04:02,  5.28s/it]
                                                   
{'debug/num_tok_total': 2850.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2850.0, 'debug/num_lat_loss': 1775.0, 'epoch': 9.37}

 94%|█████████▎| 684/730 [1:00:42<04:02,  5.28s/it]
                                                   
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.37979912757873535, 'epoch': 9.37}

 94%|█████████▎| 684/730 [1:00:42<04:02,  5.28s/it]
                                                   
{'train/learning_rate_real': 2.59489959540192e-07, 'epoch': 9.37}

 94%|█████████▎| 684/730 [1:00:42<04:02,  5.28s/it]
                                                   
{'debug/num_tok_total': 2917.0, 'debug/num_tok_loss': 1816.0, 'debug/num_lat_total': 2917.0, 'debug/num_lat_loss': 1816.0, 'epoch': 9.37}

 94%|█████████▎| 684/730 [1:00:43<04:02,  5.28s/it]
                                                   
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.3567681610584259, 'epoch': 9.37}

 94%|█████████▎| 684/730 [1:00:43<04:02,  5.28s/it]
                                                   
{'train/learning_rate_real': 2.59489959540192e-07, 'epoch': 9.37}

 94%|█████████▎| 684/730 [1:00:43<04:02,  5.28s/it]
 94%|█████████▍| 685/730 [1:00:44<04:01,  5.37s/it]
                                                   
{'debug/num_tok_total': 2208.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2208.0, 'debug/num_lat_loss': 1781.0, 'epoch': 9.38}

 94%|█████████▍| 685/730 [1:00:45<04:01,  5.37s/it]
                                                   
{'train/ce_loss': 1.6875, 'train/diffusion_loss': 0.4729526937007904, 'epoch': 9.38}

 94%|█████████▍| 685/730 [1:00:45<04:01,  5.37s/it]
                                                   
{'train/learning_rate_real': 2.4836752627936525e-07, 'epoch': 9.38}

 94%|█████████▍| 685/730 [1:00:45<04:01,  5.37s/it]
                                                   
{'debug/num_tok_total': 2091.0, 'debug/num_tok_loss': 1650.0, 'debug/num_lat_total': 2091.0, 'debug/num_lat_loss': 1650.0, 'epoch': 9.38}

 94%|█████████▍| 685/730 [1:00:46<04:01,  5.37s/it]
                                                   
{'train/ce_loss': 1.671875, 'train/diffusion_loss': 0.4808712601661682, 'epoch': 9.38}

 94%|█████████▍| 685/730 [1:00:46<04:01,  5.37s/it]
                                                   
{'train/learning_rate_real': 2.4836752627936525e-07, 'epoch': 9.38}

 94%|█████████▍| 685/730 [1:00:46<04:01,  5.37s/it]
                                                   
{'debug/num_tok_total': 2915.0, 'debug/num_tok_loss': 1701.0, 'debug/num_lat_total': 2915.0, 'debug/num_lat_loss': 1701.0, 'epoch': 9.38}

 94%|█████████▍| 685/730 [1:00:48<04:01,  5.37s/it]
                                                   
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.3057163953781128, 'epoch': 9.38}

 94%|█████████▍| 685/730 [1:00:48<04:01,  5.37s/it]
                                                   
{'train/learning_rate_real': 2.4836752627936525e-07, 'epoch': 9.38}

 94%|█████████▍| 685/730 [1:00:48<04:01,  5.37s/it]
                                                   
{'debug/num_tok_total': 3025.0, 'debug/num_tok_loss': 1755.0, 'debug/num_lat_total': 3025.0, 'debug/num_lat_loss': 1755.0, 'epoch': 9.38}

 94%|█████████▍| 685/730 [1:00:49<04:01,  5.37s/it]
                                                   
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.34745216369628906, 'epoch': 9.38}

 94%|█████████▍| 685/730 [1:00:49<04:01,  5.37s/it]
                                                   
{'train/learning_rate_real': 2.4836752627936525e-07, 'epoch': 9.38}

 94%|█████████▍| 685/730 [1:00:49<04:01,  5.37s/it]
 94%|█████████▍| 686/730 [1:00:49<03:57,  5.39s/it]
                                                   
{'debug/num_tok_total': 3069.0, 'debug/num_tok_loss': 1773.0, 'debug/num_lat_total': 3069.0, 'debug/num_lat_loss': 1773.0, 'epoch': 9.4}

 94%|█████████▍| 686/730 [1:00:51<03:57,  5.39s/it]
                                                   
{'train/ce_loss': 1.640625, 'train/diffusion_loss': 0.2984721064567566, 'epoch': 9.4}

 94%|█████████▍| 686/730 [1:00:51<03:57,  5.39s/it]
                                                   
{'train/learning_rate_real': 2.374863203306571e-07, 'epoch': 9.4}

 94%|█████████▍| 686/730 [1:00:51<03:57,  5.39s/it]
                                                   
{'debug/num_tok_total': 2896.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 2896.0, 'debug/num_lat_loss': 1801.0, 'epoch': 9.4}

 94%|█████████▍| 686/730 [1:00:52<03:57,  5.39s/it]
                                                   
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.3753450810909271, 'epoch': 9.4}

 94%|█████████▍| 686/730 [1:00:52<03:57,  5.39s/it]
                                                   
{'train/learning_rate_real': 2.374863203306571e-07, 'epoch': 9.4}

 94%|█████████▍| 686/730 [1:00:52<03:57,  5.39s/it]
                                                   
{'debug/num_tok_total': 2237.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2237.0, 'debug/num_lat_loss': 1795.0, 'epoch': 9.4}

 94%|█████████▍| 686/730 [1:00:53<03:57,  5.39s/it]
                                                   
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.4801611006259918, 'epoch': 9.4}

 94%|█████████▍| 686/730 [1:00:53<03:57,  5.39s/it]
                                                   
{'train/learning_rate_real': 2.374863203306571e-07, 'epoch': 9.4}

 94%|█████████▍| 686/730 [1:00:53<03:57,  5.39s/it]
                                                   
{'debug/num_tok_total': 2602.0, 'debug/num_tok_loss': 1753.0, 'debug/num_lat_total': 2602.0, 'debug/num_lat_loss': 1753.0, 'epoch': 9.4}

 94%|█████████▍| 686/730 [1:00:54<03:57,  5.39s/it]
                                                   
{'train/ce_loss': 1.8125, 'train/diffusion_loss': 0.3727896809577942, 'epoch': 9.4}

 94%|█████████▍| 686/730 [1:00:54<03:57,  5.39s/it]
                                                   
{'train/learning_rate_real': 2.374863203306571e-07, 'epoch': 9.4}

 94%|█████████▍| 686/730 [1:00:54<03:57,  5.39s/it]
 94%|█████████▍| 687/730 [1:00:55<03:52,  5.40s/it]
                                                   
{'debug/num_tok_total': 2854.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2854.0, 'debug/num_lat_loss': 1786.0, 'epoch': 9.41}

 94%|█████████▍| 687/730 [1:00:56<03:52,  5.40s/it]
                                                   
{'train/ce_loss': 1.5546875, 'train/diffusion_loss': 0.35017839074134827, 'epoch': 9.41}

 94%|█████████▍| 687/730 [1:00:56<03:52,  5.40s/it]
                                                   
{'train/learning_rate_real': 2.268465559384994e-07, 'epoch': 9.41}

 94%|█████████▍| 687/730 [1:00:56<03:52,  5.40s/it]
                                                   
{'debug/num_tok_total': 3065.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 3065.0, 'debug/num_lat_loss': 1783.0, 'epoch': 9.41}

 94%|█████████▍| 687/730 [1:00:57<03:52,  5.40s/it]
                                                   
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.31907767057418823, 'epoch': 9.41}

 94%|█████████▍| 687/730 [1:00:57<03:52,  5.40s/it]
                                                   
{'train/learning_rate_real': 2.268465559384994e-07, 'epoch': 9.41}

 94%|█████████▍| 687/730 [1:00:57<03:52,  5.40s/it]
                                                   
{'debug/num_tok_total': 2830.0, 'debug/num_tok_loss': 1760.0, 'debug/num_lat_total': 2830.0, 'debug/num_lat_loss': 1760.0, 'epoch': 9.41}

 94%|█████████▍| 687/730 [1:00:59<03:52,  5.40s/it]
                                                   
{'train/ce_loss': 1.625, 'train/diffusion_loss': 0.33773767948150635, 'epoch': 9.41}

 94%|█████████▍| 687/730 [1:00:59<03:52,  5.40s/it]
                                                   
{'train/learning_rate_real': 2.268465559384994e-07, 'epoch': 9.41}

 94%|█████████▍| 687/730 [1:00:59<03:52,  5.40s/it]
                                                   
{'debug/num_tok_total': 3085.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 3085.0, 'debug/num_lat_loss': 1791.0, 'epoch': 9.41}

 94%|█████████▍| 687/730 [1:01:00<03:52,  5.40s/it]
                                                   
{'train/ce_loss': 1.65625, 'train/diffusion_loss': 0.33730846643447876, 'epoch': 9.41}

 94%|█████████▍| 687/730 [1:01:00<03:52,  5.40s/it]
                                                   
{'train/learning_rate_real': 2.268465559384994e-07, 'epoch': 9.41}

 94%|█████████▍| 687/730 [1:01:00<03:52,  5.40s/it]
 94%|█████████▍| 688/730 [1:01:01<03:50,  5.49s/it]
                                                   
{'debug/num_tok_total': 2226.0, 'debug/num_tok_loss': 1791.0, 'debug/num_lat_total': 2226.0, 'debug/num_lat_loss': 1791.0, 'epoch': 9.42}

 94%|█████████▍| 688/730 [1:01:02<03:50,  5.49s/it]
                                                   
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.4850248396396637, 'epoch': 9.42}

 94%|█████████▍| 688/730 [1:01:02<03:50,  5.49s/it]
                                                   
{'train/learning_rate_real': 2.164484425934879e-07, 'epoch': 9.42}

 94%|█████████▍| 688/730 [1:01:02<03:50,  5.49s/it]
                                                   
{'debug/num_tok_total': 2717.0, 'debug/num_tok_loss': 1825.0, 'debug/num_lat_total': 2717.0, 'debug/num_lat_loss': 1825.0, 'epoch': 9.42}

 94%|█████████▍| 688/730 [1:01:03<03:50,  5.49s/it]
                                                   
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.393517404794693, 'epoch': 9.42}

 94%|█████████▍| 688/730 [1:01:03<03:50,  5.49s/it]
                                                   
{'train/learning_rate_real': 2.164484425934879e-07, 'epoch': 9.42}

 94%|█████████▍| 688/730 [1:01:03<03:50,  5.49s/it]
                                                   
{'debug/num_tok_total': 2642.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2642.0, 'debug/num_lat_loss': 1779.0, 'epoch': 9.42}

 94%|█████████▍| 688/730 [1:01:04<03:50,  5.49s/it]
                                                   
{'train/ce_loss': 1.8984375, 'train/diffusion_loss': 0.37709566950798035, 'epoch': 9.42}

 94%|█████████▍| 688/730 [1:01:04<03:50,  5.49s/it]
                                                   
{'train/learning_rate_real': 2.164484425934879e-07, 'epoch': 9.42}

 94%|█████████▍| 688/730 [1:01:04<03:50,  5.49s/it]
                                                   
{'debug/num_tok_total': 2785.0, 'debug/num_tok_loss': 1736.0, 'debug/num_lat_total': 2785.0, 'debug/num_lat_loss': 1736.0, 'epoch': 9.42}

 94%|█████████▍| 688/730 [1:01:05<03:50,  5.49s/it]
                                                   
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.3425300717353821, 'epoch': 9.42}

 94%|█████████▍| 688/730 [1:01:05<03:50,  5.49s/it]
                                                   
{'train/learning_rate_real': 2.164484425934879e-07, 'epoch': 9.42}

 94%|█████████▍| 688/730 [1:01:05<03:50,  5.49s/it]
 94%|█████████▍| 689/730 [1:01:06<03:43,  5.45s/it]
                                                   
{'debug/num_tok_total': 3048.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 3048.0, 'debug/num_lat_loss': 1771.0, 'epoch': 9.44}

 94%|█████████▍| 689/730 [1:01:07<03:43,  5.45s/it]
                                                   
{'train/ce_loss': 1.6484375, 'train/diffusion_loss': 0.3530248701572418, 'epoch': 9.44}

 94%|█████████▍| 689/730 [1:01:07<03:43,  5.45s/it]
                                                   
{'train/learning_rate_real': 2.0629218502825343e-07, 'epoch': 9.44}

 94%|█████████▍| 689/730 [1:01:07<03:43,  5.45s/it]
                                                   
{'debug/num_tok_total': 2678.0, 'debug/num_tok_loss': 1811.0, 'debug/num_lat_total': 2678.0, 'debug/num_lat_loss': 1811.0, 'epoch': 9.44}

 94%|█████████▍| 689/730 [1:01:08<03:43,  5.45s/it]
                                                   
{'train/ce_loss': 1.8984375, 'train/diffusion_loss': 0.41056036949157715, 'epoch': 9.44}

 94%|█████████▍| 689/730 [1:01:08<03:43,  5.45s/it]
                                                   
{'train/learning_rate_real': 2.0629218502825343e-07, 'epoch': 9.44}

 94%|█████████▍| 689/730 [1:01:08<03:43,  5.45s/it]
                                                   
{'debug/num_tok_total': 3230.0, 'debug/num_tok_loss': 1754.0, 'debug/num_lat_total': 3230.0, 'debug/num_lat_loss': 1754.0, 'epoch': 9.44}

 94%|█████████▍| 689/730 [1:01:10<03:43,  5.45s/it]
                                                   
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.2824682891368866, 'epoch': 9.44}

 94%|█████████▍| 689/730 [1:01:10<03:43,  5.45s/it]
                                                   
{'train/learning_rate_real': 2.0629218502825343e-07, 'epoch': 9.44}

 94%|█████████▍| 689/730 [1:01:10<03:43,  5.45s/it]
                                                   
{'debug/num_tok_total': 2084.0, 'debug/num_tok_loss': 1641.0, 'debug/num_lat_total': 2084.0, 'debug/num_lat_loss': 1641.0, 'epoch': 9.44}

 94%|█████████▍| 689/730 [1:01:11<03:43,  5.45s/it]
                                                   
{'train/ce_loss': 2.015625, 'train/diffusion_loss': 0.5047212839126587, 'epoch': 9.44}

 94%|█████████▍| 689/730 [1:01:11<03:43,  5.45s/it]
                                                   
{'train/learning_rate_real': 2.0629218502825343e-07, 'epoch': 9.44}

 94%|█████████▍| 689/730 [1:01:11<03:43,  5.45s/it]03/16/2026 07:48:43 - INFO - __main__ - LoRA debug step 690: changed A 196/196, changed B 196/196, lora_B_zero_now=0.

 95%|█████████▍| 690/730 [1:01:11<03:37,  5.45s/it]
                                                   
{'loss': 2.4613, 'grad_norm': 1.2346093654632568, 'learning_rate': 2.0629218502825343e-07, 'epoch': 9.45}

 95%|█████████▍| 690/730 [1:01:11<03:37,  5.45s/it]
                                                   
{'debug/num_tok_total': 2219.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2219.0, 'debug/num_lat_loss': 1780.0, 'epoch': 9.45}

 95%|█████████▍| 690/730 [1:01:13<03:37,  5.45s/it]
                                                   
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.4930492639541626, 'epoch': 9.45}

 95%|█████████▍| 690/730 [1:01:13<03:37,  5.45s/it]
                                                   
{'train/learning_rate_real': 1.9637798321342772e-07, 'epoch': 9.45}

 95%|█████████▍| 690/730 [1:01:13<03:37,  5.45s/it]
                                                   
{'debug/num_tok_total': 2392.0, 'debug/num_tok_loss': 1664.0, 'debug/num_lat_total': 2392.0, 'debug/num_lat_loss': 1664.0, 'epoch': 9.45}

 95%|█████████▍| 690/730 [1:01:14<03:37,  5.45s/it]
                                                   
{'train/ce_loss': 1.6484375, 'train/diffusion_loss': 0.47528937458992004, 'epoch': 9.45}

 95%|█████████▍| 690/730 [1:01:14<03:37,  5.45s/it]
                                                   
{'train/learning_rate_real': 1.9637798321342772e-07, 'epoch': 9.45}

 95%|█████████▍| 690/730 [1:01:14<03:37,  5.45s/it]
                                                   
{'debug/num_tok_total': 2469.0, 'debug/num_tok_loss': 1804.0, 'debug/num_lat_total': 2469.0, 'debug/num_lat_loss': 1804.0, 'epoch': 9.45}

 95%|█████████▍| 690/730 [1:01:15<03:37,  5.45s/it]
                                                   
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.44549497961997986, 'epoch': 9.45}

 95%|█████████▍| 690/730 [1:01:15<03:37,  5.45s/it]
                                                   
{'train/learning_rate_real': 1.9637798321342772e-07, 'epoch': 9.45}

 95%|█████████▍| 690/730 [1:01:15<03:37,  5.45s/it]
                                                   
{'debug/num_tok_total': 2616.0, 'debug/num_tok_loss': 1552.0, 'debug/num_lat_total': 2616.0, 'debug/num_lat_loss': 1552.0, 'epoch': 9.45}

 95%|█████████▍| 690/730 [1:01:17<03:37,  5.45s/it]
                                                   
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.3060680031776428, 'epoch': 9.45}

 95%|█████████▍| 690/730 [1:01:17<03:37,  5.45s/it]
                                                   
{'train/learning_rate_real': 1.9637798321342772e-07, 'epoch': 9.45}

 95%|█████████▍| 690/730 [1:01:17<03:37,  5.45s/it]
 95%|█████████▍| 691/730 [1:01:17<03:34,  5.50s/it]
                                                   
{'debug/num_tok_total': 3099.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 3099.0, 'debug/num_lat_loss': 1797.0, 'epoch': 9.47}

 95%|█████████▍| 691/730 [1:01:18<03:34,  5.50s/it]
                                                   
{'train/ce_loss': 1.9140625, 'train/diffusion_loss': 0.2966419458389282, 'epoch': 9.47}

 95%|█████████▍| 691/730 [1:01:18<03:34,  5.50s/it]
                                                   
{'train/learning_rate_real': 1.8670603235371176e-07, 'epoch': 9.47}

 95%|█████████▍| 691/730 [1:01:18<03:34,  5.50s/it]
                                                   
{'debug/num_tok_total': 2187.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 2187.0, 'debug/num_lat_loss': 1765.0, 'epoch': 9.47}

 95%|█████████▍| 691/730 [1:01:19<03:34,  5.50s/it]
                                                   
{'train/ce_loss': 1.8515625, 'train/diffusion_loss': 0.4698784351348877, 'epoch': 9.47}

 95%|█████████▍| 691/730 [1:01:19<03:34,  5.50s/it]
                                                   
{'train/learning_rate_real': 1.8670603235371176e-07, 'epoch': 9.47}

 95%|█████████▍| 691/730 [1:01:19<03:34,  5.50s/it]
                                                   
{'debug/num_tok_total': 2856.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2856.0, 'debug/num_lat_loss': 1779.0, 'epoch': 9.47}

 95%|█████████▍| 691/730 [1:01:21<03:34,  5.50s/it]
                                                   
{'train/ce_loss': 1.875, 'train/diffusion_loss': 0.3419833779335022, 'epoch': 9.47}

 95%|█████████▍| 691/730 [1:01:21<03:34,  5.50s/it]
                                                   
{'train/learning_rate_real': 1.8670603235371176e-07, 'epoch': 9.47}

 95%|█████████▍| 691/730 [1:01:21<03:34,  5.50s/it]
                                                   
{'debug/num_tok_total': 2633.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2633.0, 'debug/num_lat_loss': 1771.0, 'epoch': 9.47}

 95%|█████████▍| 691/730 [1:01:22<03:34,  5.50s/it]
                                                   
{'train/ce_loss': 1.6640625, 'train/diffusion_loss': 0.40130603313446045, 'epoch': 9.47}

 95%|█████████▍| 691/730 [1:01:22<03:34,  5.50s/it]
                                                   
{'train/learning_rate_real': 1.8670603235371176e-07, 'epoch': 9.47}

 95%|█████████▍| 691/730 [1:01:22<03:34,  5.50s/it]
 95%|█████████▍| 692/730 [1:01:22<03:26,  5.45s/it]
                                                   
{'debug/num_tok_total': 2932.0, 'debug/num_tok_loss': 1820.0, 'debug/num_lat_total': 2932.0, 'debug/num_lat_loss': 1820.0, 'epoch': 9.48}

 95%|█████████▍| 692/730 [1:01:24<03:26,  5.45s/it]
                                                   
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.3788697123527527, 'epoch': 9.48}

 95%|█████████▍| 692/730 [1:01:24<03:26,  5.45s/it]
                                                   
{'train/learning_rate_real': 1.7727652288403174e-07, 'epoch': 9.48}

 95%|█████████▍| 692/730 [1:01:24<03:26,  5.45s/it]
                                                   
{'debug/num_tok_total': 3070.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 3070.0, 'debug/num_lat_loss': 1780.0, 'epoch': 9.48}

 95%|█████████▍| 692/730 [1:01:25<03:26,  5.45s/it]
                                                   
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.34722447395324707, 'epoch': 9.48}

 95%|█████████▍| 692/730 [1:01:25<03:26,  5.45s/it]
                                                   
{'train/learning_rate_real': 1.7727652288403174e-07, 'epoch': 9.48}

 95%|█████████▍| 692/730 [1:01:25<03:26,  5.45s/it]
                                                   
{'debug/num_tok_total': 1765.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 1765.0, 'debug/num_lat_loss': 1765.0, 'epoch': 9.48}

 95%|█████████▍| 692/730 [1:01:26<03:26,  5.45s/it]
                                                   
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.5842077136039734, 'epoch': 9.48}

 95%|█████████▍| 692/730 [1:01:26<03:26,  5.45s/it]
                                                   
{'train/learning_rate_real': 1.7727652288403174e-07, 'epoch': 9.48}

 95%|█████████▍| 692/730 [1:01:26<03:26,  5.45s/it]
                                                   
{'debug/num_tok_total': 3091.0, 'debug/num_tok_loss': 1793.0, 'debug/num_lat_total': 3091.0, 'debug/num_lat_loss': 1793.0, 'epoch': 9.48}

 95%|█████████▍| 692/730 [1:01:27<03:26,  5.45s/it]
                                                   
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.32360801100730896, 'epoch': 9.48}

 95%|█████████▍| 692/730 [1:01:27<03:26,  5.45s/it]
                                                   
{'train/learning_rate_real': 1.7727652288403174e-07, 'epoch': 9.48}

 95%|█████████▍| 692/730 [1:01:27<03:26,  5.45s/it]
 95%|█████████▍| 693/730 [1:01:28<03:18,  5.37s/it]
                                                   
{'debug/num_tok_total': 2682.0, 'debug/num_tok_loss': 1821.0, 'debug/num_lat_total': 2682.0, 'debug/num_lat_loss': 1821.0, 'epoch': 9.49}

 95%|█████████▍| 693/730 [1:01:29<03:18,  5.37s/it]
                                                   
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.3931686282157898, 'epoch': 9.49}

 95%|█████████▍| 693/730 [1:01:29<03:18,  5.37s/it]
                                                   
{'train/learning_rate_real': 1.680896404657864e-07, 'epoch': 9.49}

 95%|█████████▍| 693/730 [1:01:29<03:18,  5.37s/it]
                                                   
{'debug/num_tok_total': 2722.0, 'debug/num_tok_loss': 1616.0, 'debug/num_lat_total': 2722.0, 'debug/num_lat_loss': 1616.0, 'epoch': 9.49}

 95%|█████████▍| 693/730 [1:01:30<03:18,  5.37s/it]
                                                   
{'train/ce_loss': 1.6328125, 'train/diffusion_loss': 0.34922388195991516, 'epoch': 9.49}

 95%|█████████▍| 693/730 [1:01:30<03:18,  5.37s/it]
                                                   
{'train/learning_rate_real': 1.680896404657864e-07, 'epoch': 9.49}

 95%|█████████▍| 693/730 [1:01:30<03:18,  5.37s/it]
                                                   
{'debug/num_tok_total': 2401.0, 'debug/num_tok_loss': 1765.0, 'debug/num_lat_total': 2401.0, 'debug/num_lat_loss': 1765.0, 'epoch': 9.49}

 95%|█████████▍| 693/730 [1:01:31<03:18,  5.37s/it]
                                                   
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.43786799907684326, 'epoch': 9.49}

 95%|█████████▍| 693/730 [1:01:31<03:18,  5.37s/it]
                                                   
{'train/learning_rate_real': 1.680896404657864e-07, 'epoch': 9.49}

 95%|█████████▍| 693/730 [1:01:31<03:18,  5.37s/it]
                                                   
{'debug/num_tok_total': 2424.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2424.0, 'debug/num_lat_loss': 1766.0, 'epoch': 9.49}

 95%|█████████▍| 693/730 [1:01:32<03:18,  5.37s/it]
                                                   
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.4229397475719452, 'epoch': 9.49}

 95%|█████████▍| 693/730 [1:01:32<03:18,  5.37s/it]
                                                   
{'train/learning_rate_real': 1.680896404657864e-07, 'epoch': 9.49}

 95%|█████████▍| 693/730 [1:01:32<03:18,  5.37s/it]
 95%|█████████▌| 694/730 [1:01:33<03:12,  5.34s/it]
                                                   
{'debug/num_tok_total': 2918.0, 'debug/num_tok_loss': 1704.0, 'debug/num_lat_total': 2918.0, 'debug/num_lat_loss': 1704.0, 'epoch': 9.51}

 95%|█████████▌| 694/730 [1:01:34<03:12,  5.34s/it]
                                                   
{'train/ce_loss': 1.6484375, 'train/diffusion_loss': 0.30899664759635925, 'epoch': 9.51}

 95%|█████████▌| 694/730 [1:01:34<03:12,  5.34s/it]
                                                   
{'train/learning_rate_real': 1.5914556598319307e-07, 'epoch': 9.51}

 95%|█████████▌| 694/730 [1:01:34<03:12,  5.34s/it]
                                                   
{'debug/num_tok_total': 3127.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 3127.0, 'debug/num_lat_loss': 1801.0, 'epoch': 9.51}

 95%|█████████▌| 694/730 [1:01:35<03:12,  5.34s/it]
                                                   
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.3055134415626526, 'epoch': 9.51}

 95%|█████████▌| 694/730 [1:01:35<03:12,  5.34s/it]
                                                   
{'train/learning_rate_real': 1.5914556598319307e-07, 'epoch': 9.51}

 95%|█████████▌| 694/730 [1:01:35<03:12,  5.34s/it]
                                                   
{'debug/num_tok_total': 2079.0, 'debug/num_tok_loss': 1597.0, 'debug/num_lat_total': 2079.0, 'debug/num_lat_loss': 1597.0, 'epoch': 9.51}

 95%|█████████▌| 694/730 [1:01:36<03:12,  5.34s/it]
                                                   
{'train/ce_loss': 1.5859375, 'train/diffusion_loss': 0.4280615746974945, 'epoch': 9.51}

 95%|█████████▌| 694/730 [1:01:36<03:12,  5.34s/it]
                                                   
{'train/learning_rate_real': 1.5914556598319307e-07, 'epoch': 9.51}

 95%|█████████▌| 694/730 [1:01:36<03:12,  5.34s/it]
                                                   
{'debug/num_tok_total': 3096.0, 'debug/num_tok_loss': 1797.0, 'debug/num_lat_total': 3096.0, 'debug/num_lat_loss': 1797.0, 'epoch': 9.51}

 95%|█████████▌| 694/730 [1:01:38<03:12,  5.34s/it]
                                                   
{'train/ce_loss': 1.6015625, 'train/diffusion_loss': 0.3622535467147827, 'epoch': 9.51}

 95%|█████████▌| 694/730 [1:01:38<03:12,  5.34s/it]
                                                   
{'train/learning_rate_real': 1.5914556598319307e-07, 'epoch': 9.51}

 95%|█████████▌| 694/730 [1:01:38<03:12,  5.34s/it]
 95%|█████████▌| 695/730 [1:01:38<03:07,  5.37s/it]
                                                   
{'debug/num_tok_total': 2645.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2645.0, 'debug/num_lat_loss': 1777.0, 'epoch': 9.52}

 95%|█████████▌| 695/730 [1:01:39<03:07,  5.37s/it]
                                                   
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.38599494099617004, 'epoch': 9.52}

 95%|█████████▌| 695/730 [1:01:39<03:07,  5.37s/it]
                                                   
{'train/learning_rate_real': 1.5044447553972664e-07, 'epoch': 9.52}

 95%|█████████▌| 695/730 [1:01:39<03:07,  5.37s/it]
                                                   
{'debug/num_tok_total': 2198.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2198.0, 'debug/num_lat_loss': 1771.0, 'epoch': 9.52}

 95%|█████████▌| 695/730 [1:01:41<03:07,  5.37s/it]
                                                   
{'train/ce_loss': 2.015625, 'train/diffusion_loss': 0.5096920132637024, 'epoch': 9.52}

 95%|█████████▌| 695/730 [1:01:41<03:07,  5.37s/it]
                                                   
{'train/learning_rate_real': 1.5044447553972664e-07, 'epoch': 9.52}

 95%|█████████▌| 695/730 [1:01:41<03:07,  5.37s/it]
                                                   
{'debug/num_tok_total': 2199.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2199.0, 'debug/num_lat_loss': 1770.0, 'epoch': 9.52}

 95%|█████████▌| 695/730 [1:01:42<03:07,  5.37s/it]
                                                   
{'train/ce_loss': 1.8515625, 'train/diffusion_loss': 0.47885861992836, 'epoch': 9.52}

 95%|█████████▌| 695/730 [1:01:42<03:07,  5.37s/it]
                                                   
{'train/learning_rate_real': 1.5044447553972664e-07, 'epoch': 9.52}

 95%|█████████▌| 695/730 [1:01:42<03:07,  5.37s/it]
                                                   
{'debug/num_tok_total': 2235.0, 'debug/num_tok_loss': 1809.0, 'debug/num_lat_total': 2235.0, 'debug/num_lat_loss': 1809.0, 'epoch': 9.52}

 95%|█████████▌| 695/730 [1:01:43<03:07,  5.37s/it]
                                                   
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.4682413339614868, 'epoch': 9.52}

 95%|█████████▌| 695/730 [1:01:43<03:07,  5.37s/it]
                                                   
{'train/learning_rate_real': 1.5044447553972664e-07, 'epoch': 9.52}

 95%|█████████▌| 695/730 [1:01:43<03:07,  5.37s/it]
 95%|█████████▌| 696/730 [1:01:43<02:58,  5.26s/it]
                                                   
{'debug/num_tok_total': 2357.0, 'debug/num_tok_loss': 1712.0, 'debug/num_lat_total': 2357.0, 'debug/num_lat_loss': 1712.0, 'epoch': 9.53}

 95%|█████████▌| 696/730 [1:01:44<02:58,  5.26s/it]
                                                   
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.44130364060401917, 'epoch': 9.53}

 95%|█████████▌| 696/730 [1:01:44<02:58,  5.26s/it]
                                                   
{'train/learning_rate_real': 1.419865404546486e-07, 'epoch': 9.53}

 95%|█████████▌| 696/730 [1:01:44<02:58,  5.26s/it]
                                                   
{'debug/num_tok_total': 2601.0, 'debug/num_tok_loss': 1754.0, 'debug/num_lat_total': 2601.0, 'debug/num_lat_loss': 1754.0, 'epoch': 9.53}

 95%|█████████▌| 696/730 [1:01:46<02:58,  5.26s/it]
                                                   
{'train/ce_loss': 1.6875, 'train/diffusion_loss': 0.40242642164230347, 'epoch': 9.53}

 95%|█████████▌| 696/730 [1:01:46<02:58,  5.26s/it]
                                                   
{'train/learning_rate_real': 1.419865404546486e-07, 'epoch': 9.53}

 95%|█████████▌| 696/730 [1:01:46<02:58,  5.26s/it]
                                                   
{'debug/num_tok_total': 2470.0, 'debug/num_tok_loss': 1817.0, 'debug/num_lat_total': 2470.0, 'debug/num_lat_loss': 1817.0, 'epoch': 9.53}

 95%|█████████▌| 696/730 [1:01:47<02:58,  5.26s/it]
                                                   
{'train/ce_loss': 1.640625, 'train/diffusion_loss': 0.43924930691719055, 'epoch': 9.53}

 95%|█████████▌| 696/730 [1:01:47<02:58,  5.26s/it]
                                                   
{'train/learning_rate_real': 1.419865404546486e-07, 'epoch': 9.53}

 95%|█████████▌| 696/730 [1:01:47<02:58,  5.26s/it]
                                                   
{'debug/num_tok_total': 2856.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2856.0, 'debug/num_lat_loss': 1783.0, 'epoch': 9.53}

 95%|█████████▌| 696/730 [1:01:48<02:58,  5.26s/it]
                                                   
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.4007810950279236, 'epoch': 9.53}

 95%|█████████▌| 696/730 [1:01:48<02:58,  5.26s/it]
                                                   
{'train/learning_rate_real': 1.419865404546486e-07, 'epoch': 9.53}

 95%|█████████▌| 696/730 [1:01:48<02:58,  5.26s/it]
 95%|█████████▌| 697/730 [1:01:49<02:54,  5.28s/it]
                                                   
{'debug/num_tok_total': 3040.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 3040.0, 'debug/num_lat_loss': 1770.0, 'epoch': 9.55}

 95%|█████████▌| 697/730 [1:01:50<02:54,  5.28s/it]
                                                   
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.3225712180137634, 'epoch': 9.55}

 95%|█████████▌| 697/730 [1:01:50<02:54,  5.28s/it]
                                                   
{'train/learning_rate_real': 1.3377192725964327e-07, 'epoch': 9.55}

 95%|█████████▌| 697/730 [1:01:50<02:54,  5.28s/it]
                                                   
{'debug/num_tok_total': 1864.0, 'debug/num_tok_loss': 1651.0, 'debug/num_lat_total': 1864.0, 'debug/num_lat_loss': 1651.0, 'epoch': 9.55}

 95%|█████████▌| 697/730 [1:01:51<02:54,  5.28s/it]
                                                   
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.5283498167991638, 'epoch': 9.55}

 95%|█████████▌| 697/730 [1:01:51<02:54,  5.28s/it]
                                                   
{'train/learning_rate_real': 1.3377192725964327e-07, 'epoch': 9.55}

 95%|█████████▌| 697/730 [1:01:51<02:54,  5.28s/it]
                                                   
{'debug/num_tok_total': 2416.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2416.0, 'debug/num_lat_loss': 1775.0, 'epoch': 9.55}

 95%|█████████▌| 697/730 [1:01:52<02:54,  5.28s/it]
                                                   
{'train/ce_loss': 1.6484375, 'train/diffusion_loss': 0.44085338711738586, 'epoch': 9.55}

 95%|█████████▌| 697/730 [1:01:52<02:54,  5.28s/it]
                                                   
{'train/learning_rate_real': 1.3377192725964327e-07, 'epoch': 9.55}

 95%|█████████▌| 697/730 [1:01:52<02:54,  5.28s/it]
                                                   
{'debug/num_tok_total': 2403.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2403.0, 'debug/num_lat_loss': 1774.0, 'epoch': 9.55}

 95%|█████████▌| 697/730 [1:01:53<02:54,  5.28s/it]
                                                   
{'train/ce_loss': 1.625, 'train/diffusion_loss': 0.4403183162212372, 'epoch': 9.55}

 95%|█████████▌| 697/730 [1:01:53<02:54,  5.28s/it]
                                                   
{'train/learning_rate_real': 1.3377192725964327e-07, 'epoch': 9.55}

 95%|█████████▌| 697/730 [1:01:53<02:54,  5.28s/it]
 96%|█████████▌| 698/730 [1:01:54<02:46,  5.19s/it]
                                                   
{'debug/num_tok_total': 2006.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2006.0, 'debug/num_lat_loss': 1788.0, 'epoch': 9.56}

 96%|█████████▌| 698/730 [1:01:55<02:46,  5.19s/it]
                                                   
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.5193678140640259, 'epoch': 9.56}

 96%|█████████▌| 698/730 [1:01:55<02:46,  5.19s/it]
                                                   
{'train/learning_rate_real': 1.2580079769552722e-07, 'epoch': 9.56}

 96%|█████████▌| 698/730 [1:01:55<02:46,  5.19s/it]
                                                   
{'debug/num_tok_total': 2854.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 2854.0, 'debug/num_lat_loss': 1772.0, 'epoch': 9.56}

 96%|█████████▌| 698/730 [1:01:56<02:46,  5.19s/it]
                                                   
{'train/ce_loss': 1.9375, 'train/diffusion_loss': 0.3780355155467987, 'epoch': 9.56}

 96%|█████████▌| 698/730 [1:01:56<02:46,  5.19s/it]
                                                   
{'train/learning_rate_real': 1.2580079769552722e-07, 'epoch': 9.56}

 96%|█████████▌| 698/730 [1:01:56<02:46,  5.19s/it]
                                                   
{'debug/num_tok_total': 1977.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 1977.0, 'debug/num_lat_loss': 1772.0, 'epoch': 9.56}

 96%|█████████▌| 698/730 [1:01:57<02:46,  5.19s/it]
                                                   
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.5437299013137817, 'epoch': 9.56}

 96%|█████████▌| 698/730 [1:01:57<02:46,  5.19s/it]
                                                   
{'train/learning_rate_real': 1.2580079769552722e-07, 'epoch': 9.56}

 96%|█████████▌| 698/730 [1:01:57<02:46,  5.19s/it]
                                                   
{'debug/num_tok_total': 2411.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2411.0, 'debug/num_lat_loss': 1774.0, 'epoch': 9.56}

 96%|█████████▌| 698/730 [1:01:58<02:46,  5.19s/it]
                                                   
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.42947497963905334, 'epoch': 9.56}

 96%|█████████▌| 698/730 [1:01:58<02:46,  5.19s/it]
                                                   
{'train/learning_rate_real': 1.2580079769552722e-07, 'epoch': 9.56}

 96%|█████████▌| 698/730 [1:01:58<02:46,  5.19s/it]
 96%|█████████▌| 699/730 [1:01:58<02:37,  5.09s/it]
                                                   
{'debug/num_tok_total': 2401.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 2401.0, 'debug/num_lat_loss': 1767.0, 'epoch': 9.58}

 96%|█████████▌| 699/730 [1:01:59<02:37,  5.09s/it]
                                                   
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.48144063353538513, 'epoch': 9.58}

 96%|█████████▌| 699/730 [1:01:59<02:37,  5.09s/it]
                                                   
{'train/learning_rate_real': 1.180733087090713e-07, 'epoch': 9.58}

 96%|█████████▌| 699/730 [1:01:59<02:37,  5.09s/it]
                                                   
{'debug/num_tok_total': 2820.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2820.0, 'debug/num_lat_loss': 1764.0, 'epoch': 9.58}

 96%|█████████▌| 699/730 [1:02:01<02:37,  5.09s/it]
                                                   
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.3487808406352997, 'epoch': 9.58}

 96%|█████████▌| 699/730 [1:02:01<02:37,  5.09s/it]
                                                   
{'train/learning_rate_real': 1.180733087090713e-07, 'epoch': 9.58}

 96%|█████████▌| 699/730 [1:02:01<02:37,  5.09s/it]
                                                   
{'debug/num_tok_total': 2398.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2398.0, 'debug/num_lat_loss': 1764.0, 'epoch': 9.58}

 96%|█████████▌| 699/730 [1:02:02<02:37,  5.09s/it]
                                                   
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.46408864855766296, 'epoch': 9.58}

 96%|█████████▌| 699/730 [1:02:02<02:37,  5.09s/it]
                                                   
{'train/learning_rate_real': 1.180733087090713e-07, 'epoch': 9.58}

 96%|█████████▌| 699/730 [1:02:02<02:37,  5.09s/it]
                                                   
{'debug/num_tok_total': 2699.0, 'debug/num_tok_loss': 1826.0, 'debug/num_lat_total': 2699.0, 'debug/num_lat_loss': 1826.0, 'epoch': 9.58}

 96%|█████████▌| 699/730 [1:02:03<02:37,  5.09s/it]
                                                   
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.4172912836074829, 'epoch': 9.58}

 96%|█████████▌| 699/730 [1:02:03<02:37,  5.09s/it]
                                                   
{'train/learning_rate_real': 1.180733087090713e-07, 'epoch': 9.58}

 96%|█████████▌| 699/730 [1:02:03<02:37,  5.09s/it]03/16/2026 07:49:36 - INFO - __main__ - LoRA debug step 700: changed A 195/196, changed B 196/196, lora_B_zero_now=0.

 96%|█████████▌| 700/730 [1:02:04<02:35,  5.17s/it]
                                                   
{'loss': 2.6216, 'grad_norm': 1.2621430158615112, 'learning_rate': 1.180733087090713e-07, 'epoch': 9.59}

 96%|█████████▌| 700/730 [1:02:04<02:35,  5.17s/it]
                                                   
{'debug/num_tok_total': 2241.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 2241.0, 'debug/num_lat_loss': 1801.0, 'epoch': 9.59}

 96%|█████████▌| 700/730 [1:02:05<02:35,  5.17s/it]
                                                   
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.4956441819667816, 'epoch': 9.59}

 96%|█████████▌| 700/730 [1:02:05<02:35,  5.17s/it]
                                                   
{'train/learning_rate_real': 1.1058961244991144e-07, 'epoch': 9.59}

 96%|█████████▌| 700/730 [1:02:05<02:35,  5.17s/it]
                                                   
{'debug/num_tok_total': 2653.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 2653.0, 'debug/num_lat_loss': 1792.0, 'epoch': 9.59}

 96%|█████████▌| 700/730 [1:02:06<02:35,  5.17s/it]
                                                   
{'train/ce_loss': 1.5625, 'train/diffusion_loss': 0.3868063986301422, 'epoch': 9.59}

 96%|█████████▌| 700/730 [1:02:06<02:35,  5.17s/it]
                                                   
{'train/learning_rate_real': 1.1058961244991144e-07, 'epoch': 9.59}

 96%|█████████▌| 700/730 [1:02:06<02:35,  5.17s/it]
                                                   
{'debug/num_tok_total': 2420.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2420.0, 'debug/num_lat_loss': 1780.0, 'epoch': 9.59}

 96%|█████████▌| 700/730 [1:02:07<02:35,  5.17s/it]
                                                   
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.43790626525878906, 'epoch': 9.59}

 96%|█████████▌| 700/730 [1:02:07<02:35,  5.17s/it]
                                                   
{'train/learning_rate_real': 1.1058961244991144e-07, 'epoch': 9.59}

 96%|█████████▌| 700/730 [1:02:07<02:35,  5.17s/it]
                                                   
{'debug/num_tok_total': 2635.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2635.0, 'debug/num_lat_loss': 1774.0, 'epoch': 9.59}

 96%|█████████▌| 700/730 [1:02:08<02:35,  5.17s/it]
                                                   
{'train/ce_loss': 1.9609375, 'train/diffusion_loss': 0.4034920930862427, 'epoch': 9.59}

 96%|█████████▌| 700/730 [1:02:08<02:35,  5.17s/it]
                                                   
{'train/learning_rate_real': 1.1058961244991144e-07, 'epoch': 9.59}

 96%|█████████▌| 700/730 [1:02:08<02:35,  5.17s/it]
 96%|█████████▌| 701/730 [1:02:09<02:29,  5.17s/it]
                                                   
{'debug/num_tok_total': 1911.0, 'debug/num_tok_loss': 1691.0, 'debug/num_lat_total': 1911.0, 'debug/num_lat_loss': 1691.0, 'epoch': 9.6}

 96%|█████████▌| 701/730 [1:02:10<02:29,  5.17s/it]
                                                   
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.5285207033157349, 'epoch': 9.6}

 96%|█████████▌| 701/730 [1:02:10<02:29,  5.17s/it]
                                                   
{'train/learning_rate_real': 1.0334985626754556e-07, 'epoch': 9.6}

 96%|█████████▌| 701/730 [1:02:10<02:29,  5.17s/it]
                                                   
{'debug/num_tok_total': 2407.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2407.0, 'debug/num_lat_loss': 1764.0, 'epoch': 9.6}

 96%|█████████▌| 701/730 [1:02:11<02:29,  5.17s/it]
                                                   
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.43502768874168396, 'epoch': 9.6}

 96%|█████████▌| 701/730 [1:02:11<02:29,  5.17s/it]
                                                   
{'train/learning_rate_real': 1.0334985626754556e-07, 'epoch': 9.6}

 96%|█████████▌| 701/730 [1:02:11<02:29,  5.17s/it]
                                                   
{'debug/num_tok_total': 2445.0, 'debug/num_tok_loss': 1802.0, 'debug/num_lat_total': 2445.0, 'debug/num_lat_loss': 1802.0, 'epoch': 9.6}

 96%|█████████▌| 701/730 [1:02:12<02:29,  5.17s/it]
                                                   
{'train/ce_loss': 1.578125, 'train/diffusion_loss': 0.4240323007106781, 'epoch': 9.6}

 96%|█████████▌| 701/730 [1:02:12<02:29,  5.17s/it]
                                                   
{'train/learning_rate_real': 1.0334985626754556e-07, 'epoch': 9.6}

 96%|█████████▌| 701/730 [1:02:12<02:29,  5.17s/it]
                                                   
{'debug/num_tok_total': 2458.0, 'debug/num_tok_loss': 1804.0, 'debug/num_lat_total': 2458.0, 'debug/num_lat_loss': 1804.0, 'epoch': 9.6}

 96%|█████████▌| 701/730 [1:02:13<02:29,  5.17s/it]
                                                   
{'train/ce_loss': 1.5078125, 'train/diffusion_loss': 0.4224860668182373, 'epoch': 9.6}

 96%|█████████▌| 701/730 [1:02:13<02:29,  5.17s/it]
                                                   
{'train/learning_rate_real': 1.0334985626754556e-07, 'epoch': 9.6}

 96%|█████████▌| 701/730 [1:02:13<02:29,  5.17s/it]
 96%|█████████▌| 702/730 [1:02:14<02:21,  5.06s/it]
                                                   
{'debug/num_tok_total': 2851.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2851.0, 'debug/num_lat_loss': 1777.0, 'epoch': 9.62}

 96%|█████████▌| 702/730 [1:02:15<02:21,  5.06s/it]
                                                   
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.36427396535873413, 'epoch': 9.62}

 96%|█████████▌| 702/730 [1:02:15<02:21,  5.06s/it]
                                                   
{'train/learning_rate_real': 9.63541827084427e-08, 'epoch': 9.62}

 96%|█████████▌| 702/730 [1:02:15<02:21,  5.06s/it]
                                                   
{'debug/num_tok_total': 2425.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2425.0, 'debug/num_lat_loss': 1782.0, 'epoch': 9.62}

 96%|█████████▌| 702/730 [1:02:16<02:21,  5.06s/it]
                                                   
{'train/ce_loss': 1.578125, 'train/diffusion_loss': 0.43577179312705994, 'epoch': 9.62}

 96%|█████████▌| 702/730 [1:02:16<02:21,  5.06s/it]
                                                   
{'train/learning_rate_real': 9.63541827084427e-08, 'epoch': 9.62}

 96%|█████████▌| 702/730 [1:02:16<02:21,  5.06s/it]
                                                   
{'debug/num_tok_total': 2403.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2403.0, 'debug/num_lat_loss': 1764.0, 'epoch': 9.62}

 96%|█████████▌| 702/730 [1:02:17<02:21,  5.06s/it]
                                                   
{'train/ce_loss': 2.0, 'train/diffusion_loss': 0.43532517552375793, 'epoch': 9.62}

 96%|█████████▌| 702/730 [1:02:17<02:21,  5.06s/it]
                                                   
{'train/learning_rate_real': 9.63541827084427e-08, 'epoch': 9.62}

 96%|█████████▌| 702/730 [1:02:17<02:21,  5.06s/it]
                                                   
{'debug/num_tok_total': 2894.0, 'debug/num_tok_loss': 1807.0, 'debug/num_lat_total': 2894.0, 'debug/num_lat_loss': 1807.0, 'epoch': 9.62}

 96%|█████████▌| 702/730 [1:02:19<02:21,  5.06s/it]
                                                   
{'train/ce_loss': 1.6328125, 'train/diffusion_loss': 0.37546589970588684, 'epoch': 9.62}

 96%|█████████▌| 702/730 [1:02:19<02:21,  5.06s/it]
                                                   
{'train/learning_rate_real': 9.63541827084427e-08, 'epoch': 9.62}

 96%|█████████▌| 702/730 [1:02:19<02:21,  5.06s/it]
 96%|█████████▋| 703/730 [1:02:19<02:18,  5.11s/it]
                                                   
{'debug/num_tok_total': 2428.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2428.0, 'debug/num_lat_loss': 1782.0, 'epoch': 9.63}

 96%|█████████▋| 703/730 [1:02:20<02:18,  5.11s/it]
                                                   
{'train/ce_loss': 1.953125, 'train/diffusion_loss': 0.43466341495513916, 'epoch': 9.63}

 96%|█████████▋| 703/730 [1:02:20<02:18,  5.11s/it]
                                                   
{'train/learning_rate_real': 8.960272951323012e-08, 'epoch': 9.63}

 96%|█████████▋| 703/730 [1:02:20<02:18,  5.11s/it]
                                                   
{'debug/num_tok_total': 3059.0, 'debug/num_tok_loss': 1772.0, 'debug/num_lat_total': 3059.0, 'debug/num_lat_loss': 1772.0, 'epoch': 9.63}

 96%|█████████▋| 703/730 [1:02:21<02:18,  5.11s/it]
                                                   
{'train/ce_loss': 1.6875, 'train/diffusion_loss': 0.29623809456825256, 'epoch': 9.63}

 96%|█████████▋| 703/730 [1:02:21<02:18,  5.11s/it]
                                                   
{'train/learning_rate_real': 8.960272951323012e-08, 'epoch': 9.63}

 96%|█████████▋| 703/730 [1:02:21<02:18,  5.11s/it]
                                                   
{'debug/num_tok_total': 3106.0, 'debug/num_tok_loss': 1798.0, 'debug/num_lat_total': 3106.0, 'debug/num_lat_loss': 1798.0, 'epoch': 9.63}

 96%|█████████▋| 703/730 [1:02:23<02:18,  5.11s/it]
                                                   
{'train/ce_loss': 1.9609375, 'train/diffusion_loss': 0.30641260743141174, 'epoch': 9.63}

 96%|█████████▋| 703/730 [1:02:23<02:18,  5.11s/it]
                                                   
{'train/learning_rate_real': 8.960272951323012e-08, 'epoch': 9.63}

 96%|█████████▋| 703/730 [1:02:23<02:18,  5.11s/it]
                                                   
{'debug/num_tok_total': 2878.0, 'debug/num_tok_loss': 1795.0, 'debug/num_lat_total': 2878.0, 'debug/num_lat_loss': 1795.0, 'epoch': 9.63}

 96%|█████████▋| 703/730 [1:02:24<02:18,  5.11s/it]
                                                   
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.3505135476589203, 'epoch': 9.63}

 96%|█████████▋| 703/730 [1:02:24<02:18,  5.11s/it]
                                                   
{'train/learning_rate_real': 8.960272951323012e-08, 'epoch': 9.63}

 96%|█████████▋| 703/730 [1:02:24<02:18,  5.11s/it]
 96%|█████████▋| 704/730 [1:02:25<02:17,  5.28s/it]
                                                   
{'debug/num_tok_total': 2311.0, 'debug/num_tok_loss': 1721.0, 'debug/num_lat_total': 2311.0, 'debug/num_lat_loss': 1721.0, 'epoch': 9.64}

 96%|█████████▋| 704/730 [1:02:26<02:17,  5.28s/it]
                                                   
{'train/ce_loss': 1.96875, 'train/diffusion_loss': 0.4674111008644104, 'epoch': 9.64}

 96%|█████████▋| 704/730 [1:02:26<02:17,  5.28s/it]
                                                   
{'train/learning_rate_real': 8.30956296139801e-08, 'epoch': 9.64}

 96%|█████████▋| 704/730 [1:02:26<02:17,  5.28s/it]
                                                   
{'debug/num_tok_total': 2872.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2872.0, 'debug/num_lat_loss': 1783.0, 'epoch': 9.64}

 96%|█████████▋| 704/730 [1:02:27<02:17,  5.28s/it]
                                                   
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.34188514947891235, 'epoch': 9.64}

 96%|█████████▋| 704/730 [1:02:27<02:17,  5.28s/it]
                                                   
{'train/learning_rate_real': 8.30956296139801e-08, 'epoch': 9.64}

 96%|█████████▋| 704/730 [1:02:27<02:17,  5.28s/it]
                                                   
{'debug/num_tok_total': 2458.0, 'debug/num_tok_loss': 1803.0, 'debug/num_lat_total': 2458.0, 'debug/num_lat_loss': 1803.0, 'epoch': 9.64}

 96%|█████████▋| 704/730 [1:02:28<02:17,  5.28s/it]
                                                   
{'train/ce_loss': 1.875, 'train/diffusion_loss': 0.4340336322784424, 'epoch': 9.64}

 96%|█████████▋| 704/730 [1:02:28<02:17,  5.28s/it]
                                                   
{'train/learning_rate_real': 8.30956296139801e-08, 'epoch': 9.64}

 96%|█████████▋| 704/730 [1:02:28<02:17,  5.28s/it]
                                                   
{'debug/num_tok_total': 3056.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 3056.0, 'debug/num_lat_loss': 1775.0, 'epoch': 9.64}

 96%|█████████▋| 704/730 [1:02:30<02:17,  5.28s/it]
                                                   
{'train/ce_loss': 1.59375, 'train/diffusion_loss': 0.29883047938346863, 'epoch': 9.64}

 96%|█████████▋| 704/730 [1:02:30<02:17,  5.28s/it]
                                                   
{'train/learning_rate_real': 8.30956296139801e-08, 'epoch': 9.64}

 96%|█████████▋| 704/730 [1:02:30<02:17,  5.28s/it]
 97%|█████████▋| 705/730 [1:02:30<02:12,  5.30s/it]
                                                   
{'debug/num_tok_total': 2627.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 2627.0, 'debug/num_lat_loss': 1767.0, 'epoch': 9.66}

 97%|█████████▋| 705/730 [1:02:31<02:12,  5.30s/it]
                                                   
{'train/ce_loss': 1.8203125, 'train/diffusion_loss': 0.42639684677124023, 'epoch': 9.66}

 97%|█████████▋| 705/730 [1:02:31<02:12,  5.30s/it]
                                                   
{'train/learning_rate_real': 7.683301113159541e-08, 'epoch': 9.66}

 97%|█████████▋| 705/730 [1:02:31<02:12,  5.30s/it]
                                                   
{'debug/num_tok_total': 2177.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2177.0, 'debug/num_lat_loss': 1763.0, 'epoch': 9.66}

 97%|█████████▋| 705/730 [1:02:32<02:12,  5.30s/it]
                                                   
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.4822332561016083, 'epoch': 9.66}

 97%|█████████▋| 705/730 [1:02:32<02:12,  5.30s/it]
                                                   
{'train/learning_rate_real': 7.683301113159541e-08, 'epoch': 9.66}

 97%|█████████▋| 705/730 [1:02:32<02:12,  5.30s/it]
                                                   
{'debug/num_tok_total': 2626.0, 'debug/num_tok_loss': 1769.0, 'debug/num_lat_total': 2626.0, 'debug/num_lat_loss': 1769.0, 'epoch': 9.66}

 97%|█████████▋| 705/730 [1:02:33<02:12,  5.30s/it]
                                                   
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.4060656428337097, 'epoch': 9.66}

 97%|█████████▋| 705/730 [1:02:33<02:12,  5.30s/it]
                                                   
{'train/learning_rate_real': 7.683301113159541e-08, 'epoch': 9.66}

 97%|█████████▋| 705/730 [1:02:33<02:12,  5.30s/it]
                                                   
{'debug/num_tok_total': 2829.0, 'debug/num_tok_loss': 1759.0, 'debug/num_lat_total': 2829.0, 'debug/num_lat_loss': 1759.0, 'epoch': 9.66}

 97%|█████████▋| 705/730 [1:02:35<02:12,  5.30s/it]
                                                   
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.3616708219051361, 'epoch': 9.66}

 97%|█████████▋| 705/730 [1:02:35<02:12,  5.30s/it]
                                                   
{'train/learning_rate_real': 7.683301113159541e-08, 'epoch': 9.66}

 97%|█████████▋| 705/730 [1:02:35<02:12,  5.30s/it]
 97%|█████████▋| 706/730 [1:02:35<02:05,  5.24s/it]
                                                   
{'debug/num_tok_total': 3054.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 3054.0, 'debug/num_lat_loss': 1771.0, 'epoch': 9.67}

 97%|█████████▋| 706/730 [1:02:36<02:05,  5.24s/it]
                                                   
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.3385874330997467, 'epoch': 9.67}

 97%|█████████▋| 706/730 [1:02:36<02:05,  5.24s/it]
                                                   
{'train/learning_rate_real': 7.081499737328634e-08, 'epoch': 9.67}

 97%|█████████▋| 706/730 [1:02:36<02:05,  5.24s/it]
                                                   
{'debug/num_tok_total': 2251.0, 'debug/num_tok_loss': 1807.0, 'debug/num_lat_total': 2251.0, 'debug/num_lat_loss': 1807.0, 'epoch': 9.67}

 97%|█████████▋| 706/730 [1:02:37<02:05,  5.24s/it]
                                                   
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.5062499046325684, 'epoch': 9.67}

 97%|█████████▋| 706/730 [1:02:37<02:05,  5.24s/it]
                                                   
{'train/learning_rate_real': 7.081499737328634e-08, 'epoch': 9.67}

 97%|█████████▋| 706/730 [1:02:37<02:05,  5.24s/it]
                                                   
{'debug/num_tok_total': 3119.0, 'debug/num_tok_loss': 1805.0, 'debug/num_lat_total': 3119.0, 'debug/num_lat_loss': 1805.0, 'epoch': 9.67}

 97%|█████████▋| 706/730 [1:02:39<02:05,  5.24s/it]
                                                   
{'train/ce_loss': 1.640625, 'train/diffusion_loss': 0.34558552503585815, 'epoch': 9.67}

 97%|█████████▋| 706/730 [1:02:39<02:05,  5.24s/it]
                                                   
{'train/learning_rate_real': 7.081499737328634e-08, 'epoch': 9.67}

 97%|█████████▋| 706/730 [1:02:39<02:05,  5.24s/it]
                                                   
{'debug/num_tok_total': 2642.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2642.0, 'debug/num_lat_loss': 1784.0, 'epoch': 9.67}

 97%|█████████▋| 706/730 [1:02:40<02:05,  5.24s/it]
                                                   
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.39465242624282837, 'epoch': 9.67}

 97%|█████████▋| 706/730 [1:02:40<02:05,  5.24s/it]
                                                   
{'train/learning_rate_real': 7.081499737328634e-08, 'epoch': 9.67}

 97%|█████████▋| 706/730 [1:02:40<02:05,  5.24s/it]
 97%|█████████▋| 707/730 [1:02:41<02:02,  5.30s/it]
                                                   
{'debug/num_tok_total': 2643.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2643.0, 'debug/num_lat_loss': 1782.0, 'epoch': 9.68}

 97%|█████████▋| 707/730 [1:02:42<02:02,  5.30s/it]
                                                   
{'train/ce_loss': 1.8828125, 'train/diffusion_loss': 0.39784106612205505, 'epoch': 9.68}

 97%|█████████▋| 707/730 [1:02:42<02:02,  5.30s/it]
                                                   
{'train/learning_rate_real': 6.504170683014343e-08, 'epoch': 9.68}

 97%|█████████▋| 707/730 [1:02:42<02:02,  5.30s/it]
                                                   
{'debug/num_tok_total': 2440.0, 'debug/num_tok_loss': 1789.0, 'debug/num_lat_total': 2440.0, 'debug/num_lat_loss': 1789.0, 'epoch': 9.68}

 97%|█████████▋| 707/730 [1:02:43<02:02,  5.30s/it]
                                                   
{'train/ce_loss': 1.890625, 'train/diffusion_loss': 0.44118279218673706, 'epoch': 9.68}

 97%|█████████▋| 707/730 [1:02:43<02:02,  5.30s/it]
                                                   
{'train/learning_rate_real': 6.504170683014343e-08, 'epoch': 9.68}

 97%|█████████▋| 707/730 [1:02:43<02:02,  5.30s/it]
                                                   
{'debug/num_tok_total': 2319.0, 'debug/num_tok_loss': 1712.0, 'debug/num_lat_total': 2319.0, 'debug/num_lat_loss': 1712.0, 'epoch': 9.68}

 97%|█████████▋| 707/730 [1:02:44<02:02,  5.30s/it]
                                                   
{'train/ce_loss': 1.90625, 'train/diffusion_loss': 0.47346311807632446, 'epoch': 9.68}

 97%|█████████▋| 707/730 [1:02:44<02:02,  5.30s/it]
                                                   
{'train/learning_rate_real': 6.504170683014343e-08, 'epoch': 9.68}

 97%|█████████▋| 707/730 [1:02:44<02:02,  5.30s/it]
                                                   
{'debug/num_tok_total': 2905.0, 'debug/num_tok_loss': 1808.0, 'debug/num_lat_total': 2905.0, 'debug/num_lat_loss': 1808.0, 'epoch': 9.68}

 97%|█████████▋| 707/730 [1:02:45<02:02,  5.30s/it]
                                                   
{'train/ce_loss': 1.8515625, 'train/diffusion_loss': 0.3560147285461426, 'epoch': 9.68}

 97%|█████████▋| 707/730 [1:02:45<02:02,  5.30s/it]
                                                   
{'train/learning_rate_real': 6.504170683014343e-08, 'epoch': 9.68}

 97%|█████████▋| 707/730 [1:02:45<02:02,  5.30s/it]
 97%|█████████▋| 708/730 [1:02:46<01:56,  5.30s/it]
                                                   
{'debug/num_tok_total': 2849.0, 'debug/num_tok_loss': 1780.0, 'debug/num_lat_total': 2849.0, 'debug/num_lat_loss': 1780.0, 'epoch': 9.7}

 97%|█████████▋| 708/730 [1:02:47<01:56,  5.30s/it]
                                                   
{'train/ce_loss': 1.65625, 'train/diffusion_loss': 0.3673299551010132, 'epoch': 9.7}

 97%|█████████▋| 708/730 [1:02:47<01:56,  5.30s/it]
                                                   
{'train/learning_rate_real': 5.951325317480189e-08, 'epoch': 9.7}

 97%|█████████▋| 708/730 [1:02:47<01:56,  5.30s/it]
                                                   
{'debug/num_tok_total': 1895.0, 'debug/num_tok_loss': 1584.0, 'debug/num_lat_total': 1895.0, 'debug/num_lat_loss': 1584.0, 'epoch': 9.7}

 97%|█████████▋| 708/730 [1:02:48<01:56,  5.30s/it]
                                                   
{'train/ce_loss': 1.8515625, 'train/diffusion_loss': 0.5336419939994812, 'epoch': 9.7}

 97%|█████████▋| 708/730 [1:02:48<01:56,  5.30s/it]
                                                   
{'train/learning_rate_real': 5.951325317480189e-08, 'epoch': 9.7}

 97%|█████████▋| 708/730 [1:02:48<01:56,  5.30s/it]
                                                   
{'debug/num_tok_total': 2863.0, 'debug/num_tok_loss': 1787.0, 'debug/num_lat_total': 2863.0, 'debug/num_lat_loss': 1787.0, 'epoch': 9.7}

 97%|█████████▋| 708/730 [1:02:50<01:56,  5.30s/it]
                                                   
{'train/ce_loss': 1.578125, 'train/diffusion_loss': 0.3259786367416382, 'epoch': 9.7}

 97%|█████████▋| 708/730 [1:02:50<01:56,  5.30s/it]
                                                   
{'train/learning_rate_real': 5.951325317480189e-08, 'epoch': 9.7}

 97%|█████████▋| 708/730 [1:02:50<01:56,  5.30s/it]
                                                   
{'debug/num_tok_total': 2849.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2849.0, 'debug/num_lat_loss': 1783.0, 'epoch': 9.7}

 97%|█████████▋| 708/730 [1:02:51<01:56,  5.30s/it]
                                                   
{'train/ce_loss': 1.6640625, 'train/diffusion_loss': 0.383899986743927, 'epoch': 9.7}

 97%|█████████▋| 708/730 [1:02:51<01:56,  5.30s/it]
                                                   
{'train/learning_rate_real': 5.951325317480189e-08, 'epoch': 9.7}

 97%|█████████▋| 708/730 [1:02:51<01:56,  5.30s/it]
 97%|█████████▋| 709/730 [1:02:51<01:51,  5.33s/it]
                                                   
{'debug/num_tok_total': 2879.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2879.0, 'debug/num_lat_loss': 1799.0, 'epoch': 9.71}

 97%|█████████▋| 709/730 [1:02:52<01:51,  5.33s/it]
                                                   
{'train/ce_loss': 1.890625, 'train/diffusion_loss': 0.345508337020874, 'epoch': 9.71}

 97%|█████████▋| 709/730 [1:02:52<01:51,  5.33s/it]
                                                   
{'train/learning_rate_real': 5.422974525920721e-08, 'epoch': 9.71}

 97%|█████████▋| 709/730 [1:02:52<01:51,  5.33s/it]
                                                   
{'debug/num_tok_total': 2200.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2200.0, 'debug/num_lat_loss': 1775.0, 'epoch': 9.71}

 97%|█████████▋| 709/730 [1:02:54<01:51,  5.33s/it]
                                                   
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.5013023614883423, 'epoch': 9.71}

 97%|█████████▋| 709/730 [1:02:54<01:51,  5.33s/it]
                                                   
{'train/learning_rate_real': 5.422974525920721e-08, 'epoch': 9.71}

 97%|█████████▋| 709/730 [1:02:54<01:51,  5.33s/it]
                                                   
{'debug/num_tok_total': 2592.0, 'debug/num_tok_loss': 1757.0, 'debug/num_lat_total': 2592.0, 'debug/num_lat_loss': 1757.0, 'epoch': 9.71}

 97%|█████████▋| 709/730 [1:02:55<01:51,  5.33s/it]
                                                   
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.41165047883987427, 'epoch': 9.71}

 97%|█████████▋| 709/730 [1:02:55<01:51,  5.33s/it]
                                                   
{'train/learning_rate_real': 5.422974525920721e-08, 'epoch': 9.71}

 97%|█████████▋| 709/730 [1:02:55<01:51,  5.33s/it]
                                                   
{'debug/num_tok_total': 1992.0, 'debug/num_tok_loss': 1682.0, 'debug/num_lat_total': 1992.0, 'debug/num_lat_loss': 1682.0, 'epoch': 9.71}

 97%|█████████▋| 709/730 [1:02:56<01:51,  5.33s/it]
                                                   
{'train/ce_loss': 1.7421875, 'train/diffusion_loss': 0.49568971991539, 'epoch': 9.71}

 97%|█████████▋| 709/730 [1:02:56<01:51,  5.33s/it]
                                                   
{'train/learning_rate_real': 5.422974525920721e-08, 'epoch': 9.71}

 97%|█████████▋| 709/730 [1:02:56<01:51,  5.33s/it]03/16/2026 07:50:28 - INFO - __main__ - LoRA debug step 710: changed A 194/196, changed B 196/196, lora_B_zero_now=0.

 97%|█████████▋| 710/730 [1:02:57<01:46,  5.32s/it]
                                                   
{'loss': 2.5756, 'grad_norm': 1.4254082441329956, 'learning_rate': 5.422974525920721e-08, 'epoch': 9.73}

 97%|█████████▋| 710/730 [1:02:57<01:46,  5.32s/it]
                                                   
{'debug/num_tok_total': 2585.0, 'debug/num_tok_loss': 1717.0, 'debug/num_lat_total': 2585.0, 'debug/num_lat_loss': 1717.0, 'epoch': 9.73}

 97%|█████████▋| 710/730 [1:02:58<01:46,  5.32s/it]
                                                   
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.39922332763671875, 'epoch': 9.73}

 97%|█████████▋| 710/730 [1:02:58<01:46,  5.32s/it]
                                                   
{'train/learning_rate_real': 4.919128711246834e-08, 'epoch': 9.73}

 97%|█████████▋| 710/730 [1:02:58<01:46,  5.32s/it]
                                                   
{'debug/num_tok_total': 2631.0, 'debug/num_tok_loss': 1783.0, 'debug/num_lat_total': 2631.0, 'debug/num_lat_loss': 1783.0, 'epoch': 9.73}

 97%|█████████▋| 710/730 [1:02:59<01:46,  5.32s/it]
                                                   
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.4310094118118286, 'epoch': 9.73}

 97%|█████████▋| 710/730 [1:02:59<01:46,  5.32s/it]
                                                   
{'train/learning_rate_real': 4.919128711246834e-08, 'epoch': 9.73}

 97%|█████████▋| 710/730 [1:02:59<01:46,  5.32s/it]
                                                   
{'debug/num_tok_total': 2652.0, 'debug/num_tok_loss': 1584.0, 'debug/num_lat_total': 2652.0, 'debug/num_lat_loss': 1584.0, 'epoch': 9.73}

 97%|█████████▋| 710/730 [1:03:00<01:46,  5.32s/it]
                                                   
{'train/ce_loss': 1.625, 'train/diffusion_loss': 0.338084876537323, 'epoch': 9.73}

 97%|█████████▋| 710/730 [1:03:00<01:46,  5.32s/it]
                                                   
{'train/learning_rate_real': 4.919128711246834e-08, 'epoch': 9.73}

 97%|█████████▋| 710/730 [1:03:00<01:46,  5.32s/it]
                                                   
{'debug/num_tok_total': 2471.0, 'debug/num_tok_loss': 1820.0, 'debug/num_lat_total': 2471.0, 'debug/num_lat_loss': 1820.0, 'epoch': 9.73}

 97%|█████████▋| 710/730 [1:03:01<01:46,  5.32s/it]
                                                   
{'train/ce_loss': 1.65625, 'train/diffusion_loss': 0.4350317716598511, 'epoch': 9.73}

 97%|█████████▋| 710/730 [1:03:01<01:46,  5.32s/it]
                                                   
{'train/learning_rate_real': 4.919128711246834e-08, 'epoch': 9.73}

 97%|█████████▋| 710/730 [1:03:01<01:46,  5.32s/it]
 97%|█████████▋| 711/730 [1:03:02<01:41,  5.32s/it]
                                                   
{'debug/num_tok_total': 2662.0, 'debug/num_tok_loss': 1786.0, 'debug/num_lat_total': 2662.0, 'debug/num_lat_loss': 1786.0, 'epoch': 9.74}

 97%|█████████▋| 711/730 [1:03:03<01:41,  5.32s/it]
                                                   
{'train/ce_loss': 1.6015625, 'train/diffusion_loss': 0.3902421295642853, 'epoch': 9.74}

 97%|█████████▋| 711/730 [1:03:03<01:41,  5.32s/it]
                                                   
{'train/learning_rate_real': 4.4397977938810675e-08, 'epoch': 9.74}

 97%|█████████▋| 711/730 [1:03:03<01:41,  5.32s/it]
                                                   
{'debug/num_tok_total': 2208.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2208.0, 'debug/num_lat_loss': 1785.0, 'epoch': 9.74}

 97%|█████████▋| 711/730 [1:03:04<01:41,  5.32s/it]
                                                   
{'train/ce_loss': 1.9609375, 'train/diffusion_loss': 0.5114588141441345, 'epoch': 9.74}

 97%|█████████▋| 711/730 [1:03:04<01:41,  5.32s/it]
                                                   
{'train/learning_rate_real': 4.4397977938810675e-08, 'epoch': 9.74}

 97%|█████████▋| 711/730 [1:03:04<01:41,  5.32s/it]
                                                   
{'debug/num_tok_total': 2279.0, 'debug/num_tok_loss': 1626.0, 'debug/num_lat_total': 2279.0, 'debug/num_lat_loss': 1626.0, 'epoch': 9.74}

 97%|█████████▋| 711/730 [1:03:05<01:41,  5.32s/it]
                                                   
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.44641950726509094, 'epoch': 9.74}

 97%|█████████▋| 711/730 [1:03:05<01:41,  5.32s/it]
                                                   
{'train/learning_rate_real': 4.4397977938810675e-08, 'epoch': 9.74}

 97%|█████████▋| 711/730 [1:03:05<01:41,  5.32s/it]
                                                   
{'debug/num_tok_total': 1982.0, 'debug/num_tok_loss': 1762.0, 'debug/num_lat_total': 1982.0, 'debug/num_lat_loss': 1762.0, 'epoch': 9.74}

 97%|█████████▋| 711/730 [1:03:06<01:41,  5.32s/it]
                                                   
{'train/ce_loss': 1.96875, 'train/diffusion_loss': 0.524817705154419, 'epoch': 9.74}

 97%|█████████▋| 711/730 [1:03:06<01:41,  5.32s/it]
                                                   
{'train/learning_rate_real': 4.4397977938810675e-08, 'epoch': 9.74}

 97%|█████████▋| 711/730 [1:03:06<01:41,  5.32s/it]
 98%|█████████▊| 712/730 [1:03:07<01:33,  5.21s/it]
                                                   
{'debug/num_tok_total': 2619.0, 'debug/num_tok_loss': 1770.0, 'debug/num_lat_total': 2619.0, 'debug/num_lat_loss': 1770.0, 'epoch': 9.75}

 98%|█████████▊| 712/730 [1:03:08<01:33,  5.21s/it]
                                                   
{'train/ce_loss': 1.6171875, 'train/diffusion_loss': 0.39153748750686646, 'epoch': 9.75}

 98%|█████████▊| 712/730 [1:03:08<01:33,  5.21s/it]
                                                   
{'train/learning_rate_real': 3.9849912115623446e-08, 'epoch': 9.75}

 98%|█████████▊| 712/730 [1:03:08<01:33,  5.21s/it]
                                                   
{'debug/num_tok_total': 3038.0, 'debug/num_tok_loss': 1761.0, 'debug/num_lat_total': 3038.0, 'debug/num_lat_loss': 1761.0, 'epoch': 9.75}

 98%|█████████▊| 712/730 [1:03:09<01:33,  5.21s/it]
                                                   
{'train/ce_loss': 1.59375, 'train/diffusion_loss': 0.32352352142333984, 'epoch': 9.75}

 98%|█████████▊| 712/730 [1:03:09<01:33,  5.21s/it]
                                                   
{'train/learning_rate_real': 3.9849912115623446e-08, 'epoch': 9.75}

 98%|█████████▊| 712/730 [1:03:09<01:33,  5.21s/it]
                                                   
{'debug/num_tok_total': 2310.0, 'debug/num_tok_loss': 1622.0, 'debug/num_lat_total': 2310.0, 'debug/num_lat_loss': 1622.0, 'epoch': 9.75}

 98%|█████████▊| 712/730 [1:03:10<01:33,  5.21s/it]
                                                   
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.4297730326652527, 'epoch': 9.75}

 98%|█████████▊| 712/730 [1:03:10<01:33,  5.21s/it]
                                                   
{'train/learning_rate_real': 3.9849912115623446e-08, 'epoch': 9.75}

 98%|█████████▊| 712/730 [1:03:10<01:33,  5.21s/it]
                                                   
{'debug/num_tok_total': 2863.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2863.0, 'debug/num_lat_loss': 1788.0, 'epoch': 9.75}

 98%|█████████▊| 712/730 [1:03:12<01:33,  5.21s/it]
                                                   
{'train/ce_loss': 1.59375, 'train/diffusion_loss': 0.3572707176208496, 'epoch': 9.75}

 98%|█████████▊| 712/730 [1:03:12<01:33,  5.21s/it]
                                                   
{'train/learning_rate_real': 3.9849912115623446e-08, 'epoch': 9.75}

 98%|█████████▊| 712/730 [1:03:12<01:33,  5.21s/it]
 98%|█████████▊| 713/730 [1:03:12<01:29,  5.24s/it]
                                                   
{'debug/num_tok_total': 1896.0, 'debug/num_tok_loss': 1621.0, 'debug/num_lat_total': 1896.0, 'debug/num_lat_loss': 1621.0, 'epoch': 9.77}

 98%|█████████▊| 713/730 [1:03:13<01:29,  5.24s/it]
                                                   
{'train/ce_loss': 1.625, 'train/diffusion_loss': 0.5068615674972534, 'epoch': 9.77}

 98%|█████████▊| 713/730 [1:03:13<01:29,  5.24s/it]
                                                   
{'train/learning_rate_real': 3.55471791916015e-08, 'epoch': 9.77}

 98%|█████████▊| 713/730 [1:03:13<01:29,  5.24s/it]
                                                   
{'debug/num_tok_total': 2635.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2635.0, 'debug/num_lat_loss': 1782.0, 'epoch': 9.77}

 98%|█████████▊| 713/730 [1:03:14<01:29,  5.24s/it]
                                                   
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.4365597665309906, 'epoch': 9.77}

 98%|█████████▊| 713/730 [1:03:14<01:29,  5.24s/it]
                                                   
{'train/learning_rate_real': 3.55471791916015e-08, 'epoch': 9.77}

 98%|█████████▊| 713/730 [1:03:14<01:29,  5.24s/it]
                                                   
{'debug/num_tok_total': 2649.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2649.0, 'debug/num_lat_loss': 1777.0, 'epoch': 9.77}

 98%|█████████▊| 713/730 [1:03:16<01:29,  5.24s/it]
                                                   
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.4015595614910126, 'epoch': 9.77}

 98%|█████████▊| 713/730 [1:03:16<01:29,  5.24s/it]
                                                   
{'train/learning_rate_real': 3.55471791916015e-08, 'epoch': 9.77}

 98%|█████████▊| 713/730 [1:03:16<01:29,  5.24s/it]
                                                   
{'debug/num_tok_total': 3112.0, 'debug/num_tok_loss': 1806.0, 'debug/num_lat_total': 3112.0, 'debug/num_lat_loss': 1806.0, 'epoch': 9.77}

 98%|█████████▊| 713/730 [1:03:17<01:29,  5.24s/it]
                                                   
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.31700658798217773, 'epoch': 9.77}

 98%|█████████▊| 713/730 [1:03:17<01:29,  5.24s/it]
                                                   
{'train/learning_rate_real': 3.55471791916015e-08, 'epoch': 9.77}

 98%|█████████▊| 713/730 [1:03:17<01:29,  5.24s/it]
 98%|█████████▊| 714/730 [1:03:18<01:24,  5.31s/it]
                                                   
{'debug/num_tok_total': 2486.0, 'debug/num_tok_loss': 1819.0, 'debug/num_lat_total': 2486.0, 'debug/num_lat_loss': 1819.0, 'epoch': 9.78}

 98%|█████████▊| 714/730 [1:03:19<01:24,  5.31s/it]
                                                   
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.4624653160572052, 'epoch': 9.78}

 98%|█████████▊| 714/730 [1:03:19<01:24,  5.31s/it]
                                                   
{'train/learning_rate_real': 3.148986388498143e-08, 'epoch': 9.78}

 98%|█████████▊| 714/730 [1:03:19<01:24,  5.31s/it]
                                                   
{'debug/num_tok_total': 2407.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 2407.0, 'debug/num_lat_loss': 1767.0, 'epoch': 9.78}

 98%|█████████▊| 714/730 [1:03:20<01:24,  5.31s/it]
                                                   
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.4392907917499542, 'epoch': 9.78}

 98%|█████████▊| 714/730 [1:03:20<01:24,  5.31s/it]
                                                   
{'train/learning_rate_real': 3.148986388498143e-08, 'epoch': 9.78}

 98%|█████████▊| 714/730 [1:03:20<01:24,  5.31s/it]
                                                   
{'debug/num_tok_total': 2824.0, 'debug/num_tok_loss': 1762.0, 'debug/num_lat_total': 2824.0, 'debug/num_lat_loss': 1762.0, 'epoch': 9.78}

 98%|█████████▊| 714/730 [1:03:21<01:24,  5.31s/it]
                                                   
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.35687097907066345, 'epoch': 9.78}

 98%|█████████▊| 714/730 [1:03:21<01:24,  5.31s/it]
                                                   
{'train/learning_rate_real': 3.148986388498143e-08, 'epoch': 9.78}

 98%|█████████▊| 714/730 [1:03:21<01:24,  5.31s/it]
                                                   
{'debug/num_tok_total': 2179.0, 'debug/num_tok_loss': 1764.0, 'debug/num_lat_total': 2179.0, 'debug/num_lat_loss': 1764.0, 'epoch': 9.78}

 98%|█████████▊| 714/730 [1:03:22<01:24,  5.31s/it]
                                                   
{'train/ce_loss': 1.671875, 'train/diffusion_loss': 0.48587295413017273, 'epoch': 9.78}

 98%|█████████▊| 714/730 [1:03:22<01:24,  5.31s/it]
                                                   
{'train/learning_rate_real': 3.148986388498143e-08, 'epoch': 9.78}

 98%|█████████▊| 714/730 [1:03:22<01:24,  5.31s/it]
 98%|█████████▊| 715/730 [1:03:23<01:18,  5.23s/it]
                                                   
{'debug/num_tok_total': 2698.0, 'debug/num_tok_loss': 1695.0, 'debug/num_lat_total': 2698.0, 'debug/num_lat_loss': 1695.0, 'epoch': 9.79}

 98%|█████████▊| 715/730 [1:03:24<01:18,  5.23s/it]
                                                   
{'train/ce_loss': 1.90625, 'train/diffusion_loss': 0.35792991518974304, 'epoch': 9.79}

 98%|█████████▊| 715/730 [1:03:24<01:18,  5.23s/it]
                                                   
{'train/learning_rate_real': 2.7678046081874853e-08, 'epoch': 9.79}

 98%|█████████▊| 715/730 [1:03:24<01:18,  5.23s/it]
                                                   
{'debug/num_tok_total': 3091.0, 'debug/num_tok_loss': 1792.0, 'debug/num_lat_total': 3091.0, 'debug/num_lat_loss': 1792.0, 'epoch': 9.79}

 98%|█████████▊| 715/730 [1:03:25<01:18,  5.23s/it]
                                                   
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.3093399703502655, 'epoch': 9.79}

 98%|█████████▊| 715/730 [1:03:25<01:18,  5.23s/it]
                                                   
{'train/learning_rate_real': 2.7678046081874853e-08, 'epoch': 9.79}

 98%|█████████▊| 715/730 [1:03:25<01:18,  5.23s/it]
                                                   
{'debug/num_tok_total': 2910.0, 'debug/num_tok_loss': 1813.0, 'debug/num_lat_total': 2910.0, 'debug/num_lat_loss': 1813.0, 'epoch': 9.79}

 98%|█████████▊| 715/730 [1:03:27<01:18,  5.23s/it]
                                                   
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.3510631024837494, 'epoch': 9.79}

 98%|█████████▊| 715/730 [1:03:27<01:18,  5.23s/it]
                                                   
{'train/learning_rate_real': 2.7678046081874853e-08, 'epoch': 9.79}

 98%|█████████▊| 715/730 [1:03:27<01:18,  5.23s/it]
                                                   
{'debug/num_tok_total': 2432.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2432.0, 'debug/num_lat_loss': 1781.0, 'epoch': 9.79}

 98%|█████████▊| 715/730 [1:03:28<01:18,  5.23s/it]
                                                   
{'train/ce_loss': 1.7734375, 'train/diffusion_loss': 0.4531056880950928, 'epoch': 9.79}

 98%|█████████▊| 715/730 [1:03:28<01:18,  5.23s/it]
                                                   
{'train/learning_rate_real': 2.7678046081874853e-08, 'epoch': 9.79}

 98%|█████████▊| 715/730 [1:03:28<01:18,  5.23s/it]
 98%|█████████▊| 716/730 [1:03:28<01:15,  5.38s/it]
                                                   
{'debug/num_tok_total': 2810.0, 'debug/num_tok_loss': 1760.0, 'debug/num_lat_total': 2810.0, 'debug/num_lat_loss': 1760.0, 'epoch': 9.81}

 98%|█████████▊| 716/730 [1:03:30<01:15,  5.38s/it]
                                                   
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.39560502767562866, 'epoch': 9.81}

 98%|█████████▊| 716/730 [1:03:30<01:15,  5.38s/it]
                                                   
{'train/learning_rate_real': 2.41118008346905e-08, 'epoch': 9.81}

 98%|█████████▊| 716/730 [1:03:30<01:15,  5.38s/it]
                                                   
{'debug/num_tok_total': 2627.0, 'debug/num_tok_loss': 1573.0, 'debug/num_lat_total': 2627.0, 'debug/num_lat_loss': 1573.0, 'epoch': 9.81}

 98%|█████████▊| 716/730 [1:03:31<01:15,  5.38s/it]
                                                   
{'train/ce_loss': 1.6171875, 'train/diffusion_loss': 0.32605454325675964, 'epoch': 9.81}

 98%|█████████▊| 716/730 [1:03:31<01:15,  5.38s/it]
                                                   
{'train/learning_rate_real': 2.41118008346905e-08, 'epoch': 9.81}

 98%|█████████▊| 716/730 [1:03:31<01:15,  5.38s/it]
                                                   
{'debug/num_tok_total': 2409.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2409.0, 'debug/num_lat_loss': 1777.0, 'epoch': 9.81}

 98%|█████████▊| 716/730 [1:03:32<01:15,  5.38s/it]
                                                   
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.4562707841396332, 'epoch': 9.81}

 98%|█████████▊| 716/730 [1:03:32<01:15,  5.38s/it]
                                                   
{'train/learning_rate_real': 2.41118008346905e-08, 'epoch': 9.81}

 98%|█████████▊| 716/730 [1:03:32<01:15,  5.38s/it]
                                                   
{'debug/num_tok_total': 2877.0, 'debug/num_tok_loss': 1784.0, 'debug/num_lat_total': 2877.0, 'debug/num_lat_loss': 1784.0, 'epoch': 9.81}

 98%|█████████▊| 716/730 [1:03:33<01:15,  5.38s/it]
                                                   
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.35457560420036316, 'epoch': 9.81}

 98%|█████████▊| 716/730 [1:03:33<01:15,  5.38s/it]
                                                   
{'train/learning_rate_real': 2.41118008346905e-08, 'epoch': 9.81}

 98%|█████████▊| 716/730 [1:03:33<01:15,  5.38s/it]
 98%|█████████▊| 717/730 [1:03:34<01:10,  5.42s/it]
                                                   
{'debug/num_tok_total': 2625.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2625.0, 'debug/num_lat_loss': 1776.0, 'epoch': 9.82}

 98%|█████████▊| 717/730 [1:03:35<01:10,  5.42s/it]
                                                   
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.3906170129776001, 'epoch': 9.82}

 98%|█████████▊| 717/730 [1:03:35<01:10,  5.42s/it]
                                                   
{'train/learning_rate_real': 2.0791198360665955e-08, 'epoch': 9.82}

 98%|█████████▊| 717/730 [1:03:35<01:10,  5.42s/it]
                                                   
{'debug/num_tok_total': 3065.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 3065.0, 'debug/num_lat_loss': 1777.0, 'epoch': 9.82}

 98%|█████████▊| 717/730 [1:03:36<01:10,  5.42s/it]
                                                   
{'train/ce_loss': 1.6015625, 'train/diffusion_loss': 0.29998779296875, 'epoch': 9.82}

 98%|█████████▊| 717/730 [1:03:36<01:10,  5.42s/it]
                                                   
{'train/learning_rate_real': 2.0791198360665955e-08, 'epoch': 9.82}

 98%|█████████▊| 717/730 [1:03:36<01:10,  5.42s/it]
                                                   
{'debug/num_tok_total': 3090.0, 'debug/num_tok_loss': 1800.0, 'debug/num_lat_total': 3090.0, 'debug/num_lat_loss': 1800.0, 'epoch': 9.82}

 98%|█████████▊| 717/730 [1:03:38<01:10,  5.42s/it]
                                                   
{'train/ce_loss': 1.578125, 'train/diffusion_loss': 0.3508630394935608, 'epoch': 9.82}

 98%|█████████▊| 717/730 [1:03:38<01:10,  5.42s/it]
                                                   
{'train/learning_rate_real': 2.0791198360665955e-08, 'epoch': 9.82}

 98%|█████████▊| 717/730 [1:03:38<01:10,  5.42s/it]
                                                   
{'debug/num_tok_total': 2816.0, 'debug/num_tok_loss': 1759.0, 'debug/num_lat_total': 2816.0, 'debug/num_lat_loss': 1759.0, 'epoch': 9.82}

 98%|█████████▊| 717/730 [1:03:39<01:10,  5.42s/it]
                                                   
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.34101805090904236, 'epoch': 9.82}

 98%|█████████▊| 717/730 [1:03:39<01:10,  5.42s/it]
                                                   
{'train/learning_rate_real': 2.0791198360665955e-08, 'epoch': 9.82}

 98%|█████████▊| 717/730 [1:03:39<01:10,  5.42s/it]
 98%|█████████▊| 718/730 [1:03:39<01:05,  5.47s/it]
                                                   
{'debug/num_tok_total': 2207.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2207.0, 'debug/num_lat_loss': 1779.0, 'epoch': 9.84}

 98%|█████████▊| 718/730 [1:03:40<01:05,  5.47s/it]
                                                   
{'train/ce_loss': 2.140625, 'train/diffusion_loss': 0.4924617409706116, 'epoch': 9.84}

 98%|█████████▊| 718/730 [1:03:40<01:05,  5.47s/it]
                                                   
{'train/learning_rate_real': 1.7716304040475697e-08, 'epoch': 9.84}

 98%|█████████▊| 718/730 [1:03:40<01:05,  5.47s/it]
                                                   
{'debug/num_tok_total': 2233.0, 'debug/num_tok_loss': 1806.0, 'debug/num_lat_total': 2233.0, 'debug/num_lat_loss': 1806.0, 'epoch': 9.84}

 98%|█████████▊| 718/730 [1:03:41<01:05,  5.47s/it]
                                                   
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.4905601143836975, 'epoch': 9.84}

 98%|█████████▊| 718/730 [1:03:42<01:05,  5.47s/it]
                                                   
{'train/learning_rate_real': 1.7716304040475697e-08, 'epoch': 9.84}

 98%|█████████▊| 718/730 [1:03:42<01:05,  5.47s/it]
                                                   
{'debug/num_tok_total': 2016.0, 'debug/num_tok_loss': 1596.0, 'debug/num_lat_total': 2016.0, 'debug/num_lat_loss': 1596.0, 'epoch': 9.84}

 98%|█████████▊| 718/730 [1:03:43<01:05,  5.47s/it]
                                                   
{'train/ce_loss': 1.671875, 'train/diffusion_loss': 0.48611798882484436, 'epoch': 9.84}

 98%|█████████▊| 718/730 [1:03:43<01:05,  5.47s/it]
                                                   
{'train/learning_rate_real': 1.7716304040475697e-08, 'epoch': 9.84}

 98%|█████████▊| 718/730 [1:03:43<01:05,  5.47s/it]
                                                   
{'debug/num_tok_total': 2647.0, 'debug/num_tok_loss': 1782.0, 'debug/num_lat_total': 2647.0, 'debug/num_lat_loss': 1782.0, 'epoch': 9.84}

 98%|█████████▊| 718/730 [1:03:44<01:05,  5.47s/it]
                                                   
{'train/ce_loss': 1.8515625, 'train/diffusion_loss': 0.4236856698989868, 'epoch': 9.84}

 98%|█████████▊| 718/730 [1:03:44<01:05,  5.47s/it]
                                                   
{'train/learning_rate_real': 1.7716304040475697e-08, 'epoch': 9.84}

 98%|█████████▊| 718/730 [1:03:44<01:05,  5.47s/it]
 98%|█████████▊| 719/730 [1:03:44<00:58,  5.30s/it]
                                                   
{'debug/num_tok_total': 2638.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2638.0, 'debug/num_lat_loss': 1779.0, 'epoch': 9.85}

 98%|█████████▊| 719/730 [1:03:46<00:58,  5.30s/it]
                                                   
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.3959931433200836, 'epoch': 9.85}

 98%|█████████▊| 719/730 [1:03:46<00:58,  5.30s/it]
                                                   
{'train/learning_rate_real': 1.488717841694881e-08, 'epoch': 9.85}

 98%|█████████▊| 719/730 [1:03:46<00:58,  5.30s/it]
                                                   
{'debug/num_tok_total': 2205.0, 'debug/num_tok_loss': 1771.0, 'debug/num_lat_total': 2205.0, 'debug/num_lat_loss': 1771.0, 'epoch': 9.85}

 98%|█████████▊| 719/730 [1:03:47<00:58,  5.30s/it]
                                                   
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.49415236711502075, 'epoch': 9.85}

 98%|█████████▊| 719/730 [1:03:47<00:58,  5.30s/it]
                                                   
{'train/learning_rate_real': 1.488717841694881e-08, 'epoch': 9.85}

 98%|█████████▊| 719/730 [1:03:47<00:58,  5.30s/it]
                                                   
{'debug/num_tok_total': 2452.0, 'debug/num_tok_loss': 1799.0, 'debug/num_lat_total': 2452.0, 'debug/num_lat_loss': 1799.0, 'epoch': 9.85}

 98%|█████████▊| 719/730 [1:03:48<00:58,  5.30s/it]
                                                   
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.44007769227027893, 'epoch': 9.85}

 98%|█████████▊| 719/730 [1:03:48<00:58,  5.30s/it]
                                                   
{'train/learning_rate_real': 1.488717841694881e-08, 'epoch': 9.85}

 98%|█████████▊| 719/730 [1:03:48<00:58,  5.30s/it]
                                                   
{'debug/num_tok_total': 2444.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2444.0, 'debug/num_lat_loss': 1796.0, 'epoch': 9.85}

 98%|█████████▊| 719/730 [1:03:49<00:58,  5.30s/it]
                                                   
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.4525756239891052, 'epoch': 9.85}

 98%|█████████▊| 719/730 [1:03:49<00:58,  5.30s/it]
                                                   
{'train/learning_rate_real': 1.488717841694881e-08, 'epoch': 9.85}

 98%|█████████▊| 719/730 [1:03:49<00:58,  5.30s/it]03/16/2026 07:51:22 - INFO - __main__ - LoRA debug step 720: changed A 194/196, changed B 196/196, lora_B_zero_now=0.

 99%|█████████▊| 720/730 [1:03:50<00:53,  5.38s/it]
                                                   
{'loss': 2.5825, 'grad_norm': 1.269769549369812, 'learning_rate': 1.488717841694881e-08, 'epoch': 9.86}

 99%|█████████▊| 720/730 [1:03:50<00:53,  5.38s/it]
                                                   
{'debug/num_tok_total': 2429.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2429.0, 'debug/num_lat_loss': 1774.0, 'epoch': 9.86}

 99%|█████████▊| 720/730 [1:03:51<00:53,  5.38s/it]
                                                   
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.4237443506717682, 'epoch': 9.86}

 99%|█████████▊| 720/730 [1:03:51<00:53,  5.38s/it]
                                                   
{'train/learning_rate_real': 1.2303877193876878e-08, 'epoch': 9.86}

 99%|█████████▊| 720/730 [1:03:51<00:53,  5.38s/it]
                                                   
{'debug/num_tok_total': 2180.0, 'debug/num_tok_loss': 1760.0, 'debug/num_lat_total': 2180.0, 'debug/num_lat_loss': 1760.0, 'epoch': 9.86}

 99%|█████████▊| 720/730 [1:03:52<00:53,  5.38s/it]
                                                   
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.5000243186950684, 'epoch': 9.86}

 99%|█████████▊| 720/730 [1:03:52<00:53,  5.38s/it]
                                                   
{'train/learning_rate_real': 1.2303877193876878e-08, 'epoch': 9.86}

 99%|█████████▊| 720/730 [1:03:52<00:53,  5.38s/it]
                                                   
{'debug/num_tok_total': 2885.0, 'debug/num_tok_loss': 1800.0, 'debug/num_lat_total': 2885.0, 'debug/num_lat_loss': 1800.0, 'epoch': 9.86}

 99%|█████████▊| 720/730 [1:03:53<00:53,  5.38s/it]
                                                   
{'train/ce_loss': 1.546875, 'train/diffusion_loss': 0.3634382486343384, 'epoch': 9.86}

 99%|█████████▊| 720/730 [1:03:53<00:53,  5.38s/it]
                                                   
{'train/learning_rate_real': 1.2303877193876878e-08, 'epoch': 9.86}

 99%|█████████▊| 720/730 [1:03:53<00:53,  5.38s/it]
                                                   
{'debug/num_tok_total': 2172.0, 'debug/num_tok_loss': 1756.0, 'debug/num_lat_total': 2172.0, 'debug/num_lat_loss': 1756.0, 'epoch': 9.86}

 99%|█████████▊| 720/730 [1:03:55<00:53,  5.38s/it]
                                                   
{'train/ce_loss': 1.8828125, 'train/diffusion_loss': 0.49669474363327026, 'epoch': 9.86}

 99%|█████████▊| 720/730 [1:03:55<00:53,  5.38s/it]
                                                   
{'train/learning_rate_real': 1.2303877193876878e-08, 'epoch': 9.86}

 99%|█████████▊| 720/730 [1:03:55<00:53,  5.38s/it]
 99%|█████████▉| 721/730 [1:03:55<00:48,  5.35s/it]
                                                   
{'debug/num_tok_total': 2454.0, 'debug/num_tok_loss': 1806.0, 'debug/num_lat_total': 2454.0, 'debug/num_lat_loss': 1806.0, 'epoch': 9.88}

 99%|█████████▉| 721/730 [1:03:56<00:48,  5.35s/it]
                                                   
{'train/ce_loss': 1.6484375, 'train/diffusion_loss': 0.45906752347946167, 'epoch': 9.88}

 99%|█████████▉| 721/730 [1:03:56<00:48,  5.35s/it]
                                                   
{'train/learning_rate_real': 9.966451234914853e-09, 'epoch': 9.88}

 99%|█████████▉| 721/730 [1:03:56<00:48,  5.35s/it]
                                                   
{'debug/num_tok_total': 2191.0, 'debug/num_tok_loss': 1766.0, 'debug/num_lat_total': 2191.0, 'debug/num_lat_loss': 1766.0, 'epoch': 9.88}

 99%|█████████▉| 721/730 [1:03:57<00:48,  5.35s/it]
                                                   
{'train/ce_loss': 1.859375, 'train/diffusion_loss': 0.4698774516582489, 'epoch': 9.88}

 99%|█████████▉| 721/730 [1:03:57<00:48,  5.35s/it]
                                                   
{'train/learning_rate_real': 9.966451234914853e-09, 'epoch': 9.88}

 99%|█████████▉| 721/730 [1:03:57<00:48,  5.35s/it]
                                                   
{'debug/num_tok_total': 2701.0, 'debug/num_tok_loss': 1819.0, 'debug/num_lat_total': 2701.0, 'debug/num_lat_loss': 1819.0, 'epoch': 9.88}

 99%|█████████▉| 721/730 [1:03:59<00:48,  5.35s/it]
                                                   
{'train/ce_loss': 1.9453125, 'train/diffusion_loss': 0.3908277750015259, 'epoch': 9.88}

 99%|█████████▉| 721/730 [1:03:59<00:48,  5.35s/it]
                                                   
{'train/learning_rate_real': 9.966451234914853e-09, 'epoch': 9.88}

 99%|█████████▉| 721/730 [1:03:59<00:48,  5.35s/it]
                                                   
{'debug/num_tok_total': 2637.0, 'debug/num_tok_loss': 1776.0, 'debug/num_lat_total': 2637.0, 'debug/num_lat_loss': 1776.0, 'epoch': 9.88}

 99%|█████████▉| 721/730 [1:04:00<00:48,  5.35s/it]
                                                   
{'train/ce_loss': 1.796875, 'train/diffusion_loss': 0.4251360297203064, 'epoch': 9.88}

 99%|█████████▉| 721/730 [1:04:00<00:48,  5.35s/it]
                                                   
{'train/learning_rate_real': 9.966451234914853e-09, 'epoch': 9.88}

 99%|█████████▉| 721/730 [1:04:00<00:48,  5.35s/it]
 99%|█████████▉| 722/730 [1:04:01<00:42,  5.36s/it]
                                                   
{'debug/num_tok_total': 2467.0, 'debug/num_tok_loss': 1816.0, 'debug/num_lat_total': 2467.0, 'debug/num_lat_loss': 1816.0, 'epoch': 9.89}

 99%|█████████▉| 722/730 [1:04:02<00:42,  5.36s/it]
                                                   
{'train/ce_loss': 1.6796875, 'train/diffusion_loss': 0.44154879450798035, 'epoch': 9.89}

 99%|█████████▉| 722/730 [1:04:02<00:42,  5.36s/it]
                                                   
{'train/learning_rate_real': 7.874946562580477e-09, 'epoch': 9.89}

 99%|█████████▉| 722/730 [1:04:02<00:42,  5.36s/it]
                                                   
{'debug/num_tok_total': 2467.0, 'debug/num_tok_loss': 1801.0, 'debug/num_lat_total': 2467.0, 'debug/num_lat_loss': 1801.0, 'epoch': 9.89}

 99%|█████████▉| 722/730 [1:04:03<00:42,  5.36s/it]
                                                   
{'train/ce_loss': 1.84375, 'train/diffusion_loss': 0.4292648732662201, 'epoch': 9.89}

 99%|█████████▉| 722/730 [1:04:03<00:42,  5.36s/it]
                                                   
{'train/learning_rate_real': 7.874946562580477e-09, 'epoch': 9.89}

 99%|█████████▉| 722/730 [1:04:03<00:42,  5.36s/it]
                                                   
{'debug/num_tok_total': 2763.0, 'debug/num_tok_loss': 1625.0, 'debug/num_lat_total': 2763.0, 'debug/num_lat_loss': 1625.0, 'epoch': 9.89}

 99%|█████████▉| 722/730 [1:04:04<00:42,  5.36s/it]
                                                   
{'train/ce_loss': 1.640625, 'train/diffusion_loss': 0.3253788352012634, 'epoch': 9.89}

 99%|█████████▉| 722/730 [1:04:04<00:42,  5.36s/it]
                                                   
{'train/learning_rate_real': 7.874946562580477e-09, 'epoch': 9.89}

 99%|█████████▉| 722/730 [1:04:04<00:42,  5.36s/it]
                                                   
{'debug/num_tok_total': 2668.0, 'debug/num_tok_loss': 1691.0, 'debug/num_lat_total': 2668.0, 'debug/num_lat_loss': 1691.0, 'epoch': 9.89}

 99%|█████████▉| 722/730 [1:04:06<00:42,  5.36s/it]
                                                   
{'train/ce_loss': 1.75, 'train/diffusion_loss': 0.3878677785396576, 'epoch': 9.89}

 99%|█████████▉| 722/730 [1:04:06<00:42,  5.36s/it]
                                                   
{'train/learning_rate_real': 7.874946562580477e-09, 'epoch': 9.89}

 99%|█████████▉| 722/730 [1:04:06<00:42,  5.36s/it]
 99%|█████████▉| 723/730 [1:04:06<00:38,  5.45s/it]
                                                   
{'debug/num_tok_total': 2405.0, 'debug/num_tok_loss': 1760.0, 'debug/num_lat_total': 2405.0, 'debug/num_lat_loss': 1760.0, 'epoch': 9.9}

 99%|█████████▉| 723/730 [1:04:07<00:38,  5.45s/it]
                                                   
{'train/ce_loss': 1.8046875, 'train/diffusion_loss': 0.42116475105285645, 'epoch': 9.9}

 99%|█████████▉| 723/730 [1:04:07<00:38,  5.45s/it]
                                                   
{'train/learning_rate_real': 6.029404357349444e-09, 'epoch': 9.9}

 99%|█████████▉| 723/730 [1:04:07<00:38,  5.45s/it]
                                                   
{'debug/num_tok_total': 3057.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 3057.0, 'debug/num_lat_loss': 1781.0, 'epoch': 9.9}

 99%|█████████▉| 723/730 [1:04:09<00:38,  5.45s/it]
                                                   
{'train/ce_loss': 1.828125, 'train/diffusion_loss': 0.3251039385795593, 'epoch': 9.9}

 99%|█████████▉| 723/730 [1:04:09<00:38,  5.45s/it]
                                                   
{'train/learning_rate_real': 6.029404357349444e-09, 'epoch': 9.9}

 99%|█████████▉| 723/730 [1:04:09<00:38,  5.45s/it]
                                                   
{'debug/num_tok_total': 1821.0, 'debug/num_tok_loss': 1609.0, 'debug/num_lat_total': 1821.0, 'debug/num_lat_loss': 1609.0, 'epoch': 9.9}

 99%|█████████▉| 723/730 [1:04:10<00:38,  5.45s/it]
                                                   
{'train/ce_loss': 1.734375, 'train/diffusion_loss': 0.5281632542610168, 'epoch': 9.9}

 99%|█████████▉| 723/730 [1:04:10<00:38,  5.45s/it]
                                                   
{'train/learning_rate_real': 6.029404357349444e-09, 'epoch': 9.9}

 99%|█████████▉| 723/730 [1:04:10<00:38,  5.45s/it]
                                                   
{'debug/num_tok_total': 2425.0, 'debug/num_tok_loss': 1785.0, 'debug/num_lat_total': 2425.0, 'debug/num_lat_loss': 1785.0, 'epoch': 9.9}

 99%|█████████▉| 723/730 [1:04:11<00:38,  5.45s/it]
                                                   
{'train/ce_loss': 1.7265625, 'train/diffusion_loss': 0.4651615023612976, 'epoch': 9.9}

 99%|█████████▉| 723/730 [1:04:11<00:38,  5.45s/it]
                                                   
{'train/learning_rate_real': 6.029404357349444e-09, 'epoch': 9.9}

 99%|█████████▉| 723/730 [1:04:11<00:38,  5.45s/it]
 99%|█████████▉| 724/730 [1:04:11<00:32,  5.38s/it]
                                                   
{'debug/num_tok_total': 2238.0, 'debug/num_tok_loss': 1809.0, 'debug/num_lat_total': 2238.0, 'debug/num_lat_loss': 1809.0, 'epoch': 9.92}

 99%|█████████▉| 724/730 [1:04:13<00:32,  5.38s/it]
                                                   
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.49770259857177734, 'epoch': 9.92}

 99%|█████████▉| 724/730 [1:04:13<00:32,  5.38s/it]
                                                   
{'train/learning_rate_real': 4.429860956842169e-09, 'epoch': 9.92}

 99%|█████████▉| 724/730 [1:04:13<00:32,  5.38s/it]
                                                   
{'debug/num_tok_total': 2196.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2196.0, 'debug/num_lat_loss': 1781.0, 'epoch': 9.92}

 99%|█████████▉| 724/730 [1:04:14<00:32,  5.38s/it]
                                                   
{'train/ce_loss': 1.59375, 'train/diffusion_loss': 0.4769535958766937, 'epoch': 9.92}

 99%|█████████▉| 724/730 [1:04:14<00:32,  5.38s/it]
                                                   
{'train/learning_rate_real': 4.429860956842169e-09, 'epoch': 9.92}

 99%|█████████▉| 724/730 [1:04:14<00:32,  5.38s/it]
                                                   
{'debug/num_tok_total': 2622.0, 'debug/num_tok_loss': 1767.0, 'debug/num_lat_total': 2622.0, 'debug/num_lat_loss': 1767.0, 'epoch': 9.92}

 99%|█████████▉| 724/730 [1:04:15<00:32,  5.38s/it]
                                                   
{'train/ce_loss': 1.5703125, 'train/diffusion_loss': 0.4050956666469574, 'epoch': 9.92}

 99%|█████████▉| 724/730 [1:04:15<00:32,  5.38s/it]
                                                   
{'train/learning_rate_real': 4.429860956842169e-09, 'epoch': 9.92}

 99%|█████████▉| 724/730 [1:04:15<00:32,  5.38s/it]
                                                   
{'debug/num_tok_total': 2231.0, 'debug/num_tok_loss': 1794.0, 'debug/num_lat_total': 2231.0, 'debug/num_lat_loss': 1794.0, 'epoch': 9.92}

 99%|█████████▉| 724/730 [1:04:16<00:32,  5.38s/it]
                                                   
{'train/ce_loss': 1.6171875, 'train/diffusion_loss': 0.500572681427002, 'epoch': 9.92}

 99%|█████████▉| 724/730 [1:04:16<00:32,  5.38s/it]
                                                   
{'train/learning_rate_real': 4.429860956842169e-09, 'epoch': 9.92}

 99%|█████████▉| 724/730 [1:04:16<00:32,  5.38s/it]
 99%|█████████▉| 725/730 [1:04:17<00:26,  5.29s/it]
                                                   
{'debug/num_tok_total': 2460.0, 'debug/num_tok_loss': 1591.0, 'debug/num_lat_total': 2460.0, 'debug/num_lat_loss': 1591.0, 'epoch': 9.93}

 99%|█████████▉| 725/730 [1:04:18<00:26,  5.29s/it]
                                                   
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.3625696301460266, 'epoch': 9.93}

 99%|█████████▉| 725/730 [1:04:18<00:26,  5.29s/it]
                                                   
{'train/learning_rate_real': 3.0763478551146254e-09, 'epoch': 9.93}

 99%|█████████▉| 725/730 [1:04:18<00:26,  5.29s/it]
                                                   
{'debug/num_tok_total': 2389.0, 'debug/num_tok_loss': 1760.0, 'debug/num_lat_total': 2389.0, 'debug/num_lat_loss': 1760.0, 'epoch': 9.93}

 99%|█████████▉| 725/730 [1:04:19<00:26,  5.29s/it]
                                                   
{'train/ce_loss': 1.7890625, 'train/diffusion_loss': 0.4412277042865753, 'epoch': 9.93}

 99%|█████████▉| 725/730 [1:04:19<00:26,  5.29s/it]
                                                   
{'train/learning_rate_real': 3.0763478551146254e-09, 'epoch': 9.93}

 99%|█████████▉| 725/730 [1:04:19<00:26,  5.29s/it]
                                                   
{'debug/num_tok_total': 2214.0, 'debug/num_tok_loss': 1498.0, 'debug/num_lat_total': 2214.0, 'debug/num_lat_loss': 1498.0, 'epoch': 9.93}

 99%|█████████▉| 725/730 [1:04:20<00:26,  5.29s/it]
                                                   
{'train/ce_loss': 1.984375, 'train/diffusion_loss': 0.40508732199668884, 'epoch': 9.93}

 99%|█████████▉| 725/730 [1:04:20<00:26,  5.29s/it]
                                                   
{'train/learning_rate_real': 3.0763478551146254e-09, 'epoch': 9.93}

 99%|█████████▉| 725/730 [1:04:20<00:26,  5.29s/it]
                                                   
{'debug/num_tok_total': 2451.0, 'debug/num_tok_loss': 1599.0, 'debug/num_lat_total': 2451.0, 'debug/num_lat_loss': 1599.0, 'epoch': 9.93}

 99%|█████████▉| 725/730 [1:04:21<00:26,  5.29s/it]
                                                   
{'train/ce_loss': 1.625, 'train/diffusion_loss': 0.3627317249774933, 'epoch': 9.93}

 99%|█████████▉| 725/730 [1:04:21<00:26,  5.29s/it]
                                                   
{'train/learning_rate_real': 3.0763478551146254e-09, 'epoch': 9.93}

 99%|█████████▉| 725/730 [1:04:21<00:26,  5.29s/it]
 99%|█████████▉| 726/730 [1:04:22<00:21,  5.27s/it]
                                                   
{'debug/num_tok_total': 2428.0, 'debug/num_tok_loss': 1779.0, 'debug/num_lat_total': 2428.0, 'debug/num_lat_loss': 1779.0, 'epoch': 9.95}

 99%|█████████▉| 726/730 [1:04:23<00:21,  5.27s/it]
                                                   
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.43506351113319397, 'epoch': 9.95}

 99%|█████████▉| 726/730 [1:04:23<00:21,  5.27s/it]
                                                   
{'train/learning_rate_real': 1.9688917020269092e-09, 'epoch': 9.95}

 99%|█████████▉| 726/730 [1:04:23<00:21,  5.27s/it]
                                                   
{'debug/num_tok_total': 2665.0, 'debug/num_tok_loss': 1796.0, 'debug/num_lat_total': 2665.0, 'debug/num_lat_loss': 1796.0, 'epoch': 9.95}

 99%|█████████▉| 726/730 [1:04:24<00:21,  5.27s/it]
                                                   
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.42050695419311523, 'epoch': 9.95}

 99%|█████████▉| 726/730 [1:04:24<00:21,  5.27s/it]
                                                   
{'train/learning_rate_real': 1.9688917020269092e-09, 'epoch': 9.95}

 99%|█████████▉| 726/730 [1:04:24<00:21,  5.27s/it]
                                                   
{'debug/num_tok_total': 2841.0, 'debug/num_tok_loss': 1775.0, 'debug/num_lat_total': 2841.0, 'debug/num_lat_loss': 1775.0, 'epoch': 9.95}

 99%|█████████▉| 726/730 [1:04:25<00:21,  5.27s/it]
                                                   
{'train/ce_loss': 1.78125, 'train/diffusion_loss': 0.3617662191390991, 'epoch': 9.95}

 99%|█████████▉| 726/730 [1:04:25<00:21,  5.27s/it]
                                                   
{'train/learning_rate_real': 1.9688917020269092e-09, 'epoch': 9.95}

 99%|█████████▉| 726/730 [1:04:25<00:21,  5.27s/it]
                                                   
{'debug/num_tok_total': 2427.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2427.0, 'debug/num_lat_loss': 1781.0, 'epoch': 9.95}

 99%|█████████▉| 726/730 [1:04:27<00:21,  5.27s/it]
                                                   
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.43363091349601746, 'epoch': 9.95}

 99%|█████████▉| 726/730 [1:04:27<00:21,  5.27s/it]
                                                   
{'train/learning_rate_real': 1.9688917020269092e-09, 'epoch': 9.95}

 99%|█████████▉| 726/730 [1:04:27<00:21,  5.27s/it]
100%|█████████▉| 727/730 [1:04:27<00:15,  5.33s/it]
                                                   
{'debug/num_tok_total': 2641.0, 'debug/num_tok_loss': 1788.0, 'debug/num_lat_total': 2641.0, 'debug/num_lat_loss': 1788.0, 'epoch': 9.96}

100%|█████████▉| 727/730 [1:04:28<00:15,  5.33s/it]
                                                   
{'train/ce_loss': 1.7578125, 'train/diffusion_loss': 0.4582788646221161, 'epoch': 9.96}

100%|█████████▉| 727/730 [1:04:28<00:15,  5.33s/it]
                                                   
{'train/learning_rate_real': 1.1075143027283741e-09, 'epoch': 9.96}

100%|█████████▉| 727/730 [1:04:28<00:15,  5.33s/it]
                                                   
{'debug/num_tok_total': 2253.0, 'debug/num_tok_loss': 1808.0, 'debug/num_lat_total': 2253.0, 'debug/num_lat_loss': 1808.0, 'epoch': 9.96}

100%|█████████▉| 727/730 [1:04:30<00:15,  5.33s/it]
                                                   
{'train/ce_loss': 1.90625, 'train/diffusion_loss': 0.505445659160614, 'epoch': 9.96}

100%|█████████▉| 727/730 [1:04:30<00:15,  5.33s/it]
                                                   
{'train/learning_rate_real': 1.1075143027283741e-09, 'epoch': 9.96}

100%|█████████▉| 727/730 [1:04:30<00:15,  5.33s/it]
                                                   
{'debug/num_tok_total': 2221.0, 'debug/num_tok_loss': 1790.0, 'debug/num_lat_total': 2221.0, 'debug/num_lat_loss': 1790.0, 'epoch': 9.96}

100%|█████████▉| 727/730 [1:04:31<00:15,  5.33s/it]
                                                   
{'train/ce_loss': 1.6953125, 'train/diffusion_loss': 0.4921615719795227, 'epoch': 9.96}

100%|█████████▉| 727/730 [1:04:31<00:15,  5.33s/it]
                                                   
{'train/learning_rate_real': 1.1075143027283741e-09, 'epoch': 9.96}

100%|█████████▉| 727/730 [1:04:31<00:15,  5.33s/it]
                                                   
{'debug/num_tok_total': 2385.0, 'debug/num_tok_loss': 1763.0, 'debug/num_lat_total': 2385.0, 'debug/num_lat_loss': 1763.0, 'epoch': 9.96}

100%|█████████▉| 727/730 [1:04:32<00:15,  5.33s/it]
                                                   
{'train/ce_loss': 1.703125, 'train/diffusion_loss': 0.46757790446281433, 'epoch': 9.96}

100%|█████████▉| 727/730 [1:04:32<00:15,  5.33s/it]
                                                   
{'train/learning_rate_real': 1.1075143027283741e-09, 'epoch': 9.96}

100%|█████████▉| 727/730 [1:04:32<00:15,  5.33s/it]
100%|█████████▉| 728/730 [1:04:32<00:10,  5.29s/it]
                                                   
{'debug/num_tok_total': 2837.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2837.0, 'debug/num_lat_loss': 1777.0, 'epoch': 9.97}

100%|█████████▉| 728/730 [1:04:34<00:10,  5.29s/it]
                                                   
{'train/ce_loss': 1.765625, 'train/diffusion_loss': 0.3639686107635498, 'epoch': 9.97}

100%|█████████▉| 728/730 [1:04:34<00:10,  5.29s/it]
                                                   
{'train/learning_rate_real': 4.922326172246416e-10, 'epoch': 9.97}

100%|█████████▉| 728/730 [1:04:34<00:10,  5.29s/it]
                                                   
{'debug/num_tok_total': 2835.0, 'debug/num_tok_loss': 1774.0, 'debug/num_lat_total': 2835.0, 'debug/num_lat_loss': 1774.0, 'epoch': 9.97}

100%|█████████▉| 728/730 [1:04:35<00:10,  5.29s/it]
                                                   
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.3863534927368164, 'epoch': 9.97}

100%|█████████▉| 728/730 [1:04:35<00:10,  5.29s/it]
                                                   
{'train/learning_rate_real': 4.922326172246416e-10, 'epoch': 9.97}

100%|█████████▉| 728/730 [1:04:35<00:10,  5.29s/it]
                                                   
{'debug/num_tok_total': 2916.0, 'debug/num_tok_loss': 1810.0, 'debug/num_lat_total': 2916.0, 'debug/num_lat_loss': 1810.0, 'epoch': 9.97}

100%|█████████▉| 728/730 [1:04:36<00:10,  5.29s/it]
                                                   
{'train/ce_loss': 1.71875, 'train/diffusion_loss': 0.37913691997528076, 'epoch': 9.97}

100%|█████████▉| 728/730 [1:04:36<00:10,  5.29s/it]
                                                   
{'train/learning_rate_real': 4.922326172246416e-10, 'epoch': 9.97}

100%|█████████▉| 728/730 [1:04:36<00:10,  5.29s/it]
                                                   
{'debug/num_tok_total': 2642.0, 'debug/num_tok_loss': 1777.0, 'debug/num_lat_total': 2642.0, 'debug/num_lat_loss': 1777.0, 'epoch': 9.97}

100%|█████████▉| 728/730 [1:04:38<00:10,  5.29s/it]
                                                   
{'train/ce_loss': 1.8359375, 'train/diffusion_loss': 0.41801196336746216, 'epoch': 9.97}

100%|█████████▉| 728/730 [1:04:38<00:10,  5.29s/it]
                                                   
{'train/learning_rate_real': 4.922326172246416e-10, 'epoch': 9.97}

100%|█████████▉| 728/730 [1:04:38<00:10,  5.29s/it]
100%|█████████▉| 729/730 [1:04:38<00:05,  5.40s/it]
                                                   
{'debug/num_tok_total': 2415.0, 'debug/num_tok_loss': 1781.0, 'debug/num_lat_total': 2415.0, 'debug/num_lat_loss': 1781.0, 'epoch': 9.99}

100%|█████████▉| 729/730 [1:04:39<00:05,  5.40s/it]
                                                   
{'train/ce_loss': 1.609375, 'train/diffusion_loss': 0.43567800521850586, 'epoch': 9.99}

100%|█████████▉| 729/730 [1:04:39<00:05,  5.40s/it]
                                                   
{'train/learning_rate_real': 1.2305876004453654e-10, 'epoch': 9.99}

100%|█████████▉| 729/730 [1:04:39<00:05,  5.40s/it]
                                                   
{'debug/num_tok_total': 2452.0, 'debug/num_tok_loss': 1689.0, 'debug/num_lat_total': 2452.0, 'debug/num_lat_loss': 1689.0, 'epoch': 9.99}

100%|█████████▉| 729/730 [1:04:40<00:05,  5.40s/it]
                                                   
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.418081134557724, 'epoch': 9.99}

100%|█████████▉| 729/730 [1:04:40<00:05,  5.40s/it]
                                                   
{'train/learning_rate_real': 1.2305876004453654e-10, 'epoch': 9.99}

100%|█████████▉| 729/730 [1:04:40<00:05,  5.40s/it]
                                                   
{'debug/num_tok_total': 2883.0, 'debug/num_tok_loss': 1798.0, 'debug/num_lat_total': 2883.0, 'debug/num_lat_loss': 1798.0, 'epoch': 9.99}

100%|█████████▉| 729/730 [1:04:42<00:05,  5.40s/it]
                                                   
{'train/ce_loss': 2.015625, 'train/diffusion_loss': 0.35037553310394287, 'epoch': 9.99}

100%|█████████▉| 729/730 [1:04:42<00:05,  5.40s/it]
                                                   
{'train/learning_rate_real': 1.2305876004453654e-10, 'epoch': 9.99}

100%|█████████▉| 729/730 [1:04:42<00:05,  5.40s/it]
                                                   
{'debug/num_tok_total': 457.0, 'debug/num_tok_loss': 457.0, 'debug/num_lat_total': 457.0, 'debug/num_lat_loss': 457.0, 'epoch': 9.99}

100%|█████████▉| 729/730 [1:04:42<00:05,  5.40s/it]
                                                   
{'train/ce_loss': 1.7109375, 'train/diffusion_loss': 0.5445239543914795, 'epoch': 9.99}

100%|█████████▉| 729/730 [1:04:42<00:05,  5.40s/it]
                                                   
{'train/learning_rate_real': 1.2305876004453654e-10, 'epoch': 9.99}

100%|█████████▉| 729/730 [1:04:42<00:05,  5.40s/it]03/16/2026 07:52:14 - INFO - __main__ - LoRA debug step 730: changed A 157/196, changed B 193/196, lora_B_zero_now=0.

100%|██████████| 730/730 [1:04:42<00:00,  5.10s/it]
                                                   
{'loss': 2.6847, 'grad_norm': 1.6357812881469727, 'learning_rate': 1.2305876004453654e-10, 'epoch': 10.0}

100%|██████████| 730/730 [1:04:42<00:00,  5.10s/it]/home/ubuntu/.local/lib/python3.10/site-packages/peft/utils/save_and_load.py:295: UserWarning: Could not find a config file in  - will assume that the vocabulary was not modified.
  warnings.warn(

                                                   
{'train_runtime': 3884.7284, 'train_samples_per_second': 5.998, 'train_steps_per_second': 0.188, 'train_loss': 2.784748409218984, 'epoch': 10.0}

100%|██████████| 730/730 [1:04:44<00:00,  5.10s/it]
100%|██████████| 730/730 [1:04:44<00:00,  5.32s/it]
/home/ubuntu/.local/lib/python3.10/site-packages/peft/utils/save_and_load.py:295: UserWarning: Could not find a config file in  - will assume that the vocabulary was not modified.
  warnings.warn(