Output:
Chunk ID: fb810e
Wall time: 0.0518 seconds
Process exited with code 0
Original token count: 5054
Output:
     1	# Upsampling neural codec output from 16kHz to 48kHz: a complete technical guide
     2
     3	**A frozen 16kHz encoder can absolutely drive a decoder producing 24–48kHz output** — this has been directly demonstrated by a 2025 modification to X-Codec 2.0 that achieved +0.29 MOS improvement over the 16kHz baseline with only decoder-side changes. The most practical path forward combines a modified ISTFT decoder head with GAN-based training, achievable within your existing Vocos architecture. Four concrete strategies are viable, ranked by implementation complexity and expected quality: (1) direct ISTFTHead rescaling for 24kHz, (2) multi-band ISTFT with PQMF synthesis for 48kHz, (3) a modular post-hoc bandwidth extension network like AP-BWE, or (4) single-step flow matching for maximum quality. All four meet the RTF ≤ 0.1 constraint.
     4
     5	---
     6
     7	## The information-theoretic argument for why this works
     8
     9	The central question — whether a decoder can synthesize frequencies above 8kHz from features that only encode up to 8kHz (the Nyquist of 16kHz audio) — has a definitive empirical answer: **yes**. Speech is a quasi-periodic signal where higher harmonics are strongly correlated with fundamentals, formant structure extends predictably across the full spectrum, and fricative/noise patterns correlate with energy envelopes in lower bands. The decoder learns statistical priors about high-frequency content during training on wideband audio.
    10
    11	Three independent systems confirm this. The X-Codec 2.0 modification (arXiv 2601.20185) froze the 16kHz encoder and HuBERT semantic encoder, modified only the decoder's STFT parameters (hop 320→960, output 16→24kHz), interpolated output projection weights, and fine-tuned. LinaCodec takes this further: it encodes at 24kHz (12kHz Nyquist) but decodes to **48kHz** using a dual-path Vocos decoder with Snake-based upsampling from BigVGAN, generating a full 12kHz of "hallucinated" bandwidth. NVSR demonstrated that even simple replication-padding of mel bins lets a vocoder generate meaningful high-frequency energy, because "prior knowledge in the vocoder maps constant energy in padded higher-frequency bands into meaningful energy distribution."
    12
    13	The practical limit appears to be roughly **3× the encoder's Nyquist frequency** — so from your 8kHz Nyquist, generating content up to ~24kHz (48kHz sample rate) is feasible but increasingly relies on learned priors rather than encoded information. Quality degrades gracefully: 24kHz output (extending to 12kHz) is easier and higher-fidelity than 48kHz output (extending to 24kHz).
    14
    15	---
    16
    17	## Strategy 1: Direct ISTFTHead rescaling — the simplest path to 24kHz
    18
    19	This is the most straightforward approach and has direct empirical validation. Your current architecture uses n_fft=1280, hop=320 at 16kHz, producing 641 frequency bins at 50 fps. To maintain the same 50 fps frame rate at higher sample rates while only modifying the decoder head:
    20
    21	| Target SR | Required hop | Suggested n_fft | Frequency bins | Output channels (mag+phase) |
    22	|-----------|-------------|-----------------|----------------|---------------------------|
    23	| 16 kHz | 320 | 1280 | 641 | 1,282 |
    24	| 24 kHz | 480 | 1920 | 961 | 1,922 |
    25	| 32 kHz | 640 | 2560 | 1,281 | 2,562 |
    26	| 48 kHz | 960 | 3840 | 1,921 | 3,842 |
    27
    28	For FFT efficiency, use n_fft=2048 (for 24kHz) or n_fft=4096 (for 48kHz) with slight frequency oversampling. The VocosBackbone remains unchanged — it still processes features at 50 fps. Only the final projection layer grows: from `backbone_dim × 1282` to `backbone_dim × 3842` parameters for 48kHz.
    29
    30	**Weight initialization via linear interpolation** (proven in the X-Codec 2.0 modification): for each new output index `i` in the larger projection, compute `x_i = (L-1)/(L'-1) × i` and interpolate `w'_i = (1 - α_i) × w_{⌊x_i⌋} + α_i × w_{⌈x_i⌉}`. This provides a meaningful starting point that preserves low-frequency reconstruction quality while the model learns high-frequency generation during fine-tuning.
    31
    32	**The key practical concern** is whether the 12-layer transformer backbone has sufficient capacity to predict 3× more spectral coefficients at 48kHz. For 24kHz (1.5× increase), existing backbone capacity is likely sufficient. For 48kHz, consider adding an intermediate projection MLP between backbone and ISTFTHead (e.g., `backbone_dim → 2×backbone_dim → 3842`) rather than expanding the backbone itself. This keeps computational cost nearly identical during the backbone forward pass, adding only a lightweight projection.
    33
    34	**Implementation priority**: Start with 24kHz (lowest risk, directly validated). If quality is acceptable, attempt 48kHz with the expanded projection head. If 48kHz quality suffers, move to Strategy 2.
    35
    36	---
    37
    38	## Strategy 2: Multi-band ISTFT with PQMF synthesis for 48kHz
    39
    40	For 48kHz output, multi-band decomposition offers a more elegant solution than a single massive ISTFTHead. This approach is validated by MB-ISTFT-VITS, MS-Vocos, and DMNet.
    41
    42	**The 4-band configuration** splits the 48kHz target into four 12kHz sub-bands via Pseudo Quadrature Mirror Filter (PQMF) banks. Each sub-band ISTFT uses n_fft=960, hop=240 at 50 fps, requiring only 481 frequency bins per band. Total output channels: `4 × 2 × 481 = 3,848` — similar to the direct approach but with structured frequency decomposition that allows sub-band-specific training objectives.
    43
    44	**An elegant 3-band variant** uses sub-bands at 16kHz each (n_fft=1280, hop=320), meaning the existing ISTFTHead architecture is reused exactly for one band, and two additional heads with identical architecture handle the upper bands. PQMF synthesis combines `3 × 16kHz → 48kHz`. This minimizes architectural changes and lets you initialize one band's weights from your trained 16kHz model.
    45
    46	The multi-band approach has several advantages for high-frequency generation. Different bands can receive specialized discriminator attention — the high-frequency bands benefit from stochastic generation (noise-like fricatives), while low-frequency bands need precise harmonic modeling. Sub-band discriminators can target frequency-specific artifacts. The PQMF filters themselves are fixed (no learned parameters; order 63 is standard), and implementations are available in torchaudio and in the DAC/SNAC codebases.
    47
    48	**MS-Vocos** (from the streaming vocoder literature) validates this specifically for Vocos: it splits the output along the channel axis into 4 sub-spectrograms, applies iSTFT per sub-band with reduced frame parameters, and uses a trainable synthesis filter. Quality matches standard Vocos with improved streaming characteristics.
    49
    50	---
    51
    52	## Strategy 3: Modular post-hoc bandwidth extension
    53
    54	If architectural modifications to the decoder are undesirable, a separate bandwidth extension network after the 16kHz decoder output offers the most modular path. Three options stand out, ranked by quality-speed tradeoff:
    55
    56	**AP-BWE** (Lu et al., IEEE/ACM TASLP 2024) is the strongest candidate for real-time applications. It uses a dual-stream all-CNN architecture that predicts amplitude and phase spectra in parallel with mutual interaction between streams, using ConvNeXt blocks throughout. It achieves **292× real-time on RTX 4090 GPU and 18× real-time on CPU** — comfortably within the RTF ≤ 0.1 requirement even on CPU. It was the first method to directly extend high-frequency phase (not just magnitude), which is critical for avoiding metallic artifacts. Training uses multi-period discriminators at the waveform level plus multi-resolution amplitude and phase discriminators. Open-source at github.com/yxlu-0102/AP-BWE.
    57
    58	**HiFi-GAN+ BWE** (brentspell/hifi-gan-bwe on GitHub, MIT license) provides the simplest deployment path. It uses a ~1M parameter WaveNet-style residual stack with 16 dilated convolution layers. A single pre-trained model handles arbitrary input sample rates (8/16/24kHz → 48kHz). It operates as a waveform-to-waveform transform — the input is first resampled to 48kHz via bandlimited interpolation, then the network fills in high-frequency content. Available via pip install.
    59
    60	**MusicHiFi-BWE** (Adobe Research, 2024) uses a HiFi-GAN generator with skip connection from the upsampled input, achieving **1786× real-time on A100**. Its design specifically targets post-processing of generative model outputs, making it architecturally aligned with the codec use case. The skip connection from the bilinearly upsampled input is a key design choice — it lets the network focus on generating only the high-frequency residual rather than reconstructing the entire signal.
    61
    62	**Common failure modes of GAN-based BWE and how to avoid them**: Over-smoothing in the >12kHz range (mitigated by high-frequency-specific discriminators), phase incoherence causing metallic artifacts (mitigated by AP-BWE's dual-stream phase prediction or complex STFT discriminators), mode collapse to limited high-frequency patterns (mitigated by diverse training data and multi-period discriminators), and mismatch between training conditions and codec artifacts (mitigated by training on actual codec output rather than clean low-pass filtered audio).
    63
    64	---
    65
    66	## Strategy 4: Single-step generation models for maximum quality
    67
    68	For applications where slightly higher latency is acceptable (RTF ~0.07 rather than ~0.003), two 2025 methods push quality beyond what pure GANs achieve.
    69
    70	**FLowHigh** (ICASSP 2025, Resemble AI) is the first flow-matching model for audio super-resolution. It uses a transformer-based vector field estimator (2 layers, 16-head attention, 1024 embedding dims) operating on mel-spectrograms, with specially designed conditional probability paths enabling **single-step Euler ODE sampling**. It achieves state-of-the-art LSD and ViSQOL on VCTK across all input rates (8/12/16/24kHz → 48kHz). The catch: it requires a BigVGAN vocoder for mel→waveform conversion, adding latency. Open-source at github.com/resemble-ai/flowhigh.
    71
    72	**FlashSR** (ICASSP 2025, KAIST) distills AudioSR's 50-step latent diffusion model into a single-step student using three losses (distillation, adversarial, distribution-matching distillation). Only ~45M parameters are updated via LoRA on attention modules of a 258M total model. It introduces an "SR Vocoder" that conditions waveform generation on both the predicted mel-spectrogram and the low-resolution input waveform, eliminating the need for low-frequency replacement post-processing. Achieves **22× speedup over AudioSR** (0.36s for 5.12s audio on A6000) with competitive subjective quality. Handles 4–32kHz input → 48kHz versatilely.
    73
    74	**UniverSR** (2025) eliminates the vocoder bottleneck entirely by directly predicting complex STFT coefficients via flow matching and reconstructing via iSTFT — architecturally aligned with the Vocos philosophy.
    75
    76	---
    77
    78	## How production systems actually handle this problem
    79
    80	An important finding from surveying production TTS: **most systems avoid the BWE problem entirely** by training codecs at the target sample rate from the start. This is the dominant industry pattern, but it requires control over the encoder — which you don't have.
    81
    82	Cartesia Sonic outputs at 24kHz or 44.1kHz (configurable via API) using SSM-based architecture; internal upsampling details are proprietary. Sesame CSM uses Kyutai's Mimi codec at 24kHz natively with no separate BWE stage. F5-TTS generates mel-spectrograms decoded by Vocos at 24kHz. CosyVoice 2 uses a flow-matching model to generate 50Hz mel features decoded by HiFi-GAN at 22.05kHz. Parler-TTS uniquely uses DAC at 44.1kHz — the codec itself operates at the target rate.
    83
    84	**DAC at 44.1kHz** uses encoder strides [2,4,8,8] (total 512×) with a decoder dimension of **1536** (vs 64 for the encoder), demonstrating significant asymmetric capacity allocation. The decoder uses transposed convolutions with Snake activations. EnCodec's 48kHz model uses chunk-based non-causal processing and music-specific training. Both use fundamentally the same conv encoder → RVQ → conv decoder architecture as their lower-rate counterparts — the difference is stride patterns and decoder capacity, not architectural paradigm.
    85
    86	**WavTokenizer** is particularly relevant: it operates at 24kHz with a single quantizer (like your FSQ) and uses a Vocos-inspired iFFT decoder structure with attention networks, achieving 40–75 tokens/second. Its design validates that Vocos-style decoders with attention can produce high-quality output from single-codebook representations.
    87
    88	---
    89
    90	## Training recipe for the frozen-encoder scenario
    91
    92	Based on patterns validated across DisCoder, MelCap, LFSC, HH-Codec, and the X-Codec 2.0 modification, here is a concrete training recipe:
    93
    94	**Data preparation**: Use HiFi-TTS2 (31.7k hours at 44.1kHz+, available on HuggingFace as nvidia/hifitts-2). Filter by the dataset's bandwidth estimation metadata for recordings with genuine high-frequency content (>20kHz estimated bandwidth). Even **100–500 hours of filtered high-bandwidth data is sufficient** — AP-BWE achieved SOTA with just 44 hours of VCTK, and SNAC trained on 2,730 hours. Create training pairs by encoding 48kHz audio through the frozen encoder+FSQ at 16kHz, then training the modified decoder to reconstruct the original 48kHz target.
    95
    96	**Loss function recipe** (ordered by importance):
    97
    98	- Multi-resolution mel-spectrogram L1 loss across 5–7 mel-bin counts (32, 64, 128, 256, 512) and multiple STFT configurations — this is the primary reconstruction signal
    99	- Multi-resolution STFT loss (spectral convergence + log magnitude) at configurations like (2048,512), (1024,256), (512,128), (256,64)
   100	- GAN adversarial loss (least-squares formulation) from all discriminators
   101	- Feature matching loss (L1 on intermediate discriminator features)
   102	- If operating in spectral domain: amplitude spectrum MSE + anti-wrapping phase losses (instantaneous phase error, group delay continuity, instantaneous frequency continuity)
   103
   104	**Discriminator configuration**: Use MPD with periods [2,3,5,7,11] for harmonic structure, complex multi-scale STFT discriminator (from DAC/SNAC) for high-frequency and phase fidelity, and optionally multi-resolution amplitude/phase discriminators from AP-BWE if predicting spectral coefficients. The `auraloss` library provides ready-to-use multi-resolution STFT loss implementations.
   105
   106	**Training schedule** (three phases):
   107
   108	1. **Warmup (0–20k steps)**: Reconstruction losses only (no GAN). Learning rate 2×10⁻⁴ for decoder with AdamW. This stabilizes the modified output dimensions before adversarial training.
   109	2. **GAN activation (20k–100k steps)**: Gradually introduce discriminator losses. Use separate Adam optimizer for discriminators (lr=2×10⁻⁴). EnCodec's gradient balancer mechanism (where each loss weight defines its fraction of the overall gradient) helps with loss balancing.
   110	3. **Full training (100k–400k steps)**: All losses active. Exponential LR decay (~0.999996 per step). SNAC demonstrates stable training without gradient clipping when using depthwise convolutions.
   111
   112	**Fine-tuning vs. training from scratch**: Fine-tune from the existing 16kHz Vocos decoder weights. The lower transformer layers encode semantic and structural information that transfers directly. Use a lower learning rate (1×10⁻⁵) for pre-trained backbone layers and higher (1×10⁻⁴) for the new/modified ISTFTHead projection. DisCoder, MelCap, and the X-Codec 2.0 modification all validate this staged approach of freezing encoder+VQ and fine-tuning the decoder.
   113
   114	---
   115
   116	## Prioritized implementation roadmap
   117
   118	Based on likelihood of success, implementation complexity, and evidence strength, here is the recommended order of attempts:
   119
   120	**Tier 1 — Try first (highest confidence, lowest risk)**
   121
   122	1. **Direct ISTFTHead rescaling to 24kHz** (n_fft=2048, hop=480). Directly validated by X-Codec 2.0 modification paper. Initialize via weight interpolation. Fine-tune 100–200k steps. Expected result: +0.29 MOS or better over 16kHz baseline. RTF ~0.001 on GPU.
   123
   124	2. **AP-BWE as post-hoc module** for 16kHz→48kHz. Pre-existing open-source code and methodology. Train on codec output → 48kHz target pairs. RTF ~0.003 on GPU, ~0.055 on CPU. Completely modular — no changes to existing codec.
   125
   126	**Tier 2 — Try if Tier 1 quality is insufficient**
   127
   128	3. **Multi-band ISTFTHead (4 bands) for 48kHz**. Each sub-band at 12kHz with PQMF synthesis. Reuse existing 16kHz head architecture where possible. Enables sub-band-specific discriminators. More architectural work but better frequency decomposition for 48kHz.
   129
   130	4. **MusicHiFi-BWE-style separate upsampler**. HiFi-GAN generator with 3× upsampling + skip connection from input. ~1786× real-time. Simple to implement and proven for generative model post-processing.
   131
   132	**Tier 3 — For maximum quality when latency budget allows**
   133
   134	5. **FLowHigh single-step flow matching** for mel-spectrogram super-resolution + Vocos/BigVGAN vocoder. SOTA metrics on benchmarks. RTF ~0.07 — tight but within budget.
   135
   136	6. **Consistency model distillation** of a diffusion-based audio SR model. No dedicated implementation exists yet, but CoMoSpeech demonstrates the approach achieves RTF ~0.007 for TTS — the same distillation framework applies to super-resolution.
   137
   138	**Tier 4 — Experimental / longer-term**
   139
   140	7. **Controllable multi-resolution decoder** with sample-rate conditioning embedding (extending Vocos's existing bandwidth_id mechanism). Train a single decoder that outputs 16/24/32/48kHz based on a conditioning signal. Requires more training data diversity but offers maximum deployment flexibility.
   141
   142	8. **VQ-based upsampling** where a small transformer predicts 48kHz codec tokens from 16kHz codec tokens. Interesting but autoregressive prediction creates a latency bottleneck unless non-autoregressive methods (masked prediction) are used.
   143
   144	---
   145
   146	## Conclusion
   147
   148	The frozen-encoder constraint is not the obstacle it might appear. Multiple 2024–2025 systems demonstrate that decoder-only modifications can bridge 16kHz encoding to 24–48kHz output with high fidelity. **The most important architectural insight is that Vocos's ISTFT-based design makes sample-rate changes nearly trivial** — the backbone operates at a fixed frame rate regardless of target sample rate, and only the output projection and STFT parameters change. This is fundamentally easier than modifying transposed-convolution-based decoders (like EnCodec's SEANet) where stride patterns are deeply embedded in the architecture.
   149
   150	For immediate deployment, the combination of **direct ISTFTHead rescaling to 24kHz** (for the best quality-latency tradeoff) plus **AP-BWE as a modular 24→48kHz stage** (if 48kHz is required) offers the lowest implementation risk with proven results. For a unified 48kHz solution, the multi-band ISTFT approach with PQMF synthesis provides the cleanest architecture. The field is converging on the view that bandwidth extension is better handled inside the decoder via increased capacity than as a separate post-processing stage — but the modular approach remains a valid engineering choice when decoder modifications are costly or risky.


Output:
Chunk ID: 4bd633
Wall time: 0.0505 seconds
Process exited with code 0
Original token count: 4159
Output:
     1	You’re not doing something obviously dumb. The problem is the framing.
     2
     3	What you have is not really “upsampling” in the magical recover-the-lost-bits sense. Your current codec was trained to reconstruct 16 kHz audio, so anything above 8 kHz is gone and has to be *re-synthesized plausibly*, not recovered exactly. That is why blind post-hoc waveform upsamplers so often sound airy-but-fake, or just weirdly tunnel-y in a new costume. The better results in recent work come from moving the bandwidth extension closer to the codec decoder or even into the latent space, where the model still has token, speaker, prosody, and voicing context. There is even a January 2026 X-Codec-2.0 technical report that did almost exactly the kind of surgery you’re considering: froze the encoder, changed the hop size, fine-tuned only the decoder, and reported a +0.29 UTMOSv2 gain for 24 kHz output. Promising clue, not holy scripture, but definitely a signal. ([arXiv][1])
     4
     5	My blunt take: for your use case, the best path is **decoder-aware bandwidth extension**, not another blind waveform SR model hanging off the end like a decorative tax. If I had to rank the practical options for a voice-agent/TTS stack, it would be:
     6
     7	1. **24 kHz decoder-only retrain first**
     8	2. **Residual high-band restorer conditioned on codec latents**
     9	3. **A stronger non-diffusion decoder replacement with harmonic/phase priors**
    10	4. **Single-step latent flow or distilled diffusion only as an optional HQ tier**
    11
    12	### What has actually worked in the literature
    13
    14	Three families keep showing up when people stop making charts and start making things that sound good.
    15
    16	**First:** direct spectral decoders. APNet2, APCodec, and ComplexDec all lean into direct amplitude/phase or complex-spectrum modeling instead of long waveform upsampling stacks. APNet2 reports better speech quality than APNet and Vocos at 22.05 kHz, APCodec targets streamable 48 kHz audio with 6.67 ms fixed latency, and ComplexDec pushes full-band 48 kHz codec decoding with complex spectral I/O for better robustness. ([arXiv][2])
    17
    18	**Second:** harmonic or source-filter priors. This is a big one for speech. Comparative work on fast vocoders notes that Vocos is extremely fast, but complex-spectrogram prediction without a strong harmonic inductive bias can be less robust to F0 variation. HiFTNet adds a harmonic-plus-noise source/filter in the time-frequency domain and reports ground-truth-level subjective performance on LJSpeech while being much lighter than BigVGAN. Wavehax adds a harmonic prior plus 2D spectral modeling, and reports high-F0 robustness with less than 5% of HiFi-GAN V1’s MACs/params and over 4x faster CPU inference. ([arXiv][3])
    19
    20	**Third:** lightweight bandwidth-extension models that predict only the missing band. AP-BWE uses dual amplitude and phase streams with ConvNeXt backbones and reports SOTA speech BWE quality at both 16 kHz and 48 kHz while running very fast. MS-BWE extends that idea to flexible source/target sample rates with stage-wise blocks and teacher forcing, reporting one-stage 48 kHz generation at about 1271x real-time on a 4090 and about 59.7x real-time on CPU. BAE-Net targets unknown or varying effective bandwidths in streaming settings, and a 2024 DSP-informed paper shows that a very explicit exciter plus linear time-varying filter split can outperform plain black-box generators while using lighter exciters. ([arXiv][4])
    21
    22	### So what should *you* build?
    23
    24	#### 1) Start with a **24 kHz decoder-only retarget**, not 48 kHz
    25
    26	For speech TTS, 24 kHz is the highest-ROI first move. You only need to invent 8-12 kHz, instead of 8-24 kHz, and that is where a lot of the “tunnel” feeling disappears. Also, HiFiTTS-2 explicitly gives you large high-bandwidth training targets for this regime: about 36.7k hours for 22.05 kHz training and 31.7k hours for 44.1 kHz training. ([arXiv][5])
    27
    28	In your specific geometry, keep the **50 Hz token rate** and just retarget the decoder’s waveform geometry:
    29
    30	* 16 kHz: hop 320, win/n_fft 1280
    31	* 24 kHz: hop 480, win/n_fft about 1920
    32	* 32 kHz: hop 640, win/n_fft about 2560
    33	* 48 kHz: hop 960, win/n_fft about 3840
    34
    35	That preserves the same **20 ms frame rate** and **80 ms window duration** in real time, which is exactly what you want. The expensive part of your decoder is the 50 Hz transformer trunk, so if you keep the token rate fixed, most of the extra cost lands in the output head and iSTFT, not in sequence modeling. That is a very favorable trade for realtime deployment.
    36
    37	For the first retrain, I would **not** keep the current Vocos-style single linear mag/phase head unchanged and just hope vibes fix it. Recent work says the fast frame-level spectral approach is good, but speech quality improves when you add more phase structure or harmonic prior. So I would try one of these two replacements first:
    38
    39	* **APNet2 / APCodec-style head**: separate amplitude and phase streams after a shared trunk.
    40	* **Wavehax-lite / HiFTNet-lite head**: harmonic-prior branch plus spectral decoder.
    41
    42	That is where the current research points, and it fits your architecture well. ([arXiv][2])
    43
    44	#### 2) Your easiest production-friendly win is a **residual high-band decoder branch**
    45
    46	This is probably the best balance of quality, latency, and engineering sanity.
    47
    48	Pipeline:
    49
    50	1. Keep your current 16 kHz decoder exactly as the **base low-band path**.
    51	2. Deterministically upsample it to 24/32/48 kHz with a good sinc/polyphase resampler.
    52	3. Add a tiny **high-band residual model** that predicts only the missing band in STFT space.
    53	4. Condition that branch on:
    54
    55	   * `vq_post_emb` or decoder hidden states
    56	   * the upsampled base waveform or its STFT
    57	   * optional F0 / voiced-unvoiced / energy features
    58
    59	This is basically the “stop trying to rewrite what already works” strategy. And it is backed by multiple lines of work: SoundStream explicitly showed decoder-side enhancement with no extra latency, AP-BWE/MS-BWE predict missing-band spectra directly, and AudioLBM reports that low-frequency replacement improves both low-band fidelity and later high-frequency generation in cascaded SR systems. ([arXiv][6])
    60
    61	Concretely, I would make that HF branch predict **only**:
    62
    63	* residual log-magnitude for bins above 8 kHz
    64	* direct phase for those bins, or complex residual if you want a cleaner implementation
    65
    66	And I would hard-bypass or replace the low band from the base decoder. That one design choice alone usually saves a lot of misery. Humans love training a model to “enhance” audio, then act shocked when it mangles the perfectly fine part too.
    67
    68	#### 3) Add **harmonic / voicing structure** explicitly
    69
    70	Speech high-band is not just “more treble.” It is a mix of harmonics, breathiness, and noisy unvoiced consonants. The literature keeps converging on this point from different directions:
    71
    72	* HiFTNet: harmonic-plus-noise source/filter helps.
    73	* Wavehax: harmonic prior improves robustness, especially high F0.
    74	* CodecFlow: explicit voicing-aware conditioning helps recover high-frequency unvoiced detail in codec-latent BWE. ([arXiv][7])
    75
    76	So I would add a tiny auxiliary head that predicts **F0 + voiced/unvoiced + maybe band energy** from the frozen post-VQ embeddings, using teacher labels from a pitch extractor during training. Then use those features to condition the HF branch.
    77
    78	That gives you a very cheap but very meaningful inductive bias. It is also much cleaner than hoping the decoder silently learns that /s/, /sh/, breath, and female vowels need different high-band treatment.
    79
    80	### If you want a stronger decoder replacement, these are the best bets
    81
    82	**Best non-diffusion ceiling:** replace the current Vocos decoder with one of these ideas:
    83
    84	* **APCodec-style decoder**
    85	  Modified ConvNeXt-v2 backbone, amplitude/phase representation, streamable design, 6.67 ms fixed latency at 48 kHz. That is probably the closest precedent to “high-rate codec decoder that still cares about realtime.” ([arXiv][8])
    86
    87	* **Wavehax-lite**
    88	  If your issue is crispness and harmonic realism, this is very attractive. It is spectral-domain, aliasing-aware, harmonic-prior-driven, and fast. Recent streaming analysis also found that multi-stream Wavehax delivered top throughput under low-latency conditions under 80 ms, and got nearly non-causal quality with only one-frame lookahead. ([arXiv][9])
    89
    90	* **HiFTNet-lite**
    91	  If you want something closer to iSTFTNet but more speech-specialized, HiFTNet is a really good signal that source/filter inductive bias pays off. ([arXiv][7])
    92
    93	I would *not* bet my first serious experiment on plain Vocos-at-higher-sample-rate. It is fast, yes. But the comparative papers are basically telling you the thing you’re already hearing: speed alone is not enough if the model lacks harmonic bias. ([arXiv][3])
    94
    95	### Conditional 16 / 24 / 32 / 48 is absolutely possible
    96
    97	This is very doable, and I would prefer **additive band gating** over one giant “do everything” head.
    98
    99	A clean design is:
   100
   101	* **Base16** branch: reconstructs the existing 0-8 kHz band
   102	* **+HF24** branch: adds 8-12 kHz
   103	* **+HF32** branch: adds 12-16 kHz
   104	* **+HF48** branch: adds 16-24 kHz
   105
   106	Each branch is shallow, residual, and conditioned on a **target-rate embedding**. For 16 kHz mode, none of the HF branches run. For 24 kHz, only the first one runs. For 48 kHz, all of them run. That gives you controllability *and* compute that scales with target quality, which is exactly what you want in a production stack.
   107
   108	There is good precedent for both parts of this idea: MS-BWE does stage-wise extension across multiple source/target sample-rate pairs with teacher forcing, and AudioLBM explicitly conditions on source and target frequency information to learn any-to-any upsampling. ([arXiv][10])
   109
   110	If you want the simplest version, do a **shared trunk + rate-specific shallow heads**. If you want the most controllable version, do **cascaded band branches**. I’d pick the second one.
   111
   112	### Losses and training tricks that matter here
   113
   114	This part is boring, which means it matters.
   115
   116	I would not rely on plain full-band mel loss plus generic GAN losses and hope for sparkle. For missing-band reconstruction, that objective is too blunt. The models that work well here tend to use **band-aware spectral objectives** and often explicit **phase losses** or amplitude/phase discriminators. AP-BWE and MS-BWE both predict residual log-amplitude and direct phase, and use waveform, amplitude, and phase discriminators instead of only one undifferentiated adversarial bucket. ([arXiv][4])
   117
   118	My recommended training recipe:
   119
   120	* **Low-band identity loss or hard low-band replacement**
   121	* **High-band-weighted complex STFT loss**
   122	* **Amplitude loss + phase loss** if using dual-stream output
   123	* **A small waveform adversarial loss**
   124	* **One spectral discriminator focused on the new band**
   125	* **Optional F0 / UV auxiliary loss** if you add voicing conditioning
   126
   127	Also, if you do a cascaded setup, copy the **teacher forcing / scheduled sampling** idea from MS-BWE so stage 2 doesn’t collapse when stage 1 makes slightly ugly mistakes. That paper did the annoying but necessary work of showing why this helps. ([arXiv][10])
   128
   129	### How I would structure the experiments
   130
   131	This is the order I’d run, because research is just organized disappointment unless you control the ablations.
   132
   133	**E0.** Baseline: current 16 kHz decoder + best deterministic resampler to 24/48 kHz.
   134	This tells you how much improvement is real and how much is placebo from reconstruction filtering.
   135
   136	**E1.** 24 kHz decoder-only retarget.
   137	Same frozen encoder/quantizer, same cached codes, new 24 kHz head. Train only head + maybe last decoder blocks.
   138
   139	**E2.** 24 kHz dual-stream spectral head.
   140	Same as E1, but amplitude/phase split like APNet2/AP-BWE.
   141
   142	**E3.** 24 kHz residual HF branch.
   143	Keep current decoder as low-band base, predict only >8 kHz residual.
   144
   145	**E4.** 24 -> 48 cascaded stage.
   146	Second stage only predicts 12-24 kHz from tokens + 24 kHz output, with teacher forcing.
   147
   148	**E5.** Optional HQ tier.
   149	Single-step flow or distilled diffusion on latent/STFT residual only, not the whole waveform path.
   150
   151	That ladder gives you an honest answer about whether the real gain is coming from direct high-rate decoding or from a cheap HF restorer.
   152
   153	### Where diffusion fits, if you really want it
   154
   155	If you allow *some* generative overhead, don’t use classic multi-step waveform diffusion in your default path. That is the scenic route to latency regret.
   156
   157	The smarter versions right now are:
   158
   159	* **single-step distilled diffusion**, like FlashSR, which reports competitive 48 kHz SR while being about 22x faster than the previous SOTA it compares against;
   160	* **single-step flow matching**, like FLowHigh, which reports strong VCTK SR results with one-step sampling;
   161	* **latent-space probabilistic models**, which are much more aligned with your cached-token setup than waveform diffusion. Latent-domain upsampling work reports up to 100x efficiency gains over raw-audio post-processing, and the very fresh March 2026 CodecFlow paper specifically does voicing-aware bandwidth extension in codec latent space. ([arXiv][11])
   162
   163	There is also a very relevant codec-side precedent: ScoreDec adds a diffusion post-filter to AudioDec and reports human-level naturalness for full-band 48 kHz speech at 24 kbps. That tells you diffusion *can* close the gap, but in a voice-agent product I would keep it as a premium or offline quality mode, not the default call path. ([arXiv][12])
   164
   165	### My final recommendation
   166
   167	If this were my stack, I would do this, in exactly this order:
   168
   169	1. **Train a 24 kHz decoder-only variant first** using your cached codes and original high-rate waveforms.
   170	2. **Replace the current head with a dual amplitude/phase or harmonic-prior decoder**, not just a naïve bigger ISTFT head.
   171	3. **Add a residual high-band branch with low-band bypass**, conditioned on `vq_post_emb` and F0/UV.
   172	4. **Only then** build a cascaded 24 -> 48 stage if 24 kHz still isn’t enough.
   173	5. Keep **single-step latent flow/diffusion** as an optional HQ path, not the default.
   174
   175	That path has the best odds of giving you a real audible win without detonating latency. And it lines up with what has actually worked in fast speech vocoders, BWE systems, and recent codec research, instead of sending you back into another week of ritual suffering with blind upsamplers.
   176
   177	[1]: https://arxiv.org/pdf/2601.20185 "https://arxiv.org/pdf/2601.20185"
   178	[2]: https://arxiv.org/abs/2311.11545 "https://arxiv.org/abs/2311.11545"
   179	[3]: https://arxiv.org/pdf/2506.03554 "https://arxiv.org/pdf/2506.03554"
   180	[4]: https://arxiv.org/html/2401.06387v1 "https://arxiv.org/html/2401.06387v1"
   181	[5]: https://arxiv.org/abs/2506.04152 "https://arxiv.org/abs/2506.04152"
   182	[6]: https://arxiv.org/abs/2107.03312 "https://arxiv.org/abs/2107.03312"
   183	[7]: https://arxiv.org/abs/2309.09493 "https://arxiv.org/abs/2309.09493"
   184	[8]: https://arxiv.org/pdf/2402.10533 "https://arxiv.org/pdf/2402.10533"
   185	[9]: https://arxiv.org/abs/2411.06807 "https://arxiv.org/abs/2411.06807"
   186	[10]: https://arxiv.org/pdf/2406.02250 "https://arxiv.org/pdf/2406.02250"
   187	[11]: https://arxiv.org/abs/2501.10807 "https://arxiv.org/abs/2501.10807"
   188	[12]: https://arxiv.org/abs/2401.12160 "[2401.12160] ScoreDec: A Phase-preserving High-Fidelity Audio Codec with A Generalized Score-based Diffusion Post-filter"


# Plan Mode (Conversational)

You work in 3 phases, and you should *chat your way* to a great plan before finalizing it. A great plan is very detailed—intent- and implementation-wise—so that it can be handed to another engineer or agent to be implemented right away. It must be **decision complete**, where the implementer does not need to make any decisions.

## Mode rules (strict)

You are in **Plan Mode** until a developer message explicitly ends it.

Plan Mode is not changed by user intent, tone, or imperative language. If a user asks for execution while still in Plan Mode, treat it as a request to **plan the execution**, not perform it.

## Plan Mode vs update_plan tool

Plan Mode is a collaboration mode that can involve requesting user input and eventually issuing a `<proposed_plan>` block.

Separately, `update_plan` is a checklist/progress/TODOs tool; it does not enter or exit Plan Mode. Do not confuse it with Plan mode or try to use it while in Plan mode. If you try to use `update_plan` in Plan mode, it will return an error.

## Execution vs. mutation in Plan Mode

You may explore and execute **non-mutating** actions that improve the plan. You must not perform **mutating** actions.

### Allowed (non-mutating, plan-improving)

Actions that gather truth, reduce ambiguity, or validate feasibility without changing repo-tracked state. Examples:

* Reading or searching files, configs, schemas, types, manifests, and docs
* Static analysis, inspection, and repo exploration
* Dry-run style commands when they do not edit repo-tracked files
* Tests, builds, or checks that may write to caches or build artifacts (for example, `target/`, `.cache/`, or snapshots) so long as they do not edit repo-tracked files

### Not allowed (mutating, plan-executing)

Actions that implement the plan or change repo-tracked state. Examples:

* Editing or writing files
* Running formatters or linters that rewrite files
* Applying patches, migrations, or codegen that updates repo-tracked files
* Side-effectful commands whose purpose is to carry out the plan rather than refine it

When in doubt: if the action would reasonably be described as "doing the work" rather than "planning the work," do not do it.

## PHASE 1 — Ground in the environment (explore first, ask second)

Begin by grounding yourself in the actual environment. Eliminate unknowns in the prompt by discovering facts, not by asking the user. Resolve all questions that can be answered through exploration or inspection. Identify missing or ambiguous details only if they cannot be derived from the environment. Silent exploration between turns is allowed and encouraged.

Before asking the user any question, perform at least one targeted non-mutating exploration pass (for example: search relevant files, inspect likely entrypoints/configs, confirm current implementation shape), unless no local environment/repo is available.

Exception: you may ask clarifying questions about the user's prompt before exploring, ONLY if there are obvious ambiguities or contradictions in the prompt itself. However, if ambiguity might be resolved by exploring, always prefer exploring first.

Do not ask questions that can be answered from the repo or system (for example, "where is this struct?" or "which UI component should we use?" when exploration can make it clear). Only ask once you have exhausted reasonable non-mutating exploration.

## PHASE 2 — Intent chat (what they actually want)

* Keep asking until you can clearly state: goal + success criteria, audience, in/out of scope, constraints, current state, and the key preferences/tradeoffs.
* Bias toward questions over guessing: if any high-impact ambiguity remains, do NOT plan yet—ask.

## PHASE 3 — Implementation chat (what/how we’ll build)

* Once intent is stable, keep asking until the spec is decision complete: approach, interfaces (APIs/schemas/I/O), data flow, edge cases/failure modes, testing + acceptance criteria, rollout/monitoring, and any migrations/compat constraints.

## Asking questions

Critical rules:

* Strongly prefer using the `request_user_input` tool to ask any questions.
* Offer only meaningful multiple‑choice options; don’t include filler choices that are obviously wrong or irrelevant.
* In rare cases where an unavoidable, important question can’t be expressed with reasonable multiple‑choice options (due to extreme ambiguity), you may ask it directly without the tool.

You SHOULD ask many questions, but each question must:

* materially change the spec/plan, OR
* confirm/lock an assumption, OR
* choose between meaningful tradeoffs.
* not be answerable by non-mutating commands.

Use the `request_user_input` tool only for decisions that materially change the plan, for confirming important assumptions, or for information that cannot be discovered via non-mutating exploration.

## Two kinds of unknowns (treat differently)

1. **Discoverable facts** (repo/system truth): explore first.

   * Before asking, run targeted searches and check likely sources of truth (configs/manifests/entrypoints/schemas/types/constants).
   * Ask only if: multiple plausible candidates; nothing found but you need a missing identifier/context; or ambiguity is actually product intent.
   * If asking, present concrete candidates (paths/service names) + recommend one.
   * Never ask questions you can answer from your environment (e.g., “where is this struct”).

2. **Preferences/tradeoffs** (not discoverable): ask early.

   * These are intent or implementation preferences that cannot be derived from exploration.
   * Provide 2–4 mutually exclusive options + a recommended default.
   * If unanswered, proceed with the recommended option and record it as an assumption in the final plan.

## Finalization rule

Only output the final plan when it is decision complete and leaves no decisions to the implementer.

When you present the official plan, wrap it in a `<proposed_plan>` block so the client can render it specially:

1) The opening tag must be on its own line.
2) Start the plan content on the next line (no text on the same line as the tag).
3) The closing tag must be on its own line.
4) Use Markdown inside the block.
5) Keep the tags exactly as `<proposed_plan>` and `</proposed_plan>` (do not translate or rename them), even if the plan content is in another language.

Example:

<proposed_plan>
plan content
</proposed_plan>

plan content should be human and agent digestible. The final plan must be plan-only and include:

* A clear title
* A brief summary section
* Important changes or additions to public APIs/interfaces/types
* Test cases and scenarios
* Explicit assumptions and defaults chosen where needed

Do not ask "should I proceed?" in the final output. The user can easily switch out of Plan mode and request implementation if you have included a `<proposed_plan>` block in your response. Alternatively, they can decide to stay in Plan mode and continue refining the plan.

Only produce at most one `<proposed_plan>` block per turn, and only when you are presenting a complete spec.

If the user stays in Plan mode and asks for revisions after a prior `<proposed_plan>`, any new `<proposed_plan>` must be a complete replacement.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


I now have Claude and GPT_PRO fully extracted. I’ll do the same for `perplexity.md`, then consolidate into the ID inventory and conflict-resolution logic.


Output:
Chunk ID: 30dbf1
Wall time: 0.0519 seconds
Process exited with code 0
Original token count: 4587
Output:
     1	# Upsampling Strategies for X-Codec 2.0: From 16kHz to High-Fidelity Audio
     2
     3	## Executive Summary
     4
     5	X-Codec 2.0's 16kHz output (8kHz Nyquist) produces a noticeable "muffled" quality because it lacks the 8–24kHz frequency content that gives speech its brightness, air, and presence. This report systematically evaluates every viable approach to upgrading the output resolution—from minimal decoder-only modifications to full bandwidth extension post-processors—prioritizing methods that preserve low-latency inference for voice agent and TTS deployment. A recent paper has already demonstrated that X-Codec 2.0 can be upgraded to 24kHz output by only retraining the decoder with frozen encoders, validating the core strategy of decoder-side modification. Beyond that baseline, this report covers GAN-based bandwidth extension, flow matching post-filters, ISTFT head redesign, multi-band synthesis, and controllable multi-rate output architectures.[1][2]
     6
     7	## Strategy 1: Decoder-Only Retraining with Modified ISTFT Parameters
     8
     9	### The Proven Baseline: X-Codec 2.0 → 24kHz
    10
    11	A January 2026 paper titled "Improving X-Codec-2.0 for Multi-Lingual Speech" directly addresses this exact problem. The approach is strikingly simple:[2]
    12
    13	- **Freeze all encoder components** (CodecEncoder, Wav2Vec2-BERT, SemanticEncoder) and the quantizer
    14	- **Increase the decoder hop size** from 320 to 960 samples
    15	- **Add an AvgPool1d(kernel=2, stride=2)** before vector quantization to reduce the latent rate from 50Hz to 25Hz
    16	- **Retrain only the decoder** (VocosBackbone + ISTFTHead) to output 24kHz audio
    17	- **Interpolate decoder weights** from the pretrained checkpoint using 1D linear interpolation on the output projection
    18
    19	This achieved a **+0.29 MOS improvement** on UTMOSv2 over the original 16kHz X-Codec 2.0, evaluated across 116 languages on Common Voice 17. Training was done on 2× RTX 3090 Ti GPUs for 3M steps with batch size 20/device, reusing the original loss formulation (\(\lambda_{\text{mel}}=15, \lambda_{\text{adv}}=1, \lambda_{\text{sem}}=5\)).[1][2]
    20
    21	### Extending to 48kHz: Direct ISTFT Head Modification
    22
    23	Since your decoder already uses a Vocos-style ISTFT head, extending to 48kHz follows the same architectural principle with parameter scaling:
    24
    25	| Parameter | 16kHz (Current) | 24kHz (Proven) | 48kHz (Proposed) |
    26	|-----------|-----------------|----------------|------------------|
    27	| `sample_rate` | 16,000 | 24,000 | 48,000 |
    28	| `hop_length` | 320 | 960 | 960 |
    29	| `n_fft` | 1280 | ~1920 | 3840 |
    30	| `ISTFT output dim` | 1282 | ~1922 | 3842 |
    31	| Freq bins | 641 | ~961 | 1921 |
    32	| Freq resolution | 12.5 Hz | 12.5 Hz | 12.5 Hz |
    33	| Latent rate | 50 Hz | 25 Hz | 50 Hz |
    34	| Nyquist | 8 kHz | 12 kHz | 24 kHz |
    35
    36	The key insight from the Vocos architecture is that the ISTFT-based decoder operates entirely at the compressed frame rate—there are no upsampling convolutions—so increasing the sample rate primarily means the ISTFTHead must output more frequency bins. The VocosBackbone transformer still operates at the same temporal resolution; only its final linear projection and the ISTFT parameters change.[3]
    37
    38	**Practical implementation:** Modify the ISTFTHead's final `Linear(1024 → n_fft+2)` to `Linear(1024 → 3842)`, update `n_fft=3840, hop_length=960, win_length=3840`, and retrain. You may need to increase the VocosBackbone's hidden dimension (from 1024 to 1536 or 2048) or add transformer layers to provide enough capacity for the wider frequency range, since each frame now must predict 1921 magnitude + 1921 phase bins instead of 641+641.
    39
    40	**Important caveat:** Experiments by other researchers modifying Vocos `n_fft` and `hop_length` parameters have sometimes yielded poor results, particularly with aggressive compression ratios. The Vocos architecture was specifically designed with the relationship between its hidden dimension and STFT parameters carefully tuned. You should expect to need careful hyperparameter exploration and likely a wider backbone to make 48kHz work.[4]
    41
    42	## Strategy 2: GAN-Based Bandwidth Extension Post-Filter
    43
    44	### AP-BWE: The Speed Champion
    45
    46	AP-BWE (Amplitude and Phase Bandwidth Extension) is a pure CNN-based GAN that directly extends narrowband spectra to wideband. It is the most promising non-diffusion post-processor for your use case:[5]
    47
    48	- **Architecture:** Dual-stream ConvNeXt backbone with parallel amplitude and phase prediction streams plus inter-stream connections
    49	- **Speed:** 292.3× faster than real-time on a single RTX 4090 GPU, and 18.1× faster on a single CPU core for 16kHz→48kHz[5]
    50	- **Quality:** State-of-the-art on both 16kHz and 48kHz target rates
    51	- **Waveform synthesis:** Direct iSTFT from predicted amplitude + phase (no external vocoder needed)
    52	- **Parameter count:** Lightweight all-CNN architecture with no attention layers
    53
    54	The dual-stream architecture predicts the **residual** high-frequency log-amplitude spectrum (added to the original), while the phase stream predicts "pseudo-real" and "pseudo-imaginary" components to handle phase wrapping. Discriminators operate at both waveform level (MPD) and spectral level (multi-resolution amplitude and phase discriminators).[6]
    55
    56	**Integration approach:** Run X-Codec 2.0 decoder at 16kHz → upsample to 48kHz with sinc interpolation → feed through AP-BWE to fill in 8–24kHz content. Since AP-BWE operates at frame level on STFT representations, its latency is minimal (one STFT frame).
    57
    58	### MS-BWE: Multi-Stage Cascading
    59
    60	MS-BWE extends AP-BWE with cascading BWE blocks for flexible multi-rate extension:[7]
    61
    62	- Each stage extends one frequency band (e.g., 16kHz→24kHz→48kHz)
    63	- One-stage GPU speed: ~1271× real-time (4× faster than full AP-BWE per stage)
    64	- Enables **controllable output rate**: stop at any intermediate stage for 24kHz, or run all stages for 48kHz
    65
    66	This directly supports your goal of conditional 16/24/48kHz output with gating.
    67
    68	### HiFi-GAN+ (Bandwidth Extension Is All You Need)
    69
    70	The Princeton approach uses a feed-forward WaveNet with multi-domain adversarial training for 8–16kHz→48kHz extension. HiFi-GAN-2 demonstrated a complete pipeline where a 16kHz speech enhancement model is followed by a dedicated BWE stage to reach 48kHz studio quality, achieving MOS of 4.27±0.03. An open-source reproduction is available at `bshall/hifi-gan-bwe`.[8][9][10]
    71
    72	### UBGAN: Ultra-Lightweight
    73
    74	UBGAN operates in the PQMF sub-band domain and is specifically designed as a modular post-processor for existing codecs:[11][12]
    75
    76	- **Causal** with 20ms frame size and 5ms look-ahead
    77	- Significantly fewer parameters and lower complexity than AP-BWE
    78	- Both blind (no side information) and guided (with small side channel) variants
    79	- Designed to generalize across multiple codec types and bitrates
    80
    81	This is particularly appealing for voice agent deployment where you want to add BWE without modifying the codec itself.
    82
    83	## Strategy 3: Flow Matching Post-Filter (Low-Overhead Generative)
    84
    85	### FLowHigh: Single-Step Flow Matching
    86
    87	FLowHigh achieves SOTA audio super-resolution quality with **just 1 ODE step** using conditional flow matching:[13][14]
    88
    89	- **Architecture:** 2-layer transformer (16 heads, 1024 dim, 4096 FFN) = 35.4M parameters
    90	- **RTF:** 0.1769 (single step)—13.6–24.4× faster than diffusion models requiring 50–100 steps[14]
    91	- **Quality:** Outperforms NVSR, AudioSR, Nu-wave2, UDM+, mdctGAN, and Fre-painter across all input rates on the VCTK benchmark
    92	- **External vocoder:** Uses BigVGAN (pretrained at 48kHz, 256 mel bins) for final waveform synthesis
    93
    94	| Model | NFEs | RTF | LSD (16→48) | ViSQOL (16→48) |
    95	|-------|------|-----|-------------|----------------|
    96	| Nu-wave2 | 50 | 1.2337 | 0.86 | 3.00 |
    97	| UDM+ | 50 | 2.2415 | 0.94 | 2.77 |
    98	| **FLowHigh** | **1** | **0.1769** | **0.71** | **3.80** |
    99
   100	FLowHigh operates at mel-spectrogram level: it takes a 16kHz mel-spectrogram temporally interpolated to 48kHz length, applies a single flow matching step to predict the high-resolution mel, then synthesizes with BigVGAN. The post-processing step copies original low-frequency content from the input, so only high-frequency bands are generated.[14]
   101
   102	**Trade-off:** This adds a transformer forward pass + BigVGAN synthesis. However, since it's **single-step** (not iterative), the overhead is bounded and predictable—roughly equivalent to one additional vocoder pass.
   103
   104	### FlowDec: Integrated Codec Post-Filter
   105
   106	FlowDec (ICLR 2025) combines non-adversarial codec training with a flow matching post-filter for 48kHz general audio:[15][16]
   107
   108	- Reduces required DNN evaluations from 60 (ScoreDec) to **6** without distillation
   109	- Achieves FAD scores better than DAC at 48kHz
   110	- Operates at 4–7.5 kbps
   111	- Designed as an integral part of the codec, not a separate module
   112
   113	### CodecFlow: Latent-Space BWE
   114
   115	CodecFlow (March 2026) performs bandwidth extension directly in the codec's continuous latent space using voicing-aware conditional flow matching:[17]
   116
   117	- Exploits the observation that LR and HR speech remain closely aligned in continuous codec latent space
   118	- Uses a structure-constrained RVQ for improved latent alignment
   119	- End-to-end fine-tuned with the codec
   120	- Achieves strong spectral fidelity on both 8→16kHz and 8→44.1kHz tasks
   121
   122	This is conceptually the most elegant approach for your architecture—performing BWE in X-Codec 2.0's latent space (post-VQ, pre-decoder) would avoid any waveform-domain processing overhead.
   123
   124	## Strategy 4: Modified Decoder Architecture
   125
   126	### Multi-Band iSTFT Synthesis
   127
   128	iSTFTNet2 introduced 2D CNNs for frequency upsampling within the ISTFT framework:[18]
   129
   130	- Operates in a few-frequency space, then upsamples in frequency dimension
   131	- Faster and more lightweight than standard iSTFTNet
   132	- Avoids the quality degradation seen when replacing too many upsampling layers with iSTFT
   133
   134	For your VocosBackbone, this could be implemented as: keep the transformer at 1024 dim operating at frame rate, then add a lightweight 2D CNN that upsamples the frequency dimension before the final iSTFT. This separates temporal modeling (transformer) from spectral filling (CNN).
   135
   136	**Multi-band variant (MB-iSTFT):** Split the target waveform into sub-bands using PQMF, predict each sub-band's STFT independently with smaller n_fft, then recombine. MS-Vocos applies iSTFT with reduced frame length and shift (1/4 of standard), then combines reconstructed sub-band waveforms. This dramatically reduces the per-head output dimension while maintaining full-band output.[19][20][21]
   137
   138	### WaveNeXt: Replace iSTFT with Learned Linear
   139
   140	WaveNeXt replaces the iSTFT layer with a trainable linear projection (no bias), mapping directly from the ConvNeXt output to waveform samples. For 48kHz models, the architecture was adapted by changing the output channel to match the new shift length. Results showed WaveNeXt achieves higher synthesis quality than Vocos while preserving inference speed, though it underperformed HiFi-GAN for some full-band E2E TTS conditions.[22]
   141
   142	### ComplexDec: Complex Spectral Domain
   143
   144	ComplexDec (Meta, 2025) operates entirely in the complex spectral domain at 48kHz with no up/downsampling layers:[23][24]
   145
   146	- Uses complex STFT input/output (320 hop, 510 n_fft, Hann window)
   147	- 150 Hz frame rate with 256 embedding dimension
   148	- 16 codebooks at 24 kbps
   149	- Avoids information loss from waveform-domain compression
   150	- Demonstrates strong out-of-domain robustness
   151
   152	This represents a clean-slate approach for your decoder side, though it would require more substantial retraining.
   153
   154	## Strategy 5: Controllable Multi-Rate Output
   155
   156	### Gated Resolution with Conditional ISTFT Head
   157
   158	To achieve selectable 16/24/32/48kHz output, consider a multi-head architecture:
   159
   160	1. **Shared VocosBackbone** processes the VQ output at frame rate
   161	2. **Multiple ISTFT heads** with different `n_fft`/`hop_length` configurations
   162	3. **Gated selection** at inference time based on desired output resolution
   163
   164	Each head is a single `Linear(hidden_dim → n_fft+2)` + ISTFT, so the parameter overhead per resolution is minimal (~2–8M per head depending on n_fft). Train all heads simultaneously with the same backbone using multi-resolution mel loss.
   165
   166	### MS-BWE Cascading for Progressive Enhancement
   167
   168	AP-BWE's multi-stage variant provides a natural gating mechanism:[7]
   169
   170	```
   171	16kHz (base) → [BWE Block 1] → 24kHz → [BWE Block 2] → 48kHz
   172	```
   173
   174	Each stage independently extends one frequency band. At inference, you stop at the desired resolution. The per-stage overhead is approximately 1/4 of the full AP-BWE model.[7]
   175
   176	### FlexiCodec-Style Dynamic Rate
   177
   178	FlexiCodec demonstrates controllable frame rates between 3Hz and 12.5Hz using a merging threshold parameter. A similar gating concept could be applied to the output resolution, where a conditioning signal controls how many frequency bands the decoder reconstructs.[25]
   179
   180	## Recommended Implementation Roadmap
   181
   182	### Phase 1: Quick Win (1–2 weeks)
   183
   184	**Replicate the proven 24kHz modification**:[2]
   185	- Freeze encoders + quantizer
   186	- Add AvgPool1d(k=2, stride=2) before quantization
   187	- Increase hop to 960, update ISTFT params for 24kHz
   188	- Interpolate decoder weights from pretrained checkpoint
   189	- Fine-tune decoder on your HiFi-TTS2 pre-encoded data
   190	- Expected: +0.29 MOS, 12kHz Nyquist, eliminates most "muffled" quality
   191
   192	### Phase 2: Full-Band Post-Filter (2–4 weeks)
   193
   194	**Add AP-BWE or FLowHigh as a lightweight post-processor:**
   195
   196	| Method | Latency Added | Quality | Complexity |
   197	|--------|--------------|---------|------------|
   198	| AP-BWE | ~1ms (GPU) | Excellent | 292× RT, all-CNN |
   199	| FLowHigh (1 step) | ~18ms (GPU) | SOTA | 35.4M params + BigVGAN |
   200	| MS-BWE (1 stage) | ~0.8ms (GPU) | Very good | ~1271× RT per stage |
   201	| UBGAN blind | <1ms | Good | Ultra-lightweight, causal |
   202
   203	AP-BWE is the strongest recommendation for your voice agent use case: it is pure CNN (no attention overhead), processes frame-by-frame, and achieves near-studio quality at 48kHz. You can train it on paired 16kHz/48kHz data from HiFi-TTS.[5]
   204
   205	### Phase 3: Integrated Decoder Redesign (4–8 weeks)
   206
   207	**Retrain decoder from scratch for native 48kHz:**
   208	- Widen VocosBackbone to 1536 or 2048 hidden dim
   209	- Increase transformer layers from 12 to 16
   210	- Set ISTFT head to n_fft=3840, hop=960, producing 1921 freq bins
   211	- Add multi-scale sub-band CQT discriminator (from BigVGAN-v2)
   212	- Train with multi-resolution mel loss at 48kHz
   213
   214	### Phase 4: Multi-Rate Gating (Optional, +2 weeks on Phase 3)
   215
   216	- Add parallel ISTFT heads for 16k/24k/48k
   217	- Condition on a resolution embedding fed into the VocosBackbone
   218	- Train jointly with random resolution sampling per batch
   219
   220	## Methods Comparison
   221
   222	| Approach | Output SR | Retraining Scope | Added Inference Cost | Quality Potential | Implementation Difficulty |
   223	|----------|----------|-------------------|---------------------|-------------------|--------------------------|
   224	| Decoder retrain → 24kHz | 24kHz | Decoder only | None | Good (+0.29 MOS)[2] | Low |
   225	| Decoder retrain → 48kHz | 48kHz | Decoder + wider backbone | None | Good–Excellent | Medium |
   226	| AP-BWE post-filter | 48kHz | Separate model | ~1ms/utterance | Excellent[5] | Low–Medium |
   227	| FLowHigh post-filter | 48kHz | Separate model | ~18ms/utterance | SOTA[14] | Medium |
   228	| MS-BWE cascade | 16–48kHz | Separate model | 0.8ms/stage | Very Good[7] | Medium |
   229	| Latent-space flow (CodecFlow) | 44.1kHz+ | End-to-end fine-tune | 6 NFEs | Excellent[17] | High |
   230	| Multi-head decoder | 16–48kHz | Full decoder | None (head selection) | Good–Excellent | Medium–High |
   231	| MB-iSTFT sub-band | 48kHz | Full decoder | None | Good | Medium |
   232
   233	## Key Takeaways
   234
   235	The "muffled" quality issue is entirely solvable. The most pragmatic path is a two-phase approach: first, apply the proven decoder-only modification to reach 24kHz (which immediately eliminates most perceptual degradation), then layer on AP-BWE or a single-step FLowHigh post-filter for full 48kHz when needed. This modular design lets you ship the 24kHz improvement quickly while developing the full-band solution in parallel.[14][2][5]
   236
   237	For controllable multi-rate output, MS-BWE's cascading architecture offers the cleanest solution—each stage is independently trainable and adds minimal latency. Alternatively, a multi-head ISTFT decoder with resolution conditioning provides zero-overhead rate switching but requires more substantial retraining.[7]
   238
   239	Critically, since you have pre-encoded data from HiFi-TTS2 and the encoder is frozen, all decoder-side experiments can reuse the same latent codes. This dramatically reduces iteration time—you only need to train the decoder components, which are ~100M parameters (VocosBackbone) plus the heads, not the full 580M Wav2Vec2-BERT pipeline.


Summary: auto


**Synthesizing comprehensive suggestion inventory**


I’ve completed manual extraction and I’m now reconciling duplicates and conflicts into a single decision-complete ensemble spec. I’ll run one quick structured scrape of bullet/numbered action lines to validate that no actionable item was missed before finalizing.


for f in /home/ubuntu/xcodec-sr/SRPlans/*.md; do echo '===== '$(basename "$f")' ====='; rg -n '^\s*[-*]\s+|^\s*[0-9]+\.\s+' "$f"; echo; done


Output:
Chunk ID: ea18ed
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 4238
Output:
===== Claude.md =====
98:- Multi-resolution mel-spectrogram L1 loss across 5–7 mel-bin counts (32, 64, 128, 256, 512) and multiple STFT configurations — this is the primary reconstruction signal
99:- Multi-resolution STFT loss (spectral convergence + log magnitude) at configurations like (2048,512), (1024,256), (512,128), (256,64)
100:- GAN adversarial loss (least-squares formulation) from all discriminators
101:- Feature matching loss (L1 on intermediate discriminator features)
102:- If operating in spectral domain: amplitude spectrum MSE + anti-wrapping phase losses (instantaneous phase error, group delay continuity, instantaneous frequency continuity)
108:1. **Warmup (0–20k steps)**: Reconstruction losses only (no GAN). Learning rate 2×10⁻⁴ for decoder with AdamW. This stabilizes the modified output dimensions before adversarial training.
109:2. **GAN activation (20k–100k steps)**: Gradually introduce discriminator losses. Use separate Adam optimizer for discriminators (lr=2×10⁻⁴). EnCodec's gradient balancer mechanism (where each loss weight defines its fraction of the overall gradient) helps with loss balancing.
110:3. **Full training (100k–400k steps)**: All losses active. Exponential LR decay (~0.999996 per step). SNAC demonstrates stable training without gradient clipping when using depthwise convolutions.
122:1. **Direct ISTFTHead rescaling to 24kHz** (n_fft=2048, hop=480). Directly validated by X-Codec 2.0 modification paper. Initialize via weight interpolation. Fine-tune 100–200k steps. Expected result: +0.29 MOS or better over 16kHz baseline. RTF ~0.001 on GPU.
124:2. **AP-BWE as post-hoc module** for 16kHz→48kHz. Pre-existing open-source code and methodology. Train on codec output → 48kHz target pairs. RTF ~0.003 on GPU, ~0.055 on CPU. Completely modular — no changes to existing codec.
128:3. **Multi-band ISTFTHead (4 bands) for 48kHz**. Each sub-band at 12kHz with PQMF synthesis. Reuse existing 16kHz head architecture where possible. Enables sub-band-specific discriminators. More architectural work but better frequency decomposition for 48kHz.
130:4. **MusicHiFi-BWE-style separate upsampler**. HiFi-GAN generator with 3× upsampling + skip connection from input. ~1786× real-time. Simple to implement and proven for generative model post-processing.
134:5. **FLowHigh single-step flow matching** for mel-spectrogram super-resolution + Vocos/BigVGAN vocoder. SOTA metrics on benchmarks. RTF ~0.07 — tight but within budget.
136:6. **Consistency model distillation** of a diffusion-based audio SR model. No dedicated implementation exists yet, but CoMoSpeech demonstrates the approach achieves RTF ~0.007 for TTS — the same distillation framework applies to super-resolution.
140:7. **Controllable multi-resolution decoder** with sample-rate conditioning embedding (extending Vocos's existing bandwidth_id mechanism). Train a single decoder that outputs 16/24/32/48kHz based on a conditioning signal. Requires more training data diversity but offers maximum deployment flexibility.
142:8. **VQ-based upsampling** where a small transformer predicts 48kHz codec tokens from 16kHz codec tokens. Interesting but autoregressive prediction creates a latency bottleneck unless non-autoregressive methods (masked prediction) are used.

===== GPT.md =====
13:* You don’t touch the codec/quantizer.
14:* You can make it **streaming + causal**, with **single-digit ms algorithmic delay**.
15:* You can ship **16 kHz fast path** and **32/48 kHz HQ path** as a toggle.
34:1. For each HiFi-TTS2 clip, you already have **codes**.
35:2. Decode them with your current decoder → **ŷ16k_codec**.
36:3. Train BWE model: **ŷ16k_codec → y48k_gt** (or y32k_gt).
38:   * Input is upsampled to target rate with causal sinc (or polyphase). BBWEXNet does causal sinc upsampling as a preprocessing step.
42:* **Multi-resolution log-STFT losses** (several FFT sizes)
43:* **Upperband-weighted mel loss** (only care about missing band strongly)
44:* **Spectral discriminator** (on mel) + **multi-rate waveform discriminators**
55:* Keep your decoder output as “low band” anchor.
56:* Predict **only the missing band** as a residual:
58:  * 16k → 48k: missing is ~8–24 kHz
59:  * 16k → 32k: missing is ~8–16 kHz (often the sweet spot)
63:* **PQMF / subband split**: generate only the top subbands, then recombine.
64:* Or **STFT-domain high-band prediction**: predict high-frequency magnitude + phase residual.
68:* Your decoder already works in an STFT-ish space (it predicts mag+phase then ISTFT).
69:* There’s newer BWE work that explicitly models **amplitude + phase** in parallel for quality *and* speed.
87:* `hop_length = target_sr / 50`
89:  * 24k → 480
90:  * 32k → 640
91:  * 48k → 960
92:* Keep your window length in *time* consistent (you used 80 ms at 16k: 1280 samples). So:
94:  * 24k: `n_fft ≈ 1920`
95:  * 32k: `n_fft ≈ 2560`
96:  * 48k: `n_fft ≈ 3840`
102:* Your ISTFT head linear goes from `1024→1282` to `1024→3842` at 48k. That’s ~3× params in that layer, not 10×.
103:* Backbone compute stays at 50 Hz.
115:* **NU-Wave 2**: diffusion SR that’s explicitly built to handle **multiple input sample rates with conditioning** (bandwidth conditioning via BSFT). Great for your “16/24/32/48 controllability” idea, but diffusion baseline is slower.
116:* **FlashSR**: **one-step** diffusion via distillation for 48 kHz SR. This is exactly “diffusion, but not slow,” at least compared to classic samplers. ([arXiv][2])
120:* Train a **heavy teacher** (diffusion SR) offline.
121:* Distill into a **GAN / conv student** that runs real-time.
131:1. **Target-SR embedding** (FiLM-style or additive conditioning): give the SR module a learned embedding for `{16,24,32,48}` so it learns “how much band to invent.” NU-Wave 2 does bandwidth conditioning (different mechanism, same idea).
132:2. **Predict bands progressively**: always output base 16k, then optionally add a learned 8–16k band (for 32k), then optionally add 16–24k (for 48k). This maps nicely to a subband/PQMF design.
138:1. **Target 32 kHz first**, not 48. It’s where speech “opens up” the most per FLOP.
139:2. Train a **post-BWE** model on **codec-decoded 16k** → 32k/48k ground truth, using the HiFi-GAN+ style losses/discriminators (upperband-weighted).
140:3. If you want extra polish: switch to an **amplitude+phase** high-band predictor like AP-BWE style. ([arXiv][1])
141:4. Only if you still hate it: retrain the ISTFTHead geometry and make a native 32/48k decoder.

===== GPT_PRO.md =====
7:1. **24 kHz decoder-only retrain first**
8:2. **Residual high-band restorer conditioned on codec latents**
9:3. **A stronger non-diffusion decoder replacement with harmonic/phase priors**
10:4. **Single-step latent flow or distilled diffusion only as an optional HQ tier**
30:* 16 kHz: hop 320, win/n_fft 1280
31:* 24 kHz: hop 480, win/n_fft about 1920
32:* 32 kHz: hop 640, win/n_fft about 2560
33:* 48 kHz: hop 960, win/n_fft about 3840
39:* **APNet2 / APCodec-style head**: separate amplitude and phase streams after a shared trunk.
40:* **Wavehax-lite / HiFTNet-lite head**: harmonic-prior branch plus spectral decoder.
50:1. Keep your current 16 kHz decoder exactly as the **base low-band path**.
51:2. Deterministically upsample it to 24/32/48 kHz with a good sinc/polyphase resampler.
52:3. Add a tiny **high-band residual model** that predicts only the missing band in STFT space.
53:4. Condition that branch on:
55:   * `vq_post_emb` or decoder hidden states
56:   * the upsampled base waveform or its STFT
57:   * optional F0 / voiced-unvoiced / energy features
63:* residual log-magnitude for bins above 8 kHz
64:* direct phase for those bins, or complex residual if you want a cleaner implementation
72:* HiFTNet: harmonic-plus-noise source/filter helps.
73:* Wavehax: harmonic prior improves robustness, especially high F0.
74:* CodecFlow: explicit voicing-aware conditioning helps recover high-frequency unvoiced detail in codec-latent BWE. ([arXiv][7])
84:* **APCodec-style decoder**
87:* **Wavehax-lite**
90:* **HiFTNet-lite**
101:* **Base16** branch: reconstructs the existing 0-8 kHz band
102:* **+HF24** branch: adds 8-12 kHz
103:* **+HF32** branch: adds 12-16 kHz
104:* **+HF48** branch: adds 16-24 kHz
120:* **Low-band identity loss or hard low-band replacement**
121:* **High-band-weighted complex STFT loss**
122:* **Amplitude loss + phase loss** if using dual-stream output
123:* **A small waveform adversarial loss**
124:* **One spectral discriminator focused on the new band**
125:* **Optional F0 / UV auxiliary loss** if you add voicing conditioning
159:* **single-step distilled diffusion**, like FlashSR, which reports competitive 48 kHz SR while being about 22x faster than the previous SOTA it compares against;
160:* **single-step flow matching**, like FLowHigh, which reports strong VCTK SR results with one-step sampling;
161:* **latent-space probabilistic models**, which are much more aligned with your cached-token setup than waveform diffusion. Latent-domain upsampling work reports up to 100x efficiency gains over raw-audio post-processing, and the very fresh March 2026 CodecFlow paper specifically does voicing-aware bandwidth extension in codec latent space. ([arXiv][11])
169:1. **Train a 24 kHz decoder-only variant first** using your cached codes and original high-rate waveforms.
170:2. **Replace the current head with a dual amplitude/phase or harmonic-prior decoder**, not just a naïve bigger ISTFT head.
171:3. **Add a residual high-band branch with low-band bypass**, conditioned on `vq_post_emb` and F0/UV.
172:4. **Only then** build a cascaded 24 -> 48 stage if 24 kHz still isn’t enough.
173:5. Keep **single-step latent flow/diffusion** as an optional HQ path, not the default.

===== Gemini.md =====
13:* **The Concept**: Keep your current 16kHz tokens as the "base." Train a new decoder that predicts multiple sub-bands (e.g., 0-8kHz, 8-16kHz, 16-24kHz).
14:* **Why it works**: High frequencies are often more "noise-like" or periodic in specific ways. By isolating the high-frequency reconstruction, the model doesn't "smear" the base band with high-frequency artifacts.
15:* **Implementation**:
16:* Modify the `ISTFTHead` to output multiple sets of magnitude/phase pairs (one for each band).
17:* Use a **Pseudo-Quadrature Mirror Filter (PQMF)** bank to synthesize the final 48kHz waveform from these bands.
18:* **Latency**: Minimal. PQMF synthesis is a series of simple convolutions.
26:* **The Method**: Replace the Vocos Transformer's output projection with a series of **Upsampling Blocks** using **SnakeBeta** activations and **Alias-free convolutions**.
27:* **Novelty**: Most upsamplers fail because they alias high frequencies back into the audible range. By using the alias-free processing (Kaiser-sinc filters) you already have in the `Activation1d` module, you can push the resolution to 44.1kHz or 48kHz without "metallic" artifacts.
28:* **Retraining**: You can freeze the `CodecEncoder` and `Wav2Vec2-BERT` and only train the `VocosBackbone` and the new upsampling head using a Multi-Resolution Mel Loss + Multi-Period Discriminator (MPD) operating at the higher sample rate.
34:* **Technique**:
35:* The decoder predicts a **Source** (excitation) and a **Filter** (envelope) at 48kHz.
36:* The source can be generated using a periodic generator (for voiced parts) and noise (for unvoiced parts).
37:* This is much more "stable" than trying to predict raw ISTFT bins for 48kHz, which requires a very large `n_fft` (e.g., 2048 or 4096) and increases the hop-length/latency.
45:* **The Architecture**:
46:* Use a shared backbone (your current Transformer).
47:* Implement **Multiple Output Heads**: Head A (16kHz), Head B (24kHz), etc.
48:* **Gating**: Use a conditional embedding (a scalar or one-hot vector) passed into the Transformer and the heads.
49:* **Training**: Train with "Resolution Dropout"—randomly downsample your 48kHz ground truth to 16kHz or 24kHz during training, and force the model to reconstruct the original 48kHz. This teaches the model to "hallucinate" high frequencies when the input is low-res, but remain faithful when the input is high-res.
70:1. **Keep the Encoder/VQ**: Do not touch them. Your 50Hz frame rate is perfect.
71:2. **Upgrade the Decoder**:
72:* Increase the `VocosBackbone` width slightly if needed.
73:* Change the `ISTFTHead` to target 24kHz or 48kHz.
74:* **Crucial**: If you keep the hop length at 320 for 16kHz tokens, but want 48kHz output, your effective hop length in the output is $320 \times 3 = 960$.
75:* Use **Sub-band Discriminators**: Train with discriminators that specifically look at the 8-16kHz and 16-24kHz ranges (high-pass filtered) to force the model to generate energy in those bands.

===== perplexity.md =====
13:- **Freeze all encoder components** (CodecEncoder, Wav2Vec2-BERT, SemanticEncoder) and the quantizer
14:- **Increase the decoder hop size** from 320 to 960 samples
15:- **Add an AvgPool1d(kernel=2, stride=2)** before vector quantization to reduce the latent rate from 50Hz to 25Hz
16:- **Retrain only the decoder** (VocosBackbone + ISTFTHead) to output 24kHz audio
17:- **Interpolate decoder weights** from the pretrained checkpoint using 1D linear interpolation on the output projection
48:- **Architecture:** Dual-stream ConvNeXt backbone with parallel amplitude and phase prediction streams plus inter-stream connections
49:- **Speed:** 292.3× faster than real-time on a single RTX 4090 GPU, and 18.1× faster on a single CPU core for 16kHz→48kHz[5]
50:- **Quality:** State-of-the-art on both 16kHz and 48kHz target rates
51:- **Waveform synthesis:** Direct iSTFT from predicted amplitude + phase (no external vocoder needed)
52:- **Parameter count:** Lightweight all-CNN architecture with no attention layers
62:- Each stage extends one frequency band (e.g., 16kHz→24kHz→48kHz)
63:- One-stage GPU speed: ~1271× real-time (4× faster than full AP-BWE per stage)
64:- Enables **controllable output rate**: stop at any intermediate stage for 24kHz, or run all stages for 48kHz
76:- **Causal** with 20ms frame size and 5ms look-ahead
77:- Significantly fewer parameters and lower complexity than AP-BWE
78:- Both blind (no side information) and guided (with small side channel) variants
79:- Designed to generalize across multiple codec types and bitrates
89:- **Architecture:** 2-layer transformer (16 heads, 1024 dim, 4096 FFN) = 35.4M parameters
90:- **RTF:** 0.1769 (single step)—13.6–24.4× faster than diffusion models requiring 50–100 steps[14]
91:- **Quality:** Outperforms NVSR, AudioSR, Nu-wave2, UDM+, mdctGAN, and Fre-painter across all input rates on the VCTK benchmark
92:- **External vocoder:** Uses BigVGAN (pretrained at 48kHz, 256 mel bins) for final waveform synthesis
108:- Reduces required DNN evaluations from 60 (ScoreDec) to **6** without distillation
109:- Achieves FAD scores better than DAC at 48kHz
110:- Operates at 4–7.5 kbps
111:- Designed as an integral part of the codec, not a separate module
117:- Exploits the observation that LR and HR speech remain closely aligned in continuous codec latent space
118:- Uses a structure-constrained RVQ for improved latent alignment
119:- End-to-end fine-tuned with the codec
120:- Achieves strong spectral fidelity on both 8→16kHz and 8→44.1kHz tasks
130:- Operates in a few-frequency space, then upsamples in frequency dimension
131:- Faster and more lightweight than standard iSTFTNet
132:- Avoids the quality degradation seen when replacing too many upsampling layers with iSTFT
146:- Uses complex STFT input/output (320 hop, 510 n_fft, Hann window)
147:- 150 Hz frame rate with 256 embedding dimension
148:- 16 codebooks at 24 kbps
149:- Avoids information loss from waveform-domain compression
150:- Demonstrates strong out-of-domain robustness
160:1. **Shared VocosBackbone** processes the VQ output at frame rate
161:2. **Multiple ISTFT heads** with different `n_fft`/`hop_length` configurations
162:3. **Gated selection** at inference time based on desired output resolution
185:- Freeze encoders + quantizer
186:- Add AvgPool1d(k=2, stride=2) before quantization
187:- Increase hop to 960, update ISTFT params for 24kHz
188:- Interpolate decoder weights from pretrained checkpoint
189:- Fine-tune decoder on your HiFi-TTS2 pre-encoded data
190:- Expected: +0.29 MOS, 12kHz Nyquist, eliminates most "muffled" quality
208:- Widen VocosBackbone to 1536 or 2048 hidden dim
209:- Increase transformer layers from 12 to 16
210:- Set ISTFT head to n_fft=3840, hop=960, producing 1921 freq bins
211:- Add multi-scale sub-band CQT discriminator (from BigVGAN-v2)
212:- Train with multi-resolution mel loss at 48kHz
216:- Add parallel ISTFT heads for 16k/24k/48k
217:- Condition on a resolution embedding fed into the VocosBackbone
218:- Train jointly with random resolution sampling per batch


**Inspecting repo for integration points**


I’m now grounding the ensemble spec in your actual code paths (`vq/*`, `lightning_module.py`, `inference.py`) so the final architecture is implementation-ready, not just paper-level.


rg -n "class CodecDecoderVocos|class ISTFTHead|ResidualFSQ|fc_post_a|fc_prior|SemanticDecoder|VocosBackbone|def forward|vq=" /home/ubuntu/xcodec-sr/vq /home/ubuntu/xcodec-sr/lightning_module.py /home/ubuntu/xcodec-sr/inference.py


nl -ba /home/ubuntu/xcodec-sr/inference.py | sed -n '1,240p'


nl -ba /home/ubuntu/xcodec-sr/lightning_module.py | sed -n '1,320p'


Output:
Chunk ID: 54e259
Wall time: 0.0510 seconds
Process exited with code 0
Original token count: 1970
Output:
/home/ubuntu/xcodec-sr/inference.py:17:from vq.module import SemanticDecoder,SemanticEncoder
/home/ubuntu/xcodec-sr/inference.py:38:    filtered_state_dict_fc_post_a = OrderedDict()
/home/ubuntu/xcodec-sr/inference.py:39:    filtered_state_dict_fc_prior = OrderedDict()
/home/ubuntu/xcodec-sr/inference.py:49:        elif key.startswith('fc_post_a.'):
/home/ubuntu/xcodec-sr/inference.py:51:            new_key = key[len('fc_post_a.'):]
/home/ubuntu/xcodec-sr/inference.py:52:            filtered_state_dict_fc_post_a[new_key] = value
/home/ubuntu/xcodec-sr/inference.py:57:        elif key.startswith('fc_prior.'):
/home/ubuntu/xcodec-sr/inference.py:59:            new_key = key[len('fc_prior.'):]
/home/ubuntu/xcodec-sr/inference.py:60:            filtered_state_dict_fc_prior[new_key] = value
/home/ubuntu/xcodec-sr/inference.py:73:    fc_post_a = nn.Linear( 2048, 1024 )
/home/ubuntu/xcodec-sr/inference.py:74:    fc_post_a.load_state_dict(filtered_state_dict_fc_post_a)
/home/ubuntu/xcodec-sr/inference.py:75:    fc_post_a = fc_post_a.eval().cuda()
/home/ubuntu/xcodec-sr/inference.py:76:    fc_prior = nn.Linear( 2048, 2048 )
/home/ubuntu/xcodec-sr/inference.py:77:    fc_prior.load_state_dict(filtered_state_dict_fc_prior)
/home/ubuntu/xcodec-sr/inference.py:78:    fc_prior = fc_prior.eval().cuda()
/home/ubuntu/xcodec-sr/inference.py:126:            vq_emb =  fc_prior(vq_emb.transpose(1, 2)).transpose(1, 2)
/home/ubuntu/xcodec-sr/inference.py:128:            _, vq_code, _ = decoder(vq_emb, vq=True)  # vq_code here !!!!
/home/ubuntu/xcodec-sr/inference.py:132:            vq_post_emb = fc_post_a(vq_post_emb.transpose(1,2)).transpose(1,2)
/home/ubuntu/xcodec-sr/inference.py:133:            recon = decoder(vq_post_emb.transpose(1, 2), vq=False)[0].squeeze().detach().cpu().numpy()
/home/ubuntu/xcodec-sr/inference.py:134:            # recon = decoder(decoder.vq2emb(vq_code.transpose(1,2)).transpose(1,2), vq=False).squeeze().detach().cpu().numpy()
/home/ubuntu/xcodec-sr/lightning_module.py:17:from vq.module import SemanticDecoder,SemanticEncoder
/home/ubuntu/xcodec-sr/lightning_module.py:110:        self.fc_prior = nn.Linear(1024 + 1024, deccfg.vq_dim,   )
/home/ubuntu/xcodec-sr/lightning_module.py:111:        self.fc_post_a = nn.Linear(deccfg.vq_dim,  deccfg.hidden_dim )
/home/ubuntu/xcodec-sr/lightning_module.py:114:        self.SemanticDecoder_module = SemanticDecoder(1024, 1024, 1024)
/home/ubuntu/xcodec-sr/lightning_module.py:146:    def forward(self, batch):
/home/ubuntu/xcodec-sr/lightning_module.py:163:        vq_emb = self.fc_prior(vq_emb.transpose(1, 2)).transpose(1, 2)
/home/ubuntu/xcodec-sr/lightning_module.py:165:        vq_post_emb, vq_code, vq_loss = self.generator(vq_emb, vq=True)
/home/ubuntu/xcodec-sr/lightning_module.py:167:        semantic_recon = self.SemanticDecoder_module(semantic_recon)
/home/ubuntu/xcodec-sr/lightning_module.py:171:            self.fc_post_a(vq_post_emb.transpose(1, 2)) ,
/home/ubuntu/xcodec-sr/lightning_module.py:172:            vq=False
/home/ubuntu/xcodec-sr/lightning_module.py:201:        vq_post_emb, vq_code, vq_loss = self.generator(vq_emb, vq=True)
/home/ubuntu/xcodec-sr/lightning_module.py:202:        y_ = self.generator(vq_post_emb, vq=False).squeeze(1)  # [B, T]
/home/ubuntu/xcodec-sr/lightning_module.py:398:            self.fc_prior.parameters(),
/home/ubuntu/xcodec-sr/lightning_module.py:399:            self.fc_post_a.parameters(),
/home/ubuntu/xcodec-sr/lightning_module.py:401:            self.SemanticDecoder_module.parameters(),
/home/ubuntu/xcodec-sr/vq/module.py:30:    def forward(self, x):
/home/ubuntu/xcodec-sr/vq/module.py:49:    def forward(self, x):
/home/ubuntu/xcodec-sr/vq/module.py:68:    def forward(self, x):
/home/ubuntu/xcodec-sr/vq/module.py:82:    def forward(self, x):
/home/ubuntu/xcodec-sr/vq/module.py:134:    def forward(self, x: torch.Tensor, cond_embedding_id: Optional[torch.Tensor] = None) -> torch.Tensor:
/home/ubuntu/xcodec-sr/vq/module.py:172:    def forward(self, x: torch.Tensor, cond_embedding_id: torch.Tensor) -> torch.Tensor:
/home/ubuntu/xcodec-sr/vq/module.py:263:    def forward(self, x: torch.Tensor) -> torch.Tensor:
/home/ubuntu/xcodec-sr/vq/module.py:361:    def forward(self, x):
/home/ubuntu/xcodec-sr/vq/module.py:376:class SemanticDecoder(nn.Module):
/home/ubuntu/xcodec-sr/vq/module.py:385:        super(SemanticDecoder, self).__init__()
/home/ubuntu/xcodec-sr/vq/module.py:415:    def forward(self, z):
/home/ubuntu/xcodec-sr/vq/residual_vq.py:21:    def forward(self, x):
/home/ubuntu/xcodec-sr/vq/blocks.py:16:    def forward(self, x: torch.Tensor) -> torch.Tensor:
/home/ubuntu/xcodec-sr/vq/blocks.py:26:    def forward(self, x: torch.Tensor) -> torch.Tensor:
/home/ubuntu/xcodec-sr/vq/alias_free_torch/resample.py:25:    def forward(self, x):
/home/ubuntu/xcodec-sr/vq/alias_free_torch/resample.py:46:    def forward(self, x):
/home/ubuntu/xcodec-sr/vq/alias_free_torch/filter.py:86:    def forward(self, x):
/home/ubuntu/xcodec-sr/vq/codec_decoder_vocos.py:15:from vector_quantize_pytorch import ResidualFSQ
/home/ubuntu/xcodec-sr/vq/codec_decoder_vocos.py:43:    def forward(self, spec: torch.Tensor) -> torch.Tensor:
/home/ubuntu/xcodec-sr/vq/codec_decoder_vocos.py:92:    def forward(self, x: torch.Tensor) -> torch.Tensor:
/home/ubuntu/xcodec-sr/vq/codec_decoder_vocos.py:104:class ISTFTHead(FourierHead):
/home/ubuntu/xcodec-sr/vq/codec_decoder_vocos.py:122:    def forward(self, x: torch.Tensor) -> torch.Tensor:
/home/ubuntu/xcodec-sr/vq/codec_decoder_vocos.py:200:    def forward(self, x, temb=None):
/home/ubuntu/xcodec-sr/vq/codec_decoder_vocos.py:249:    def forward(self, x):
/home/ubuntu/xcodec-sr/vq/codec_decoder_vocos.py:281:    def forward(self, x: torch.Tensor, **kwargs) -> torch.Tensor:
/home/ubuntu/xcodec-sr/vq/codec_decoder_vocos.py:294:class VocosBackbone(Backbone):
/home/ubuntu/xcodec-sr/vq/codec_decoder_vocos.py:348:    def forward(self, x: torch.Tensor ) -> torch.Tensor:
/home/ubuntu/xcodec-sr/vq/codec_decoder_vocos.py:371:class CodecDecoderVocos(nn.Module):
/home/ubuntu/xcodec-sr/vq/codec_decoder_vocos.py:389:        self.quantizer = ResidualFSQ(
/home/ubuntu/xcodec-sr/vq/codec_decoder_vocos.py:407:        self.backbone = VocosBackbone( hidden_dim=hidden_dim,depth=depth,heads=heads,pos_meb_dim=pos_meb_dim)
/home/ubuntu/xcodec-sr/vq/codec_decoder_vocos.py:413:    def forward(self, x, vq=True):
/home/ubuntu/xcodec-sr/vq/codec_decoder_vocos.py:476:class CodecDecoderVocos_transpose(nn.Module):
/home/ubuntu/xcodec-sr/vq/codec_decoder_vocos.py:507:        self.backbone = VocosBackbone( hidden_dim=hidden_dim,depth=depth,heads=heads,pos_meb_dim=pos_meb_dim)
/home/ubuntu/xcodec-sr/vq/codec_decoder_vocos.py:532:    def forward(self, x, vq=True):
/home/ubuntu/xcodec-sr/vq/codec_decoder_vocos.py:614:    #         output, q, commit_loss = model(dummy_input, vq=True)
/home/ubuntu/xcodec-sr/vq/codec_decoder_vocos.py:625:        output_no_vq = model(dummy_input, vq=False)
/home/ubuntu/xcodec-sr/vq/alias_free_torch/act.py:23:    def forward(self, x):
/home/ubuntu/xcodec-sr/vq/bs_roformer5.py:21:    def forward(self, x):
/home/ubuntu/xcodec-sr/vq/bs_roformer5.py:36:    def forward(self, x):
/home/ubuntu/xcodec-sr/vq/bs_roformer5.py:60:    def forward(self, x):
/home/ubuntu/xcodec-sr/vq/bs_roformer5.py:104:    def forward(
/home/ubuntu/xcodec-sr/vq/activations.py:48:    def forward(self, x):
/home/ubuntu/xcodec-sr/vq/activations.py:107:    def forward(self, x):
/home/ubuntu/xcodec-sr/vq/codec_encoder.py:26:    def forward(self, x):
/home/ubuntu/xcodec-sr/vq/codec_encoder.py:65:    def forward(self, x):
/home/ubuntu/xcodec-sr/vq/codec_encoder.py:145:    def forward(self, x):
/home/ubuntu/xcodec-sr/vq/codec_encoder.py:221:    def forward(self, x: torch.Tensor ) -> torch.Tensor:
/home/ubuntu/xcodec-sr/vq/factorized_vector_quantize.py:29:    def forward(self, z):


Output:
Chunk ID: bdd1f8
Wall time: 0.0511 seconds
Process exited with code 0
Original token count: 1720
Output:
     1	import os
     2	import librosa
     3	import torch
     4	import torch.nn.functional as F
     5	import numpy as np
     6	import soundfile as sf
     7	from glob import glob
     8	from tqdm import tqdm
     9	from os.path import basename, join, exists
    10	from vq.codec_encoder import CodecEncoder
    11	# from vq.codec_decoder import CodecDecoder
    12	from vq.codec_decoder_vocos import CodecDecoderVocos
    13	from argparse import ArgumentParser
    14	from time import time
    15	from transformers import  AutoModel
    16	import torch.nn as nn
    17	from vq.module import SemanticDecoder,SemanticEncoder
    18	from transformers import AutoFeatureExtractor, Wav2Vec2BertModel
    19	if __name__ == '__main__':
    20	    parser = ArgumentParser()
    21	    parser.add_argument('--input-dir', type=str, default='test_audio/input_test')
    22	    parser.add_argument('--ckpt', type=str, default='ckpt/epoch=4-step=1400000.ckpt')
    23	    parser.add_argument('--output-dir',   type=str, default='test_audio/output_test')
    24
    25	    args = parser.parse_args()
    26	    sr = 16000
    27
    28	    print(f'Load codec ckpt from {args.ckpt}')
    29	    ckpt = torch.load(args.ckpt, map_location='cpu')
    30	    ckpt=ckpt['state_dict']
    31
    32	    state_dict = ckpt
    33	    from collections import OrderedDict
    34	    # 步骤 2：提取并过滤 'codec_enc' 和 'generator' 部分
    35	    filtered_state_dict_codec = OrderedDict()
    36	    filtered_state_dict_semantic_encoder = OrderedDict()
    37	    filtered_state_dict_gen = OrderedDict()
    38	    filtered_state_dict_fc_post_a = OrderedDict()
    39	    filtered_state_dict_fc_prior = OrderedDict()
    40	    for key, value in state_dict.items():
    41	        if key.startswith('CodecEnc.'):
    42	            # 去掉 'codec_enc.' 前缀，以匹配 CodecEncoder 的参数名
    43	            new_key = key[len('CodecEnc.'):]
    44	            filtered_state_dict_codec[new_key] = value
    45	        elif key.startswith('generator.'):
    46	            # 去掉 'generator.' 前缀，以匹配 CodecDecoder 的参数名
    47	            new_key = key[len('generator.'):]
    48	            filtered_state_dict_gen[new_key] = value
    49	        elif key.startswith('fc_post_a.'):
    50	            # 去掉 'generator.' 前缀，以匹配 CodecDecoder 的参数名
    51	            new_key = key[len('fc_post_a.'):]
    52	            filtered_state_dict_fc_post_a[new_key] = value
    53	        elif key.startswith('SemanticEncoder_module.'):
    54	            # 去掉 'generator.' 前缀，以匹配 CodecDecoder 的参数名
    55	            new_key = key[len('SemanticEncoder_module.'):]
    56	            filtered_state_dict_semantic_encoder[new_key] = value
    57	        elif key.startswith('fc_prior.'):
    58	            # 去掉 'generator.' 前缀，以匹配 CodecDecoder 的参数名
    59	            new_key = key[len('fc_prior.'):]
    60	            filtered_state_dict_fc_prior[new_key] = value
    61	    semantic_model = Wav2Vec2BertModel.from_pretrained("facebook/w2v-bert-2.0", output_hidden_states=True)
    62
    63	    semantic_model.eval().cuda()
    64	    SemanticEncoder_module = SemanticEncoder(1024,1024,1024)
    65	    SemanticEncoder_module.load_state_dict(filtered_state_dict_semantic_encoder)
    66	    SemanticEncoder_module = SemanticEncoder_module.eval().cuda()
    67	    encoder = CodecEncoder()
    68	    encoder.load_state_dict(filtered_state_dict_codec)
    69	    encoder = encoder.eval().cuda()
    70	    decoder = CodecDecoderVocos()
    71	    decoder.load_state_dict(filtered_state_dict_gen)
    72	    decoder = decoder.eval().cuda()
    73	    fc_post_a = nn.Linear( 2048, 1024 )
    74	    fc_post_a.load_state_dict(filtered_state_dict_fc_post_a)
    75	    fc_post_a = fc_post_a.eval().cuda()
    76	    fc_prior = nn.Linear( 2048, 2048 )
    77	    fc_prior.load_state_dict(filtered_state_dict_fc_prior)
    78	    fc_prior = fc_prior.eval().cuda()
    79	    feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/w2v-bert-2.0")
    80
    81	    wav_dir = args.output_dir
    82	    os.makedirs(wav_dir, exist_ok=True)
    83
    84
    85
    86	    # wav_paths = glob(join(args.input_dir, '*.flac')) #
    87	    # both wav and flac and mp3
    88	    wav_paths = glob(os.path.join(args.input_dir, '**', '*.wav'), recursive=True)
    89	    flac_paths = glob(os.path.join(args.input_dir, '**', '*.flac'), recursive=True)
    90	    mp3_paths = glob(os.path.join(args.input_dir, '**', '*.mp3'), recursive=True)
    91
    92	    # 合并所有路径
    93	    wav_paths = wav_paths + flac_paths + mp3_paths
    94	    print(f'Found {len(wav_paths)} wavs in {args.input_dir}')
    95
    96	    st = time()
    97	    for wav_path in tqdm(wav_paths):
    98	        target_wav_path = join(wav_dir, basename(wav_path))
    99	        wav = librosa.load(wav_path, sr=sr)[0]
   100	        wav_cpu = torch.from_numpy(wav)
   101
   102
   103	        wav = wav_cpu.unsqueeze(0).cuda()
   104	        pad_for_wav = (320 - (wav.shape[1] % 320))
   105
   106	        wav = torch.nn.functional.pad(wav, (0, pad_for_wav))
   107
   108	        feat =  feature_extractor(F.pad(wav[0,:].cpu(), (160, 160)), sampling_rate=16000, return_tensors="pt") .data['input_features']
   109
   110	        feat = feat.cuda()
   111
   112	        with torch.no_grad():
   113	            vq_emb = encoder(wav.unsqueeze(1))
   114	            vq_emb = vq_emb.transpose(1, 2)
   115
   116
   117	            semantic_target = semantic_model(feat[:,  :,:])
   118
   119	            semantic_target = semantic_target.hidden_states[16]
   120
   121	            semantic_target = semantic_target.transpose(1, 2)
   122	            semantic_target = SemanticEncoder_module(semantic_target)
   123
   124
   125	            vq_emb = torch.cat([semantic_target, vq_emb], dim=1)
   126	            vq_emb =  fc_prior(vq_emb.transpose(1, 2)).transpose(1, 2)
   127
   128	            _, vq_code, _ = decoder(vq_emb, vq=True)  # vq_code here !!!!
   129
   130	            vq_post_emb = decoder.quantizer.get_output_from_indices(vq_code.transpose(1, 2))
   131	            vq_post_emb = vq_post_emb.transpose(1, 2)
   132	            vq_post_emb = fc_post_a(vq_post_emb.transpose(1,2)).transpose(1,2)
   133	            recon = decoder(vq_post_emb.transpose(1, 2), vq=False)[0].squeeze().detach().cpu().numpy()
   134	            # recon = decoder(decoder.vq2emb(vq_code.transpose(1,2)).transpose(1,2), vq=False).squeeze().detach().cpu().numpy()
   135	        sf.write(target_wav_path, recon, sr)
   136	    et = time()
   137	    print(f'Inference ends, time: {(et-st)/60:.2f} mins')
   138
   139
   140
   141
   142