Search the web thoroughly for any existing implementations or papers that use JEPA (Joint-Embedding Predictive Architecture) for TTS (text-to-speech) or audio generation. I need to find:

1. Any GitHub repos that implement JEPA for TTS or audio synthesis
2. Any papers that use JEPA specifically for generating audio (not just representation learning)
3. A-JEPA, Audio-JEPA, or any variant used for generation
4. How other JEPA audio models handle the autoregressive generation problem (exposure bias, embedding space mismatch)
5. Any TTS models that use JEPA-style latent prediction instead of traditional approaches

Search for:
- "JEPA TTS"
- "JEPA audio generation"
- "JEPA speech synthesis"
- "A-JEPA generation"
- "joint embedding predictive architecture speech"
- GitHub repos with JEPA + audio/TTS

Also look for how SoundStorm, VALL-E, or other modern TTS handle the similar problem of predicting in latent space and avoiding exposure bias during autoregressive generation.

Be thorough - check arxiv, GitHub, papers with code, blog posts.


I don't have web search capabilities in my current toolset. Let me check what tools are available to me.


ToolSearch

Question: web search browse internet


I have web search available. Let me run multiple searches in parallel to be thorough.


WebSearch

Question: JEPA TTS text-to-speech implementation


Question: JEPA audio generation speech synthesis 2024 2025


Question: A-JEPA Audio-JEPA audio generation paper


Question: "joint embedding predictive architecture" speech audio generation


Web search results for query: "JEPA TTS text-to-speech implementation"

Links: [{"title":"JEPA-v0: a self-supervised audio encoder for real-time speech translation | Pinch Research","url":"https://startpinch.com/research/en/jepa-encoder-translation/"},{"title":"Paper page - JEPA as a Neural Tokenizer: Learning Robust Speech Representations with Density Adaptive Attention","url":"https://huggingface.co/papers/2512.07168"},{"title":"Exploring JEPA for real-time speech translation | Hacker News","url":"https://news.ycombinator.com/item?id=47332883"},{"title":"The Best Open-Source Text-to-Speech Models in 2026","url":"https://www.bentoml.com/blog/exploring-the-world-of-open-source-text-to-speech-models"},{"title":"GitHub - coqui-ai/TTS: 🐸💬 - a deep learning toolkit for Text-to-Speech, battle-tested in research and production","url":"https://github.com/coqui-ai/TTS"},{"title":"Text-to-Speech Models – Hugging Face","url":"https://huggingface.co/models?pipeline_tag=text-to-speech"},{"title":"Free Text to Speech with Gemini and ChatGPT AI Voices","url":"https://www.naturalreaders.com/online/"},{"title":"Qwen3-TTS Family is Now Open Sourced: Voice Design, ...","url":"https://qwen.ai/blog?id=qwen3tts-0115"},{"title":"TTS Tool","url":"https://ttstool.com/"},{"title":"Free Text to Speech Online - TTSMaker","url":"https://ttsmaker.com/"}]

Based on the search results, here's what I found about JEPA TTS text-to-speech implementation:

## JEPA Framework Overview

JEPA-v0 is an attempt to build an audio encoder whose representations are rich enough to power real-time speech-to-speech translation. Rather than being a traditional TTS model (text-to-speech), JEPA is primarily used for speech encoding and translation tasks.

## JEPA for TTS Applications

While not exclusively a TTS system, the encoder learns semantically meaningful embeddings at 2.5 Hz without requiring low-level waveform loss, enabling robust self-supervised pretraining and cross-task adaptability (ASR, TTS, voice conversion).

## Technical Implementation

Recent implementations combine JEPA with advanced mechanisms:

A two-stage self-supervised framework combines the Joint-Embedding Predictive Architecture (JEPA) with a Density Adaptive Attention Mechanism (DAAM) for learning robust speech representations. Stage 1 uses JEPA with DAAM to learn semantic audio features via masked prediction in latent space, fully decoupled from waveform reconstruction. Stage 2 leverages these representations for efficient tokenization using Finite Scalar Quantization (FSQ) and a mixed-radix packing scheme, followed by high-fidelity waveform reconstruction with a HiFi-GAN decoder.

## Key Advantage

JEPA representations can drive speech-to-speech translation while preserving the speaker characteristics that text-based pipelines discard. This distinguishes JEPA-based approaches from traditional cascading ASR→MT→TTS pipelines.

REMINDER: You MUST include the sources above in your response to the user using markdown hyperlinks.


Web search results for query: "JEPA audio generation speech synthesis 2024 2025"

Links: [{"title":"Audio-JEPA: Self-Supervised Audio Learning","url":"https://www.emergentmind.com/topics/audio-jepa"},{"title":"[2507.02915] Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning","url":"https://arxiv.org/abs/2507.02915"},{"title":"WavJEPA: Semantic learning unlocks robust audio foundation models for raw waveforms | OpenReview","url":"https://openreview.net/forum?id=KHcSu8HSh9"},{"title":"JEPA-v0: a self-supervised audio encoder for real-time speech translation | Pinch Research","url":"https://startpinch.com/research/en/jepa-encoder-translation/"},{"title":"A-JEPA: Joint-Embedding Predictive Architecture Can Listen","url":"https://arxiv.org/html/2311.15830v3"},{"title":"VL-JEPA: JOINT EMBEDDING PREDICTIVE","url":"https://openreview.net/pdf?id=tjimrqc2BU"},{"title":"Audio and Speech Processing Jul 2025","url":"https://arxiv.org/list/eess.AS/2025-07?skip=150&show=1000"},{"title":"[2311.15830] A-JEPA: Joint-Embedding Predictive Architecture Can Listen","url":"https://arxiv.org/abs/2311.15830"},{"title":"[2512.07168] JEPA as a Neural Tokenizer: Learning Robust Speech Representations with Density Adaptive Attention","url":"https://arxiv.org/abs/2512.07168"},{"title":"Audio-JEPA: Joint-Embedding Predictive Architecture for ...","url":"https://arxiv.org/pdf/2507.02915"}]

Based on the search results, here's what I found about JEPA audio generation and speech synthesis in 2024-2025:

## Key JEPA Audio Models (2024-2025)

**Audio-JEPA** - A self-supervised learning framework that uses Vision Transformer to predict latent representations of masked spectrogram patches rather than reconstructing raw audio. It pre-trains on unlabeled AudioSet clips and shows comparable performance to wav2vec 2.0 and data2vec while using less than one-fifth of their training data.

**WavJEPA** - A waveform-based version of the Joint-Embedding Predictive Architecture that leverages high-level semantic representation learning and substantially outperforms state-of-the-art time-domain audio foundation models while requiring considerably fewer computational resources.

**JEPA as Neural Tokenizer** - A two-stage framework that uses JEPA with Density Adaptive Attention Mechanism to learn semantic audio features via masked prediction in latent space, followed by efficient tokenization using Finite Scalar Quantization and waveform reconstruction.

## Current Applications & Limitations

Generative use cases such as separation, enhancement, and synthesis are at a nascent stage, with direct application of JEPA embeddings being plausible but unproven for streaming generative models.

JEPA-v0 is designed as an audio encoder for speech-to-speech translation that can train on millions of audio samples without labels, learning phonetic patterns, speaker characteristics, emotional valence, and rhythmic structure.

REMINDER: You MUST include the sources above in your response to the user using markdown hyperlinks.


Web search results for query: "A-JEPA Audio-JEPA audio generation paper"

Links: [{"title":"[2507.02915] Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning","url":"https://arxiv.org/abs/2507.02915"},{"title":"Audio-JEPA: Joint-Embedding Predictive Architecture for ...","url":"https://arxiv.org/pdf/2507.02915"},{"title":"Audio-JEPA: Self-Supervised Audio Learning","url":"https://www.emergentmind.com/topics/audio-jepa"},{"title":"[2311.15830] A-JEPA: Joint-Embedding Predictive Architecture Can Listen","url":"https://arxiv.org/abs/2311.15830"},{"title":"A-JEPA: Joint-Embedding Predictive Architecture Can Listen","url":"https://arxiv.org/html/2311.15830v3"},{"title":"[2507.02915v1] Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning","url":"https://arxiv.org/abs/2507.02915v1"},{"title":"Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning | Request PDF","url":"https://www.researchgate.net/publication/393477016_Audio-JEPA_Joint-Embedding_Predictive_Architecture_for_Audio_Representation_Learning"},{"title":"[PDF] Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning | Semantic Scholar","url":"https://www.semanticscholar.org/paper/29bda609222b796e30490d89769169de2add1c5a"},{"title":"@Kseniase on Hugging Face: \"12 Types of JEPA Since Yann LeCun together with Randall Balestriero released…\"","url":"https://huggingface.co/posts/Kseniase/762937246285628"},{"title":"Paper page - JEPA as a Neural Tokenizer: Learning Robust Speech Representations with Density Adaptive Attention","url":"https://huggingface.co/papers/2512.07168"}]

Based on the search results, I found information about both A-JEPA and Audio-JEPA papers related to audio representation learning:

## A-JEPA (Audio-based Joint-Embedding Predictive Architecture)

The original A-JEPA paper demonstrates that the masked-modeling principle driving the success of large foundational vision models can be effectively applied to audio by making predictions in a latent space, introducing Audio-based Joint-Embedding Predictive Architecture as a simple extension method for self-supervised learning from the audio spectrum.

The A-JEPA model encodes visible audio spectrogram patches with a curriculum masking strategy via context encoder, and predicts the representations of regions sampled at well-designed locations. The model benefits from transferring random block masking into time-frequency aware masking in a curriculum manner, considering the complexity of highly correlated data in local time and frequency in audio spectrograms.

When built with Vision Transformers structure, A-JEPA is highly scalable and sets new state-of-the-art performance on multiple audio and speech classification tasks, outperforming other recent models that use externally supervised pre-training.

## Audio-JEPA (Ludovic Tuncay et al., 2025)

A more recent Audio-JEPA paper from June 2025 also builds on the JEPA paradigm: Audio-JEPA uses a simple Vision Transformer backbone to predict latent representations of masked spectrogram patches rather than reconstructing raw audio.

The implementation shows comparable performance to wav2vec 2.0 and data2vec while using less than one-fifth of their training data and with no hyper-parameter tuning.

REMINDER: You MUST include the sources above in your response to the user using markdown hyperlinks.


Web search results for query: ""joint embedding predictive architecture" speech audio generation"

Links: [{"title":"Audio-JEPA: Joint-Embedding Predictive Architecture for ...","url":"https://arxiv.org/pdf/2507.02915"},{"title":"[2507.02915] Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning","url":"https://arxiv.org/abs/2507.02915"},{"title":"[PDF] Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning | Semantic Scholar","url":"https://www.semanticscholar.org/paper/Audio-JEPA:-Joint-Embedding-Predictive-Architecture-Tuncay-Labb'e/29bda609222b796e30490d89769169de2add1c5a"},{"title":"A-JEPA: Joint-Embedding Predictive Architecture Can Listen","url":"https://arxiv.org/html/2311.15830v3"},{"title":"[2507.02915v1] Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning","url":"https://arxiv.org/abs/2507.02915v1"},{"title":"Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning | Request PDF","url":"https://www.researchgate.net/publication/393477016_Audio-JEPA_Joint-Embedding_Predictive_Architecture_for_Audio_Representation_Learning"},{"title":"Audio-JEPA: Joint-Embedding Predictive","url":"https://arxiv.org/pdf/2507.02915v1"},{"title":"STEM-JEPA: A JOINT-EMBEDDING PREDICTIVE ...","url":"https://zenodo.org/records/14877415/files/000070.pdf"},{"title":"JEP-KD: Joint-Embedding Predictive Architecture Based ...","url":"https://ieeexplore.ieee.org/iel8/8782710/10376377/10750407.pdf"},{"title":"Joint-Embedding Predictive Architecture (JEPA)","url":"https://www.emergentmind.com/topics/joint-embedding-predictive-architecture-jepa"}]

Based on the search results, here's what I found about joint embedding predictive architecture in the context of speech and audio generation:

## Overview

Joint-Embedding Predictive Architecture (JEPA) is a self-supervised learning framework that predicts latent representations of masked regions in high-level feature spaces. Rather than working with raw data, JEPA methods aim to predict higher-level representations rather than predicting low-level details of masked inputs.

## Audio Applications

There are several implementations of JEPA for audio:

**Audio-JEPA**: Audio-JEPA uses a simple Vision Transformer backbone to predict latent representations of masked spectrogram patches rather than reconstructing raw audio, and is pre-trained on unlabeled AudioSet clips (10s, 32kHz) with random patch masking on mel-spectrograms. The results show comparable performance to wav2vec 2.0 and data2vec while using less than one-fifth of their training data.

**A-JEPA**: A-JEPA is designed to reconstruct masked spectrogram patches from audio recordings within a latent space and achieves superior performance on multiple audio and speech classification tasks.

**Stem-JEPA**: Stem-JEPA is a novel Joint-Embedding Predictive Architecture trained on a multi-track dataset using a self-supervised learning approach, with an encoder and predictor jointly trained to predict embeddings of compatible stems, enabling use in stem compatibility estimation and downstream tasks like genre or key estimation.

## Key Principles

The image-based I-JEPA demonstrated that a ViT can learn excellent representations by predicting the latent representations of masked image patches as opposed to generating pixels, and by operating in the feature space, I-JEPA forces the model to capture abstract semantic information and ignore minute pixel-level differences.

REMINDER: You MUST include the sources above in your response to the user using markdown hyperlinks.


Good, now let me search for more specific topics.


Question: JEPA TTS github repo implementation code


Question: SoundStorm VALL-E latent space prediction exposure bias autoregressive generation


Question: JEPA generative model autoregressive generation decoder latent prediction


Question: WavJEPA paper speech generation waveform


Web search results for query: "JEPA TTS github repo implementation code"

Links: [{"title":"GitHub - Sennadir/TS_JEPA: Implementation of \"TS-JEPA\"","url":"https://github.com/Sennadir/TS_JEPA"},{"title":"GitHub - facebookresearch/jepa: PyTorch code and models for V-JEPA self-supervised learning from video. · GitHub","url":"https://github.com/facebookresearch/jepa"},{"title":"GitHub - LumenPallidium/jepa: Experiments in Joint Embedding Predictive Architectures (JEPAs). · GitHub","url":"https://github.com/LumenPallidium/jepa"},{"title":"GitHub - jose-melo/t-jepa: T-JEPA official repository · GitHub","url":"https://github.com/jose-melo/t-jepa"},{"title":"GitHub - gaasher/I-JEPA: Implementation of I-JEPA from \"Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture\"","url":"https://github.com/gaasher/I-JEPA"},{"title":"GitHub - facebookresearch/vjepa2: PyTorch code and models for VJEPA2 self-supervised learning from video. · GitHub","url":"https://github.com/facebookresearch/vjepa2"},{"title":"GitHub - geriskenderi/graph-jepa: Official code for the paper Graph-level Representation Learning with Joint-Embedding Predictive Architectures published in Transactions on Machine Learning Research (TMLR) · GitHub","url":"https://github.com/geriskenderi/graph-jepa"},{"title":"GitHub - Team-GPT-2/v-jepa: PyTorch code and models for V-JEPA self-supervised learning from video. · GitHub","url":"https://github.com/Team-GPT-2/v-jepa"},{"title":"GitHub - facebookresearch/ijepa: Official codebase for I-JEPA, the Image-based Joint-Embedding Predictive Architecture. First outlined in the CVPR paper, \"Self-supervised learning from images with a joint-embedding predictive architecture.\" · GitHub","url":"https://github.com/facebookresearch/ijepa"},{"title":"GitHub - facebookresearch/eb_jepa: An open source library designed to provide community examples of Joint Embedding Predictive Architectures (JEPAs). It contains code and examples for learning representations from images, video, and action-conditioned video, as well as planning using JEPA-based models. · GitHub","url":"https://github.com/facebookresearch/eb_jepa"}]

Based on the search results, I found several JEPA-related GitHub repositories with implementation code. However, I notice your query specifically mentions "JEPA TTS" (Text-To-Speech), which is not explicitly mentioned in the results. The search returned various JEPA implementations for different domains (images, video, time series, graphs, tabular data), but not specifically a TTS/speech synthesis version.

Here are the most relevant JEPA repositories found:

**Video & General JEPA:**
- facebookresearch/jepa - Official PyTorch codebase for the video joint-embedding predictive architecture, V-JEPA, a method for self-supervised learning of visual representations from video
- facebookresearch/vjepa2 - Official Pytorch codebase for V-JEPA 2, V-JEPA 2-AC, V-JEPA 2.1

**Image-based JEPA:**
- facebookresearch/ijepa - Official codebase for I-JEPA, the Image-based Joint-Embedding Predictive Architecture, first outlined in the CVPR paper "Self-supervised learning from images with a joint-embedding predictive architecture"

**Other Domain-Specific Variants:**
- Sennadir/TS_JEPA - Implementation of TS-JEPA which was proposed on "Joint Embedding go Temporal", at the NeurIPS Workshop on Time Series in the Age of Large Models
- jose-melo/t-jepa - T-JEPA that predicts the latent representation of one subset of features from the latent representation of another subset within the same sample

If you're looking specifically for JEPA applied to Text-To-Speech, that particular variant doesn't appear in these results. You may need to search more specifically or check if there's a specific research paper you're referring to.

REMINDER: You MUST include the sources above in your response to the user using markdown hyperlinks.


Web search results for query: "SoundStorm VALL-E latent space prediction exposure bias autoregressive generation"

Links: [{"title":"SoundStorm: Efficient Parallel Audio Generation | Request PDF","url":"https://www.researchgate.net/publication/370814621_SoundStorm_Efficient_Parallel_Audio_Generation"},{"title":"[PDF] SoundStorm: Efficient Parallel Audio Generation | Semantic Scholar","url":"https://www.semanticscholar.org/paper/2183c88e9056e931b07d48f1dc44360785952073"},{"title":"SoundStorm: Efficient Parallel Audio Generation - Arxiv:2305.09636 | PDF","url":"https://www.scribd.com/document/652620168/SoundStorm-Efficient-Parallel-Audio-Generation-arXiv-2305-09636"},{"title":"Published as a conference paper at ICLR 2025 CONTINUOUS AUTOREGRESSIVE MODELING","url":"https://proceedings.iclr.cc/paper_files/paper/2025/file/7d90c28e7820709792d969211815a2b3-Paper-Conference.pdf"},{"title":"SoundStorm: Efﬁcient Parallel Audio Generation","url":"https://arxiv.org/pdf/2305.09636"},{"title":"Lessons From the Autoregressive/Non- autoregressive Battle in Speech Synthesis","url":"https://tan-xu.github.io/AR-NAR-TTS.pdf"},{"title":"MULTIMODAL LATENT LANGUAGE MODELING WITH ...","url":"https://openreview.net/pdf/3fe098b8c1e0a577a294a33f42791d349ce14865.pdf"},{"title":"SoundStorm: Efficient Parallel Audio Generation | OpenReview","url":"https://openreview.net/forum?id=KknWbD5j95"},{"title":"[2305.09636] SoundStorm: Efficient Parallel Audio Generation","url":"https://arxiv.org/abs/2305.09636"},{"title":"SoundStorm: Efficient parallel audio generation","url":"https://research.google/blog/soundstorm-efficient-parallel-audio-generation/"}]

Based on the search results, here's what I found about SoundStorm, VALL-E, and the related concepts you queried:

## SoundStorm Overview

SoundStorm is a model for efficient, non-autoregressive audio generation. It receives as input the semantic tokens of AudioLM, and relies on bidirectional attention and confidence-based parallel decoding to generate the tokens of a neural audio codec.

## Comparison with Autoregressive Generation

Compared to the autoregressive generation approach of AudioLM, SoundStorm produces audio of the same quality and with higher consistency in voice and acoustic conditions, while being two orders of magnitude faster. SoundStorm generates 30 seconds of audio in 0.5 seconds on a TPU-v4.

## VALL-E's Approach to Autoregressive Generation

VALL-E relies on a hybrid approach, where the tokens corresponding to the first RVQ level are predicted autoregressively, and the subsequent levels are produced non-autoregressively. During inference, tokens starting from the second level of the RVQ are produced iteratively, performing greedy decoding (choosing the most likely tokens) level-by-level.

## Latent Space and Codec Modeling

Modeling the tokens of a neural audio codec unlocked rapid progress in audio generation, producing high-quality, coherent audio. However, this approach requires modeling long sequences, thus affecting the training and inference costs.

The search results indicate that SoundStorm addresses the computational challenges of audio codec token generation through parallel decoding rather than sequential autoregressive prediction, providing significant efficiency gains over traditional autoregressive models like AudioLM.

REMINDER: You MUST include the sources above in your response to the user using markdown hyperlinks.


Web search results for query: "JEPA generative model autoregressive generation decoder latent prediction"

Links: [{"title":"JEPA-Reasoner: Generative Latent Space Reasoner | OpenReview","url":"https://openreview.net/forum?id=7ruA2rXG42"},{"title":"JEPA-Reasoner: Decoupling Latent Reasoning from Token Generation","url":"https://arxiv.org/pdf/2512.19171"},{"title":"VL-JEPA: Joint Embedding Predictive Architecture for Vision-language","url":"https://arxiv.org/pdf/2512.10942"},{"title":"The Anatomy of JEPA: The Architecture Behind embedded Predictive Representation Learning | by Tyler Frink | Medium","url":"https://medium.com/@frinktyler1445/the-anatomy-of-jepa-the-architecture-behind-embedded-predictive-representation-learning-994bfa0bffe0"},{"title":"JEPA-Reasoner: Decoupling Latent Reasoning from Token Generation | alphaXiv","url":"https://www.alphaxiv.org/overview/2512.19171v1"},{"title":"Rohit Bandaru | Deep Dive into Yann LeCun’s JEPA","url":"https://rohitbandaru.github.io/blog/JEPA-Deep-Dive/"},{"title":"Meta’s new VL-JEPA model shifts from generating tokens to predicting concepts","url":"https://bdtechtalks.substack.com/p/metas-new-vl-jepa-model-shifts-from"},{"title":"Preprint LLM-JEPA: LARGE LANGUAGE MODELS MEET JOINT","url":"https://arxiv.org/pdf/2509.14252"},{"title":"VL-JEPA: JOINT EMBEDDING PREDICTIVE","url":"https://openreview.net/pdf?id=tjimrqc2BU"},{"title":"ACT-JEPA: Novel Joint-Embedding Predictive Architecture for Efficient Policy Representation Learning","url":"https://arxiv.org/html/2501.14622v4"}]

Based on the search results, here's what I found about JEPA generative models with autoregressive generation, decoder, and latent prediction:

## Key Findings

**JEPA-Reasoner: Generative JEPA with Autoregressive Generation**

JEPA-Reasoner is a decoupled, generative JEPA model that autoregressively reasons in continuous latent space, offering both superior robustness to compounding error and potential for multi-threaded reasoning. While Joint-Embedding Predictive Architecture (JEPA) has emerged as a powerful architecture for learning rich latent representations, it fundamentally lacks generative abilities.

**Architecture and Latent Prediction**

JEPA-Reasoner adapts the traditional JEPA framework for autoregressive generation in latent space. Rather than predicting missing information to fill gaps, it generates sequences of abstract latent states representing reasoning chains. The key innovation is that predictions remain in the continuous latent space without immediate projection to vocabulary probabilities. Each predicted latent matrix becomes input for the next reasoning step, enabling sustained high-level reasoning without committing to discrete tokens.

**Decoder Component (Talker)**

JEPA-Reasoner is augmented with a separate action-taker model, Talker, to produce human-readable sentences. The decoders generate tokens autoregressively conditioned on previous tokens, with continuous latent guidance from the output of encoders.

**Performance Benefits**

Decoupling latent space reasoning and token generation enables JEPA-Reasoner to produce mixed latent vectors that might lay the foundation for multi-threaded reasoning, while performing autoregressive generation with superior robustness to compounding error.

REMINDER: You MUST include the sources above in your response to the user using markdown hyperlinks.


Web search results for query: "WavJEPA paper speech generation waveform"

Links: [{"title":"WAVJEPA: SEMANTIC LEARNING UNLOCKS ROBUST","url":"https://www.arxiv.org/pdf/2509.23238v2"},{"title":"GitHub - labhamlet/wavjepa: This is the official codebase for WavJEPA. Time-domain audio foundation model for holistic downstream tasks. \"Self-supervised learning from raw waveforms unlock robust audio foundation models\".\" · GitHub","url":"https://github.com/labhamlet/wavjepa"},{"title":"WavJEPA: Semantic learning unlocks robust audio foundation models for raw waveforms | OpenReview","url":"https://openreview.net/forum?id=KHcSu8HSh9"},{"title":"WavJEPA: Semantic learning unlocks robust audio foundation models for raw waveforms","url":"https://arxiv.org/html/2509.23238"},{"title":"[2509.23238] WavJEPA: Semantic learning unlocks robust audio foundation models for raw waveforms","url":"https://arxiv.org/abs/2509.23238"},{"title":"(PDF) WavJEPA: Semantic learning unlocks robust audio foundation models for raw waveforms","url":"https://www.researchgate.net/publication/395969231_WavJEPA_Semantic_learning_unlocks_robust_audio_foundation_models_for_raw_waveforms"},{"title":"WavJEPA: Audio & Jet Physics Analysis","url":"https://www.emergentmind.com/topics/wavjepa"},{"title":"labhamlet/wavjepa-base · Hugging Face","url":"https://huggingface.co/labhamlet/wavjepa-base"},{"title":"Audio-JEPA: Self-Supervised Audio Learning","url":"https://www.emergentmind.com/topics/audio-jepa"},{"title":"Audio-JEPA: Joint-Embedding Predictive Architecture for ...","url":"https://arxiv.org/pdf/2507.02915"}]

Based on the search results, here's information about the WavJEPA paper:

## Overview

WavJEPA is a waveform-based version of the Joint-Embedding Predictive Architecture that leverages high-level semantic representation learning to tackle the shortcomings of representation learning at the speech unit or token level.

## Key Innovation

Learning audio representations from raw waveforms overcomes key limitations of spectrogram-based audio representation learning, such as the long latency of spectrogram computation and the loss of phase information. Yet, while self-supervised speech representation learning from raw waveforms has been remarkably successful, these approaches have not achieved similar feats for general-purpose audio representation learning from waveforms.

## Architecture

The WavJEPA framework comprises a waveform encoder, context encoder, target encoder and a predictor. WavJEPA's objective is to predict latent representation of various targets blocks based on a single context block extracted from the same sound wave. The waveform encoder uses the feature encoder of Wav2Vec 2.0, which is composed of stacked temporal convolution layers. Similar to the original I-JEPA architecture, a Vision Transformer (ViT) is used for the target encoder, context encoder and predictor.

## Performance

This approach substantially outperforms state-of-the-art time-domain audio foundation models across a wide variety of downstream benchmark tasks, while requiring considerably fewer computational resources.

## Extensions

To overcome the performance drop that time-domain models typically exhibit in noisy and reverberant real-world acoustic environments, WavJEPA-Nat is a multi-channel extension of the WavJEPA architecture trained on simulated naturalistic scenes. WavJEPA-Nat is highly robust to reverberation and noise.

REMINDER: You MUST include the sources above in your response to the user using markdown hyperlinks.


Excellent finds. Let me dig deeper into the specific areas that matter most for your project.


Question: JEPA-Reasoner latent space autoregressive compounding error exposure bias


Question: "JEPA as a Neural Tokenizer" speech TTS generation decoder 2024


Question: TTS latent space prediction non-autoregressive exposure bias MELLE continuous token


Question: JEPA-v0 Pinch Research speech-to-speech encoder generation 2025 2026


Web search results for query: ""JEPA as a Neural Tokenizer" speech TTS generation decoder 2024"

Links: [{"title":"[2512.07168] JEPA as a Neural Tokenizer: Learning Robust Speech Representations with Density Adaptive Attention","url":"https://arxiv.org/abs/2512.07168"},{"title":"JEPA as a Neural Tokenizer: Learning Robust Speech Representations with Density Adaptive Attention","url":"https://arxiv.org/html/2512.07168v1"},{"title":"Paper page - JEPA as a Neural Tokenizer: Learning Robust Speech Representations with Density Adaptive Attention","url":"https://huggingface.co/papers/2512.07168"},{"title":"JEPA as a Neural Tokenizer: Learning Robust Speech ...","url":"https://arxiv.org/pdf/2512.07168"},{"title":"JEPA as a Neural Tokenizer: Learning Robust Speech Representations with Density Adaptive Attention | alphaXiv","url":"https://www.alphaxiv.org/overview/2512.07168"},{"title":"JEPA as a Neural Tokenizer: Learning Robust Speech Representations with Density Adaptive Attention","url":"https://arxiviq.substack.com/p/jepa-as-a-neural-tokenizer-learning"},{"title":"Divide and Conquer: JEPA as a Neural Speech Tokenizer (and Why That’s a Big Deal) | by ABV | Dec, 2025 | Medium","url":"https://abvcreative.medium.com/divide-and-conquer-jepa-as-a-neural-speech-tokenizer-and-why-thats-a-big-deal-b7ae829e6eba"},{"title":"JEPA as a Neural Tokenizer: Learning Robust Speech Representations with Density Adaptive Attention | Cool Papers - Immersive Paper Discovery","url":"https://papers.cool/arxiv/2512.07168"},{"title":"JEPA as a Neural Tokenizer: Learning Robust Speech Representations with Density Adaptive Attention | UniReps Blog","url":"https://unireps.org/blog/2025/jepa-daam_fsqvae/"},{"title":"JEPA as a Neural Tokenizer: Learning Robust Speech Representations with Density Adaptive Attention - Paper Details","url":"https://www.chatpaper.ai/dashboard/paper/f2ae1c74-770d-4120-bf57-f0bbbde1d519"}]

## Search Results: JEPA as a Neural Tokenizer

I've found comprehensive information about the paper on JEPA as a Neural Tokenizer. Here's what the research presents:

### Overview

The work introduces a two-stage self-supervised framework that combines the Joint-Embedding Predictive Architecture (JEPA) with a Density Adaptive Attention Mechanism (DAAM) for learning robust speech representations.

### Two-Stage Architecture

**Stage 1:** Uses JEPA with DAAM to learn semantic audio features via masked prediction in latent space, fully decoupled from waveform reconstruction.

**Stage 2:** Leverages these representations for efficient tokenization using Finite Scalar Quantization (FSQ) and a mixed-radix packing scheme, followed by high-fidelity waveform reconstruction with a HiFi-GAN decoder.

### Key Technical Features

By integrating Gaussian mixture-based density-adaptive gating into the JEPA encoder, the model performs adaptive temporal feature selection and discovers hierarchical speech structure at a low frame rate of 2.5 Hz.

The resulting tokens (47.5 tokens/sec) provide a reversible, highly compressed, and language-model-friendly representation that is competitive with, and often more efficient than, existing neural audio codecs.

### Advantages of the Approach

The encoder optimizes for semantic content rather than low-level waveform details, and the same encoder can support multiple downstream tasks (text-to-speech, voice conversion, automatic speech recognition, etc.).

### Practical Impact

This work represents a significant step toward efficient speech tokenization for LLM integration, with the vocabulary size of 16,384 tokens aligning well with subword vocabularies used in natural language processing, facilitating direct integration with existing Transformer architectures.

REMINDER: You MUST include the sources above in your response to the user using markdown hyperlinks.


Web search results for query: "JEPA-Reasoner latent space autoregressive compounding error exposure bias"

Links: [{"title":"[2512.19171] JEPA-Reasoner: Decoupling Latent Reasoning from Token Generation","url":"https://arxiv.org/abs/2512.19171"},{"title":"JEPA-Reasoner: Generative Latent Space Reasoner | OpenReview","url":"https://openreview.net/forum?id=7ruA2rXG42"},{"title":"JEPA-Reasoner: Decoupling Latent Reasoning from Token Generation","url":"https://arxiv.org/html/2512.19171v1"},{"title":"JEPA-Reasoner: Decoupling Latent Reasoning from Token Generation | alphaXiv","url":"https://www.alphaxiv.org/overview/2512.19171v1"},{"title":"JEPA-REASONER: GENERATIVE LATENT SPACE REA- SONER","url":"https://openreview.net/pdf/5ae56ae4791094bd38b6d094aef3c5834c81dcad.pdf"},{"title":"000 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019","url":"https://openreview.net/pdf?id=7ruA2rXG42"},{"title":"Alternatives To Next Token Prediction In Text Generation - A Survey","url":"https://arxiv.org/pdf/2509.24435"},{"title":"[Literature Review] JEPA-Reasoner: Decoupling Latent Reasoning from Token Generation","url":"https://www.themoonlight.io/en/review/jepa-reasoner-decoupling-latent-reasoning-from-token-generation"},{"title":"Discrete JEPA: Learning Discrete Token Representations without Reconstruction","url":"https://openreview.net/pdf?id=91kfdrEafN"},{"title":"Exposure Bias in Machine Learning","url":"https://www.emergentmind.com/topics/exposure-bias"}]

Based on the search results, here's what I found about JEPA-Reasoner and its relationship to latent space, autoregressive generation, compounding errors, and exposure bias:

## JEPA-Reasoner Overview

JEPA-Reasoner is a novel architectural paradigm that decouples reasoning from token generation using a Joint-Embedding Predictive Architecture (JEPA) for pure latent-space reasoning and a separate Talker module for linguistic reconstruction.

## The Problem It Addresses

Current autoregressive language models couple high-level reasoning and low-level token generation into a single sequential process, making the reasoning trajectory vulnerable to compounding expression errors. More specifically, latent space reasoning attempts for Transformer models like COCONUT improve performance, but they ultimately rely on token-by-token generation, which still accumulates compounding error.

## How It Mitigates Compounding Errors

By isolating the reasoning engine from the discrete token-sampling process, the architecture enables Error Containment, where token-level failures cannot propagate into the latent reasoning chain. By operating in a normalized latent space and offloading token generation to the Talker module, subsequent reasoning outputs do not condition on previous decisions, thus mitigating error accumulation in the autoregressive process.

## Key Innovation

Decoupling latent space reasoning and token generation enables JEPA-Reasoner to produce mixed latent vectors that might lay the foundation for multi-threaded reasoning, while performing autoregressive generation with superior robustness to compounding error.

REMINDER: You MUST include the sources above in your response to the user using markdown hyperlinks.


Web search results for query: "TTS latent space prediction non-autoregressive exposure bias MELLE continuous token"

Links: [{"title":"Continuous-Token Diffusion for Speaker-Referenced TTS in Multimodal LLMs","url":"https://arxiv.org/pdf/2510.12995"},{"title":"F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech ...","url":"https://aclanthology.org/2025.acl-long.313.pdf"},{"title":"InstructTTS: Modelling Expressive TTS in Discrete Latent ...","url":"https://arxiv.org/pdf/2301.13662"},{"title":"BRIDGING THE GAP BETWEEN TRAINING AND INFERENCE IN LM-BASED TTS MODELS","url":"https://arxiv.org/pdf/2509.17021"},{"title":"CLEAR: Continuous Latent Autoregressive Modeling for High-quality and Low-latency Speech Synthesis","url":"https://arxiv.org/html/2508.19098"},{"title":"KALL-E: Autoregressive Speech Synthesis with Next-Distribution Prediction","url":"https://arxiv.org/html/2412.16846v2"},{"title":"Speech Synthesis From Continuous Features Using Per-Token Latent Diffusion","url":"https://arxiv.org/html/2410.16048"},{"title":"IndexTTS 2.5 Technical Report","url":"https://arxiv.org/pdf/2601.03888"},{"title":"M3-TTS: Multi-modal DiT Alignment & Mel-latent for Zero-shot High-fidelity Speech Synthesis","url":"https://arxiv.org/html/2512.04720v1"},{"title":"MULTIMODAL LATENT LANGUAGE MODELING WITH ...","url":"https://openreview.net/pdf/3fe098b8c1e0a577a294a33f42791d349ce14865.pdf"}]

The search results reveal several relevant research papers on text-to-speech (TTS) systems that deal with the concepts you queried. Here are the key findings:

## Main Research Areas

**Continuous Token Modeling in TTS:**

Recent approaches use dual-head multimodal language models with diffusion heads for frame-level autoregressive generation, enabling continuous speech synthesis with natural and high-fidelity quality.

**Exposure Bias and Non-Autoregressive Approaches:**

Exposure bias in autoregressive models is mitigated by introducing masked training schemes inspired by audio generation, which addresses exposure bias in autoregressive decoding.

**MELLE - Continuous Mel-Spectrogram Prediction:**

MELLE uses regression-based loss functions (mean squared error) based on simplified distributional assumptions, which often struggle to capture complex speech patterns, leading to blurry and overly uniform predictions. More specifically, MELLE predicts Mel spectrograms autoregressively using a Gaussian sampling module and parameterizes the next frame using a Gaussian distribution, which restricts it to learning only unimodal distributions.

**Latent Space Prediction Alternatives:**

KALL-E introduces an approach that predicts the next distribution of continuous speech frames, directly modeling continuous speech distribution conditioned on text and eliminating the need for any diffusion-based components.

These papers address the interplay between latent space modeling, exposure bias mitigation, and continuous vs. discrete token representations in modern TTS systems.

REMINDER: You MUST include the sources above in your response to the user using markdown hyperlinks.


Web search results for query: "JEPA-v0 Pinch Research speech-to-speech encoder generation 2025 2026"

Links: [{"title":"JEPA-v0: a self-supervised audio encoder for real-time speech translation | Pinch Research","url":"https://startpinch.com/research/en/jepa-encoder-translation/"},{"title":"Published as a conference paper at ICLR 2026","url":"https://openreview.net/pdf?id=tjimrqc2BU"},{"title":"VL-JEPA: Joint Embedding Predictive Architecture for Vision-language | OpenReview","url":"https://openreview.net/forum?id=tjimrqc2BU"},{"title":"VL-JEPA: JOINT EMBEDDING PREDICTIVE","url":"https://openreview.net/pdf/922ead06ab504b57bcd30999e66303eae715d919.pdf"},{"title":"JEPA as a Neural Tokenizer: Learning Robust Speech Representations with Density Adaptive Attention | UniReps Blog","url":"https://unireps.org/blog/2025/jepa-daam_fsqvae/"},{"title":"VL-JEPA: Joint Embedding Predictive Architecture for Vision-language","url":"https://arxiv.org/pdf/2512.10942"},{"title":"[2512.10942] VL-JEPA: Joint Embedding Predictive Architecture for Vision-language","url":"https://arxiv.org/abs/2512.10942"},{"title":"What VL-JEPA Could Revolutionize in Multimodal Intelligence | by Inference Weekly | Medium","url":"https://medium.com/@harshit.sinha0910/what-vl-jepa-could-revolutionize-in-multimodal-intelligence-a1c2105dc62e"},{"title":"[2512.07168] JEPA as a Neural Tokenizer: Learning Robust Speech Representations with Density Adaptive Attention","url":"https://arxiv.org/abs/2512.07168"},{"title":"Meta’s new VL-JEPA model shifts from generating tokens to predicting concepts","url":"https://bdtechtalks.substack.com/p/metas-new-vl-jepa-model-shifts-from"}]

Based on the web search results, here's what I found about your query:

## JEPA-v0 and Pinch Research

JEPA-v0 is an attempt to build an audio encoder whose representations are rich enough to power real-time speech-to-speech translation, developed by Pinch Research.

To build speech-to-speech translation that preserves voice, emotion, and timing, you need an encoder that captures more than just words. The system addresses a fundamental limitation of traditional approaches: The cascade pipeline of ASR → MT → TTS treats audio as a container for text. It cracks open the container, throws the container away, translates the text inside, and builds a new container from scratch.

## Technical Implementation

The JEPA-v0 uses a self-supervised learning approach with specific mechanisms to prevent representation collapse. The stop-gradient on the target allows the target encoder to slowly follow the context encoder via EMA. Because the target changes slowly while the context encoder updates rapidly, there's always a gap between them. The context encoder is always chasing a moving target it can never fully reach, which keeps it from settling into a trivial solution.

## Research Context and Timeline

The research references work from both 2025 and 2026. Ponce et al., "Dual Perspectives on Non-Contrastive Self-Supervised Learning" (ICLR 2026) provides theoretical grounding, and Zhang et al., "X-ARES: A Comprehensive Framework for Assessing Audio Encoder Performance" (2025) provides an evaluation framework for the technology.

REMINDER: You MUST include the sources above in your response to the user using markdown hyperlinks.


Question: CLEAR continuous latent autoregressive TTS speech synthesis exposure bias 2025


Question: "bridging gap training inference" LM-based TTS exposure bias scheduled sampling


Question: github A-JEPA audio JEPA speech code implementation repository


Question: KALL-E next distribution prediction TTS continuous autoregressive latent


Web search results for query: "github A-JEPA audio JEPA speech code implementation repository"

Links: [{"title":"GitHub - SonyCSLParis/audio-representations: JEPAs for audio representation learning · GitHub","url":"https://github.com/SonyCSLParis/audio-representations"},{"title":"GitHub - Sennadir/TS_JEPA: Implementation of \"TS-JEPA\"","url":"https://github.com/Sennadir/TS_JEPA"},{"title":"GitHub - gaasher/I-JEPA: Implementation of I-JEPA from \"Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture\"","url":"https://github.com/gaasher/I-JEPA"},{"title":"GitHub - LumenPallidium/jepa: Experiments in Joint Embedding Predictive Architectures (JEPAs). · GitHub","url":"https://github.com/LumenPallidium/jepa"},{"title":"GitHub - D-JEPA/d-jepa.github.io · GitHub","url":"https://github.com/D-JEPA/d-jepa.github.io"},{"title":"[2507.02915] Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning","url":"https://arxiv.org/abs/2507.02915"},{"title":"GitHub - facebookresearch/eb_jepa: An open source library designed to provide community examples of Joint Embedding Predictive Architectures (JEPAs). It contains code and examples for learning representations from images, video, and action-conditioned video, as well as planning using JEPA-based models. · GitHub","url":"https://github.com/facebookresearch/eb_jepa"},{"title":"Audio-JEPA: Joint-Embedding Predictive Architecture for ...","url":"https://arxiv.org/pdf/2507.02915"},{"title":"jepa · PyPI","url":"https://pypi.org/project/jepa/"},{"title":"GitHub - SonyCSLParis/Stem-JEPA: Joint Embedding Predictive Architecture for Musical Stem Compatibility Estimation","url":"https://github.com/SonyCSLParis/Stem-JEPA"}]

Based on the search results, here are several relevant GitHub repositories related to A-JEPA and audio JEPA implementations:

## Primary Audio JEPA Implementation

Audio-JEPA is a Joint-Embedding Predictive Architecture tailored specifically for audio data. It uses a simple Vision Transformer backbone to predict latent representations of masked spectrogram patches rather than reconstructing raw audio, pre-training on unlabeled AudioSet clips (10s, 32kHz) with random patch masking on mel-spectrograms. All code and pretrained checkpoints will be released on GitHub.

## Related Audio JEPA Repositories

1. **SonyCSLParis/audio-representations**: This repository contains PyTorch code for investigating design choices in Joint-Embedding Predictive Architectures for general audio representation learning, presented at the SASB workshop at ICASSP 2024.

2. **SonyCSLParis/Stem-JEPA**: This repository contains the code of the Stem-JEPA paper, accepted at ISMIR 2024, which applies JEPA to musical stem compatibility estimation.

3. **facebookresearch/eb_jepa**: An open source library designed to provide community examples of Joint Embedding Predictive Architectures, containing code and examples for learning representations from images, video, and action-conditioned video.

These repositories provide implementations of JEPA architectures applied to audio and related domains with code available for training and evaluation.

REMINDER: You MUST include the sources above in your response to the user using markdown hyperlinks.


Web search results for query: "CLEAR continuous latent autoregressive TTS speech synthesis exposure bias 2025"

Links: [{"title":"[2508.19098] CLEAR: Continuous Latent Autoregressive Modeling for High-quality and Low-latency Speech Synthesis","url":"https://arxiv.org/abs/2508.19098"},{"title":"CLEAR: Continuous Latent Autoregressive Modeling for High-quality and Low-latency Speech Synthesis","url":"https://arxiv.org/html/2508.19098"},{"title":"Continuous-Token Diffusion for Speaker-Referenced TTS in Multimodal LLMs","url":"https://arxiv.org/pdf/2510.12995"},{"title":"F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech ...","url":"https://aclanthology.org/2025.acl-long.313.pdf"},{"title":"CLEAR: Continuous Latent Autoregressive Modeling for ...","url":"https://arxiv.org/pdf/2508.19098"},{"title":"CLEAR: Continuous Latent Autoregressive Model","url":"https://www.emergentmind.com/topics/continuous-latent-autoregressive-model-clear"},{"title":"[PDF] CLEAR: Continuous Latent Autoregressive Modeling for High-quality and Low-latency Speech Synthesis | Semantic Scholar","url":"https://www.semanticscholar.org/paper/CLEAR:-Continuous-Latent-Autoregressive-Modeling-Wu-Deng/bd0aebc0abab4ba927c8733edeffb069163e8d8d"},{"title":"KALL-E: Autoregressive Speech Synthesis with Next-Distribution Prediction","url":"https://arxiv.org/html/2412.16846v2"},{"title":"ARCHI-TTS: A flow-matching-based Text-to-Speech Model with Self-supervised Semantic Aligner and Accelerated Inference","url":"https://arxiv.org/html/2602.05207"},{"title":"IndexTTS 2.5 Technical Report","url":"https://arxiv.org/pdf/2601.03888"}]

Based on the search results, here's information about the CLEAR model and related continuous latent autoregressive speech synthesis research:

## CLEAR: Continuous Latent Autoregressive Modeling

CLEAR is a unified zero-shot TTS framework that directly models continuous audio representations, introducing an enhanced variational autoencoder with shortcut connections that achieves a high compression ratio to map waveforms into compact continuous latents.

### Key Technical Approach

A lightweight MLP-based rectified flow head that operates independently for each hidden state is presented to model the continuous latent probability distribution, and trained jointly with the AR model within a single-stage framework.

### Performance

Compared to state-of-the-art TTS models, CLEAR delivers competitive performance in robustness, speaker similarity and naturalness, while offering a lower real-time factor (RTF), achieving SOTA results on the LibriSpeech test-clean dataset with a word error rate of 1.88% and an RTF of 0.29. CLEAR facilitates streaming speech synthesis with a first-frame delay of 96ms, while maintaining high-quality speech synthesis.

### Problem Addressed

Conventional AR-based TTS systems relying on discrete audio tokens face the challenge of lossy compression during tokenization, requiring longer discrete token sequences to capture the same information as continuous ones, which adds inference latency and complicates AR modeling.

### Related Work on Exposure Bias

The search results also show related work addressing exposure bias in autoregressive speech synthesis: Masked training is employed to address exposure bias in autoregressive decoding, where by training under corrupted contexts, the model is encouraged to handle imperfect histories more robustly, thereby mitigating exposure bias and improving stability in long-form speech synthesis.

REMINDER: You MUST include the sources above in your response to the user using markdown hyperlinks.


Web search results for query: "KALL-E next distribution prediction TTS continuous autoregressive latent"

Links: [{"title":"[2412.16846] KALL-E:Autoregressive Speech Synthesis with Next-Distribution Prediction","url":"https://arxiv.org/abs/2412.16846"},{"title":"KALL-E: Autoregressive Speech Synthesis with Next-Distribution Prediction","url":"https://arxiv.org/html/2412.16846v2"},{"title":"Autoregressive Speech Synthesis with Next-Distribution Prediction","url":"https://arxiv.org/html/2412.16846v1"},{"title":"[Literature Review] Autoregressive Speech Synthesis with Next-Distribution Prediction","url":"https://www.themoonlight.io/en/review/autoregressive-speech-synthesis-with-next-distribution-prediction"},{"title":"Autoregressive Speech Synthesis with Next-Distribution Prediction | Request PDF","url":"https://www.researchgate.net/publication/387350898_Autoregressive_Speech_Synthesis_with_Next-Distribution_Prediction"},{"title":"Autoregressive Speech Synthesis with Next-Distribution Prediction","url":"https://zxf-icpc.github.io/kalle/"},{"title":"Paper page - Autoregressive Speech Synthesis with Next-Distribution Prediction","url":"https://huggingface.co/papers/2412.16846"},{"title":"Daily Papers - Hugging Face","url":"https://huggingface.co/papers?q=autoregressive+synthesis"},{"title":"Continuous Autoregressive Modeling with Stochastic Monotonic Alignment for Speech Synthesis | OpenReview","url":"https://openreview.net/forum?id=cuFzE8Jlvb"},{"title":"[PDF] Autoregressive Speech Synthesis without Vector Quantization | Semantic Scholar","url":"https://www.semanticscholar.org/paper/df05872bd7a1b88f89e3b8a3014318bee337a809"}]

Based on the search results, I found comprehensive information about KALL-E:

## KALL-E: Autoregressive Speech Synthesis with Next-Distribution Prediction

KALL-E is a novel autoregressive (AR) language model for text-to-speech (TTS) synthesis that operates by predicting the next distribution of continuous speech frames.

### Key Features

Unlike existing methods, KALL-E directly models the continuous speech distribution conditioned on text, eliminating the need for any diffusion-based components. Specifically, it utilizes a Flow-VAE to extract a continuous latent speech representation from waveforms, instead of relying on discrete speech tokens.

A single AR Transformer is then trained to predict these continuous speech distributions from text, optimizing a Kullback-Leibler divergence loss as its objective.

### Architecture

The framework comprises two main components:

1. **Flow-VAE**: By introducing flow in VAE, the extracted continuous features no longer obey the strict Gaussian prior. This brings more diverse generation results and more robust modeling.

2. **Autoregressive Language Model**: KALL-E uses the Kullback-Leibler (KL) divergence loss for next-distribution prediction instead of the traditional cross-entropy loss. This method models speech features in continuous space, avoiding a series of information loss issues caused by quantization.

### Performance

Experimental results demonstrate that KALL-E achieves superior speech synthesis quality and can even adapt to a target speaker from just a single sample. KALL-E outperforms open-source implementations of YourTTS, VALL-E, NaturalSpeech 2, and CosyVoice in terms of naturalness and speaker similarity in zero-shot TTS scenarios.

REMINDER: You MUST include the sources above in your response to the user using markdown hyperlinks.


Web search results for query: ""bridging gap training inference" LM-based TTS exposure bias scheduled sampling"

Links: [{"title":"BRIDGING THE GAP BETWEEN TRAINING AND INFERENCE IN LM-BASED TTS MODELS","url":"https://arxiv.org/pdf/2509.17021"},{"title":"[2509.17021] Bridging the gap between training and inference in LM-based TTS models","url":"https://arxiv.org/abs/2509.17021"},{"title":"[Literature Review] Bridging the gap between training and inference in LM-based TTS models","url":"https://www.themoonlight.io/en/review/bridging-the-gap-between-training-and-inference-in-lm-based-tts-models"},{"title":"Continuous-Token Diffusion for Speaker-Referenced TTS in Multimodal LLMs","url":"https://arxiv.org/pdf/2510.12995"},{"title":"Bridging the Gap between Training and Inference for Neural Machine Translation | Request PDF","url":"https://www.researchgate.net/publication/335778783_Bridging_the_Gap_between_Training_and_Inference_for_Neural_Machine_Translation"},{"title":"[2108.12963] Scheduled Sampling Based on Decoding Steps for Neural Machine Translation","url":"https://arxiv.org/abs/2108.12963"},{"title":"MIDAS: Multimodal Interactive Digital-humAn Synthesis via ...","url":"https://arxiv.org/pdf/2508.19320"},{"title":"Bridging the Train-Test Gap in Autoregressive Video ...","url":"https://self-forcing.github.io/static/self_forcing.pdf"},{"title":"O-1: Self-training with Oracle and 1-best Hypothesis","url":"https://www.isca-archive.org/interspeech_2023/baskar23_interspeech.pdf"},{"title":"Bridging the Gap between Training and Inference for Neural Machine Translation","url":"http://www.nlpir.org/wordpress/wp-content/uploads/2019/08/Bridging-the-Gap-between-Training-and-Inference-for-Neural-Machine-Translation.pdf"}]

Based on the search results, here's what I found about bridging the gap between training and inference in LM-based TTS models with exposure bias and scheduled sampling:

## Core Problem

In training, TTS models use ground-truth (GT) tokens as prefixes to predict the next token, while in inference these tokens are not available, a gap between training and inference that is often neglected. This mismatch is known as exposure bias.

## Main Solution Approach

A prompt-guided hybrid training scheme mitigates exposure bias in LM-based TTS systems by adopting a hybrid training paradigm that combines teacher forcing with free running, thereby introducing self-generated tokens into the training process. This makes the training mode more consistent with inference, reducing the training-inference gap.

## Key Mechanisms

An EOS prediction mechanism is incorporated during training to detect incorrect sequence termination and adaptively control the free running process. This allows the model to dynamically adjust how much self-generated token exposure it receives during training.

## Scheduled Sampling Connection

More broadly, scheduled sampling is widely used to mitigate the exposure bias problem, with its core motivation being to simulate the inference scene during training by replacing ground-truth tokens with predicted tokens, thus bridging the gap between training and inference.

## Results

Experimental results demonstrate that this method effectively narrows the training-inference gap, thereby improving the quality of synthesized long-form speech.

REMINDER: You MUST include the sources above in your response to the user using markdown hyperlinks.


Question: self-forcing autoregressive generation exposure bias diffusion latent space video audio


Question: JEPA generation not just representation learning world model planning decoder


Web search results for query: "JEPA generation not just representation learning world model planning decoder"

Links: [{"title":"The Anatomy of JEPA: The Architecture Behind embedded Predictive Representation Learning | by Tyler Frink | Medium","url":"https://medium.com/@frinktyler1445/the-anatomy-of-jepa-the-architecture-behind-embedded-predictive-representation-learning-994bfa0bffe0"},{"title":"Rohit Bandaru | Deep Dive into Yann LeCun’s JEPA","url":"https://rohitbandaru.github.io/blog/JEPA-Deep-Dive/"},{"title":"V-JEPA: The next step toward advanced machine intelligence","url":"https://ai.meta.com/blog/v-jepa-yann-lecun-ai-model-video-joint-embedding-predictive-architecture/"},{"title":"Yann LeCun’s Joint Embedding Predictive Architecture (JEPA) and the General Theory of Intelligence","url":"https://www.thesingularityproject.ai/p/yann-lecuns-joint-embedding-predictive"},{"title":"V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning","url":"https://arxiv.org/html/2506.09985v1"},{"title":"VL-JEPA: Joint Embedding Predictive Architecture for Vision-language","url":"https://arxiv.org/pdf/2512.10942"},{"title":"GitHub - facebookresearch/ijepa: Official codebase for I-JEPA, the Image-based Joint-Embedding Predictive Architecture. First outlined in the CVPR paper, \"Self-supervised learning from images with a joint-embedding predictive architecture.\" · GitHub","url":"https://github.com/facebookresearch/ijepa"},{"title":"Meta’s new VL-JEPA model shifts from generating tokens to predicting concepts","url":"https://bdtechtalks.substack.com/p/metas-new-vl-jepa-model-shifts-from"},{"title":"I-JEPA: The first AI model based on Yann LeCun’s vision for more human-like AI","url":"https://ai.meta.com/blog/yann-lecun-ai-model-i-jepa/"},{"title":"What is Joint Embedding Predictive Architecture (JEPA)?","url":"https://www.turingpost.com/p/jepa"}]

Based on the search results, here's what I found about JEPA (Joint Embedding Predictive Architecture), focusing on the distinction between generation and representation learning:

## Key Distinction: Non-Generative Representation Learning

JEPA is a non-generative architecture that predicts higher-level representations rather than raw data. This is a fundamental difference from traditional generative models.

## Why No Decoder?

JEPA generates video at the abstract level – which is often more useful for planning, control, and simulation. There's no decoder – because no human needs to see it. To produce an actual image or sentence from a JEPA, one would need an additional decoding mechanism.

## Focus on Predictable Aspects

In contrast to approaches based on video generation, the JEPA approach focuses on learning representations for predictable aspects of a scene (e.g., the trajectory of an object in motion) while ignoring unpredictable details that generative objectives emphasize, since they make pixel-level predictions.

## World Model for Planning

Unlike text models that predict the next word, or image models that fill in pixels, JEPA is trained to anticipate how the real world evolves – in motion, causality, and interaction. It's a model of "what happens next" – not just in raw features, but in space and time.

By scaling JEPA pretraining, it yields video representations with state-of-the-art understanding and prediction capabilities, and such representations can be leveraged as a basis for action-conditioned predictive models and enable zero-shot planning.

REMINDER: You MUST include the sources above in your response to the user using markdown hyperlinks.


Web search results for query: "self-forcing autoregressive generation exposure bias diffusion latent space video audio"

Links: [{"title":"MIDAS: Multimodal Interactive Digital-humAn Synthesis via ...","url":"https://arxiv.org/pdf/2508.19320"},{"title":"[2506.08009] Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion","url":"https://arxiv.org/abs/2506.08009"},{"title":"Bridging the Train-Test Gap in Autoregressive Video ...","url":"https://self-forcing.github.io/static/self_forcing.pdf"},{"title":"GitHub - thu-ml/Causal-Forcing: Official codebase for \"Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation\" · GitHub","url":"https://github.com/thu-ml/Causal-Forcing"},{"title":"Paper page - Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion","url":"https://huggingface.co/papers/2506.08009"},{"title":"Self-Forcing++: Towards Minute-Scale High-Quality Video Generation","url":"https://arxiv.org/pdf/2510.02283"},{"title":"Self-Forcing++: Scalable Video Generation","url":"https://www.emergentmind.com/topics/self-forcing"},{"title":"Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion","url":"https://arxiv.org/html/2506.08009v1"},{"title":"Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion | OpenReview","url":"https://openreview.net/forum?id=mSiN7i0BYH"},{"title":"ROLLING FORCING: AUTOREGRESSIVE LONG VIDEO DIFFUSION IN REAL TIME","url":"https://arxiv.org/pdf/2509.25161"}]

I've found comprehensive research results on this topic. Here's what the search revealed:

## Key Research Papers and Topics

**Self Forcing** is a novel training paradigm for autoregressive video diffusion models that addresses exposure bias, where models trained on ground-truth context must generate sequences conditioned on their own imperfect outputs during inference. Unlike prior methods that denoise future frames based on ground-truth context frames, Self Forcing conditions each frame's generation on previously self-generated outputs by performing autoregressive rollout with key-value (KV) caching during training.

### Technical Innovations

This strategy enables supervision through a holistic loss at the video level that directly evaluates the quality of the entire generated sequence, rather than relying solely on traditional frame-wise objectives. To ensure training efficiency, the method employs a few-step diffusion model along with a stochastic gradient truncation strategy.

### Performance Results

The approach achieves real-time streaming video generation with sub-second latency on a single GPU, while matching or even surpassing the generation quality of significantly slower and non-causal diffusion models.

### Extended Work: Self-Forcing++

For extended durations, semantic drift, exposure bias, and motion stagnation are minimized, while temporal variation is preserved. Self-Forcing++ provides a solution framework for minute-scale, high-fidelity autoregressive video generation, addressing key failure modes of error accumulation and short-horizon bias in diffusion-based architectures.

### Related Architectures

Some multimodal approaches encode diverse input signals, including audio, pose, and text, into a shared latent space, with these multimodal tokens concatenated with frame tokens to form the input sequence for the autoregressive model.

REMINDER: You MUST include the sources above in your response to the user using markdown hyperlinks.


Now let me do a few final targeted searches.


Question: eb_jepa facebook research energy based JEPA planning generation


Question: D-JEPA discrete JEPA generation tokens autoregressive


Web search results for query: "eb_jepa facebook research energy based JEPA planning generation"

Links: [{"title":"GitHub - facebookresearch/eb_jepa: An open source library designed to provide community examples of Joint Embedding Predictive Architectures (JEPAs). It contains code and examples for learning representations from images, video, and action-conditioned video, as well as planning using JEPA-based models. · GitHub","url":"https://github.com/facebookresearch/eb_jepa"},{"title":"A Lightweight Library for Energy-Based Joint-Embedding Predictive Architectures","url":"https://arxiv.org/html/2602.03604v1"},{"title":"(PDF) A Lightweight Library for Energy-Based Joint-Embedding Predictive Architectures","url":"https://www.researchgate.net/publication/400415391_A_Lightweight_Library_for_Energy-Based_Joint-Embedding_Predictive_Architectures"},{"title":"Lightweight EB-JEPA: Energy-Based Predictive Library","url":"https://www.emergentmind.com/papers/2602.03604"},{"title":"Daily Papers - Hugging Face","url":"https://huggingface.co/papers?q=Joint+Embedding+Predictive+Architecture+(JEPA)"},{"title":"Meta's EB-JEPA: Energy-Based Joint-Embedding Predictive Architectures AI research on JEPA using 1 GPU: Each example is (almost) self-contained and training takes up to a few hours on a single GPU card. https://github.com/facebookresearch/eb_jepa","url":"https://www.threads.com/@sung.kim.mw/post/DU_VzBTD75y/metas-eb-jepa-energy-based-joint-embedding-predictive-architectures-ai-research"},{"title":"Rohit Bandaru | Deep Dive into Yann LeCun’s JEPA","url":"https://rohitbandaru.github.io/blog/JEPA-Deep-Dive/"},{"title":"Energy-Based Joint-Embedding Predictive Architectures Made Accessible","url":"https://www.emergentmind.com/videos/lightweight-eb-jepa-energy-based-predictive-library-96e9c52b"},{"title":"JEPA: LeCun’s Path Towards More Human-Like AI | by Anil Jain | AI / ML Architect | Data Architect | Medium","url":"https://medium.com/@anil.jain.baba/jepa-lecuns-path-towards-more-human-like-ai-9535e48b3c65"},{"title":"From Machine Learning to Autonomous Intelligence Lecture 3 Yann LeCun","url":"https://leshouches2022.github.io/SLIDES/lecun-20220720-leshouches-03.pdf"}]

Based on the search results, here's what I found about EB-JEPA:

## Overview

EB-JEPA is an open-source library that addresses the gap in accessible JEPA implementations through modular, well-documented implementations of JEPA-based models trainable at small scale with simple, concise code designed for educational purposes and rapid experimentation.

## What It Is

EB-JEPA is an open-source library for learning representations and world models using Joint-Embedding Predictive Architectures (JEPAs). JEPAs learn to predict in representation space rather than pixel space, avoiding the pitfalls of generative modeling while capturing semantically meaningful features suitable for downstream tasks.

## Key Features

The library provides three progressively complex examples: image representation learning, video prediction, and action-conditioned planning, each trainable on a single GPU in a few hours. EB-JEPA operationalizes these principles through a concise codebase implementing three escalating modalities: self-supervised learning on static images, temporal prediction in videos, and action-conditioned world modeling for goal-directed planning.

## Research Results

Probing the representations yields 91% accuracy on CIFAR-10, multi-step prediction examples on Moving MNIST demonstrate how the same principles scale to temporal modeling, and action-conditioned world models achieve a 97% planning success rate on the Two Rooms navigation task.

## Accessibility

Code is available at https://github.com/facebookresearch/eb_jepa. The library is designed to be accessible to researchers and students with limited computational resources.

REMINDER: You MUST include the sources above in your response to the user using markdown hyperlinks.


Web search results for query: "D-JEPA discrete JEPA generation tokens autoregressive"

Links: [{"title":"Discrete JEPA: Learning Discrete Token Representations without Reconstruction","url":"https://openreview.net/pdf?id=91kfdrEafN"},{"title":"[2512.19171] JEPA-Reasoner: Decoupling Latent Reasoning from Token Generation","url":"https://arxiv.org/abs/2512.19171"},{"title":"JEPA-Reasoner: Decoupling Latent Reasoning from Token Generation","url":"https://arxiv.org/html/2512.19171"},{"title":"Discrete JEPA","url":"https://arxiv.org/pdf/2506.14373"},{"title":"Discrete JEPA: Learning Discrete Token Representations without Reconstruction","url":"https://arxiv.org/html/2506.14373v1"},{"title":"Denoising with a Joint-Embedding Predictive Architecture (D-JEPA) | D-JEPA","url":"https://d-jepa.github.io/"},{"title":"JEPA-Reasoner: Decoupling Latent Reasoning from Token Generation | alphaXiv","url":"https://www.alphaxiv.org/overview/2512.19171v1"},{"title":"Rohit Bandaru | Deep Dive into Yann LeCun’s JEPA","url":"https://rohitbandaru.github.io/blog/JEPA-Deep-Dive/"},{"title":"(PDF) Discrete JEPA: Learning Discrete Token Representations without Reconstruction","url":"https://www.researchgate.net/publication/392766499_Discrete_JEPA_Learning_Discrete_Token_Representations_without_Reconstruction"},{"title":"JEPA-Reasoner: Generative Latent Space Reasoner | OpenReview","url":"https://openreview.net/forum?id=7ruA2rXG42"}]

Based on the search results, here's what I found about D-JEPA, discrete JEPA, generation tokens, and autoregressive approaches:

## D-JEPA (Denoising with a Joint-Embedding Predictive Architecture)

D-JEPA pioneers the integration of JEPA within generative modeling. It reinterprets JEPA as a form of masked image modeling and generalized next-token prediction strategy, facilitating data generation in an auto-regressive manner while incorporating diffusion loss to model the per-token probability distribution.

## Discrete JEPA

Discrete JEPA is a related but distinct approach that addresses limitations by learning discrete semantic tokens via latent predictive coding, enabling superior symbolic reasoning capabilities. Key features include:

- **Autoregressive World Models**: Discrete JEPA trains autoregressive world models using standard Vision Transformer architecture for long-horizon sequence prediction tasks.

- **Error Mitigation**: A fundamental advantage of Discrete-JEPA emerges in its ability to prevent error accumulation over extended prediction horizons by operating in a constrained discrete index space rather than continuous representations, eliminating the compounding errors that plague continuous prediction approaches.

- **Semantic Tokens**: Traditional approaches rely on pixel-level reconstruction objectives that bias representations toward fine-grained details rather than semantic concepts essential for symbolic reasoning, a limitation that Discrete JEPA overcomes through latent predictive learning that avoids reconstruction bias.

## Relationship Between D-JEPA and Discrete JEPA

Attempts to make JEPA generative, such as D-JEPA, use learned representations to condition diffusion models for data generation (e.g., text-to-image, text-to-audio), but still fail to enable sequential reasoning or planning within the JEPA framework itself.

REMINDER: You MUST include the sources above in your response to the user using markdown hyperlinks.


Now I have comprehensive results. Here is my thorough synthesis:

---

## Research Findings: JEPA for TTS / Audio Generation

### 1. GitHub Repos with JEPA for Audio

**No JEPA-for-TTS repo exists.** Nobody has published a JEPA-based TTS system. What exists:

- **[SonyCSLParis/audio-representations](https://github.com/SonyCSLParis/audio-representations)** -- JEPA for general audio *representation learning* (ICASSP 2024 SASB workshop). Closest to what you're doing architecturally, but representation-only.
- **[labhamlet/wavjepa](https://github.com/labhamlet/wavjepa)** -- WavJEPA official code. Time-domain audio foundation model, representation learning only.
- **[SonyCSLParis/Stem-JEPA](https://github.com/SonyCSLParis/Stem-JEPA)** -- JEPA for musical stem compatibility (ISMIR 2024). Not generative.
- **[facebookresearch/eb_jepa](https://github.com/facebookresearch/eb_jepa)** -- Meta's educational JEPA library with image, video, and action-conditioned planning examples. Demonstrates JEPA for *planning* (generation of action sequences in latent space), which is conceptually closest to what you'd need for autoregressive latent generation.
- **[facebookresearch/ijepa](https://github.com/facebookresearch/ijepa)** and **[facebookresearch/jepa](https://github.com/facebookresearch/jepa)** (V-JEPA) -- Official Meta codebases for image/video JEPA.

**Your LeWM-TTS project appears to be the first attempt at JEPA-based TTS.** This is genuinely novel.

### 2. Papers Using JEPA for Audio (Representation, Not Generation)

All existing JEPA audio papers are for *representation learning*, not generation:

- **[A-JEPA](https://arxiv.org/abs/2311.15830)** (Nov 2023) -- Masked spectrogram prediction in latent space with curriculum masking (time-frequency aware). SOTA on audio/speech classification. No generation capability.
- **[Audio-JEPA](https://arxiv.org/abs/2507.02915)** (Tuncay et al., July 2025) -- ViT backbone predicting latent representations of masked spectrogram patches. Comparable to wav2vec 2.0 with 1/5th the data. Representation only.
- **[WavJEPA](https://arxiv.org/abs/2509.23238)** (Sep 2025) -- Waveform-domain JEPA. Outperforms time-domain audio foundation models. Has [code](https://github.com/labhamlet/wavjepa). Representation only.
- **[JEPA as a Neural Tokenizer](https://arxiv.org/abs/2512.07168)** (Dec 2025) -- **Most relevant to your work.** Two-stage: (1) JEPA+DAAM learns semantic features at 2.5 Hz via masked latent prediction, (2) FSQ tokenization + HiFi-GAN decoder for waveform reconstruction. Produces 47.5 tokens/sec, vocabulary of 16,384. Designed to be "LM-friendly" -- explicitly intended as a tokenizer for downstream LM-based TTS. But the paper itself does not build the TTS system on top.
- **[JEPA-v0](https://startpinch.com/research/en/jepa-encoder-translation/)** (Pinch Research, 2025-2026) -- JEPA encoder for speech-to-speech translation. Preserves speaker identity unlike ASR-MT-TTS cascades. Encoder only; no generative decoder published.

### 3. D-JEPA and Discrete JEPA -- Attempts at Making JEPA Generative

These are the closest to what you're trying to do (making JEPA generate, not just represent):

- **[D-JEPA](https://d-jepa.github.io/)** -- Reinterprets JEPA as masked image modeling + next-token prediction, then uses diffusion loss to model per-token probability distribution for image generation. **This is the key bridge paper** -- it shows how to go from JEPA representations to generation via autoregressive + diffusion.
- **[Discrete JEPA](https://arxiv.org/pdf/2506.14373)** (June 2025) -- Learns discrete semantic tokens via latent predictive coding. Key finding: **operating in discrete index space prevents error accumulation** over long horizons, unlike continuous prediction which suffers compounding errors. Trains autoregressive world models using standard ViT. Directly relevant to your exposure bias concern.
- **[JEPA-Reasoner](https://arxiv.org/abs/2512.19171)** (Dec 2025) -- **Most architecturally relevant to your problem.** Autoregressively reasons in continuous latent space, with a separate "Talker" decoder for token generation. Key insight: **decoupling latent-space prediction from output generation** contains errors -- token-level failures cannot propagate into the latent reasoning chain. Uses normalized latent space. This is essentially the same architecture pattern as your JEPA encoder + separate decoder.

### 4. How the Exposure Bias / Embedding Space Mismatch Problem Is Handled

This is the core challenge for your architecture. Here's how different systems address it:

**Direct approaches to exposure bias in latent-prediction TTS:**

- **[Bridging the Gap Between Training and Inference in LM-Based TTS](https://arxiv.org/abs/2509.17021)** (Sep 2025) -- Hybrid training combining teacher forcing with "free running" (using self-generated tokens during training). Uses an EOS prediction mechanism to adaptively control how much the model sees its own outputs. Directly applicable to your problem.

- **[Self-Forcing](https://arxiv.org/abs/2506.08009)** (June 2025, video diffusion but principle applies) -- During training, conditions on self-generated outputs rather than ground truth, with holistic sequence-level loss. Uses stochastic gradient truncation for efficiency. The **Self-Forcing++** extension addresses semantic drift over extended generation.

- **[KALL-E](https://arxiv.org/abs/2412.16846)** (Dec 2024) -- Predicts the **next distribution** (not next token) of continuous speech frames using Flow-VAE + KL divergence loss. Avoids quantization entirely. The distribution prediction sidesteps the discrete mismatch problem. Outperforms VALL-E, NaturalSpeech 2, CosyVoice.

- **[CLEAR](https://arxiv.org/abs/2508.19098)** (Aug 2025) -- Continuous Latent Autoregressive TTS. Maps waveforms to compact continuous latents via enhanced VAE, then uses MLP-based rectified flow head per hidden state. Single-stage training. 96ms first-frame delay for streaming. The continuous latent representation avoids the discrete token mismatch.

**How SoundStorm and VALL-E handle it:**

- **[SoundStorm](https://arxiv.org/abs/2305.09636)** -- Sidesteps exposure bias entirely by using **non-autoregressive parallel decoding** with confidence-based iterative refinement (MaskGIT-style). Takes semantic tokens from AudioLM, generates codec tokens in parallel. 2 orders of magnitude faster than autoregressive. The key insight: avoid autoregression altogether for the acoustic token generation stage.

- **VALL-E** -- Hybrid: first RVQ level is autoregressive, subsequent levels are non-autoregressive (greedy level-by-level). This limits exposure bias to only the first level.

### 5. Key Takeaways for Your LeWM-TTS Architecture

Based on this research, here is what matters for your project:

**You are in genuinely uncharted territory.** No one has published a JEPA-based TTS system. The closest prior work is:
- JEPA as Neural Tokenizer (provides the encoder/tokenizer half)
- JEPA-Reasoner (provides the autoregressive-in-latent-space + separate decoder pattern)
- D-JEPA (shows how to bridge JEPA to generation)

**For the exposure bias / embedding mismatch problem, the most promising strategies from the literature are:**

1. **Hybrid teacher-forcing / free-running training** (from the "Bridging the Gap" paper) -- gradually introduce self-generated latents during training
2. **Self-Forcing** -- train directly on self-generated rollouts with sequence-level loss
3. **Predict distributions, not points** (KALL-E approach) -- predict a distribution over the next latent rather than a single embedding, using KL divergence loss
4. **Discrete JEPA's finding** -- discretizing the latent space (via FSQ or VQ) may inherently limit error accumulation vs continuous prediction
5. **Non-autoregressive refinement** (SoundStorm approach) -- use AR only for coarse semantic tokens, then parallel decode the fine acoustic tokens

Sources:
- [A-JEPA: Joint-Embedding Predictive Architecture Can Listen](https://arxiv.org/abs/2311.15830)
- [Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning](https://arxiv.org/abs/2507.02915)
- [WavJEPA: Semantic learning unlocks robust audio foundation models for raw waveforms](https://arxiv.org/abs/2509.23238)
- [JEPA as a Neural Tokenizer (arxiv 2512.07168)](https://arxiv.org/abs/2512.07168)
- [JEPA-v0: Pinch Research](https://startpinch.com/research/en/jepa-encoder-translation/)
- [D-JEPA: Denoising with JEPA](https://d-jepa.github.io/)
- [Discrete JEPA (arxiv 2506.14373)](https://arxiv.org/pdf/2506.14373)
- [JEPA-Reasoner: Decoupling Latent Reasoning from Token Generation](https://arxiv.org/abs/2512.19171)
- [Bridging the Gap Between Training and Inference in LM-Based TTS](https://arxiv.org/abs/2509.17021)
- [Self-Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion](https://arxiv.org/abs/2506.08009)
- [KALL-E: Autoregressive Speech Synthesis with Next-Distribution Prediction](https://arxiv.org/abs/2412.16846)
- [CLEAR: Continuous Latent Autoregressive Modeling for Speech Synthesis](https://arxiv.org/abs/2508.19098)
- [SoundStorm: Efficient Parallel Audio Generation](https://arxiv.org/abs/2305.09636)
- [SonyCSLParis/audio-representations (GitHub)](https://github.com/SonyCSLParis/audio-representations)
- [labhamlet/wavjepa (GitHub)](https://github.com/labhamlet/wavjepa)
- [facebookresearch/eb_jepa (GitHub)](https://github.com/facebookresearch/eb_jepa)
- [JEPA-Reasoner OpenReview](https://openreview.net/forum?id=7ruA2rXG42)
- [Discrete JEPA OpenReview](https://openreview.net/pdf?id=91kfdrEafN)
- [JEPA as Neural Tokenizer UniReps Blog](https://unireps.org/blog/2025/jepa-daam_fsqvae/)
- [Exploring JEPA for Real-Time Speech Translation (Hacker News)](https://news.ycombinator.com/item?id=47332883)