https://arxiv.org/abs/2507.02915 that is this


ToolSearch

Question: select:WebFetch


Question: select:WebFetch


WebFetch

Prompt: Provide a detailed summary of this paper: title, authors, abstract, key contributions, methodology, and main results.


# Audio-JEPA Research Summary

## Title
"Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning"

## Authors
Ludovic Tuncay, Etienne Labbé, Emmanouil Benetos, and Thomas Pellegrini from IRIT-SAMoVA, QMUL, and affiliated institutions.

## Abstract Overview
The researchers developed an audio-specific adaptation of the Joint-Embedding Predictive Architecture. Their model "predicts latent representations of masked regions in high-level feature spaces" rather than reconstructing raw audio signals directly.

## Key Contributions

**Model Architecture**: The team applied a Vision Transformer backbone to process mel-spectrogram patches, introducing masking strategies adapted for audio data rather than visual inputs.

**Training Approach**: Pre-training occurred on unlabeled AudioSet clips (10 seconds at 32kHz) using random patch masking on spectrograms.

**Efficiency Gains**: The implementation achieved "comparable performance to wav2vec 2.0 and data2vec while using less than one-fifth of their training data."

## Methodology
The approach involves masked prediction in latent representation spaces, avoiding expensive raw audio reconstruction. The team evaluated results across the X-ARES benchmark suite, which encompasses speech, music, and environmental sound tasks.

## Main Results
Despite being a straightforward translation of the original JEPA framework to audio domain, Audio-JEPA demonstrated competitive performance with established models while requiring significantly fewer training resources and no custom hyperparameter optimization.


This is **Audio-JEPA** — a paper applying the Joint-Embedding Predictive Architecture (JEPA) to audio representation learning.

**Key points:**

- **Authors:** Ludovic Tuncay, Etienne Labbé, Emmanouil Benetos, Thomas Pellegrini (IRIT-SAMoVA, QMUL)
- **Idea:** Apply JEPA (originally from vision) to audio by predicting latent representations of masked regions in mel-spectrograms, rather than reconstructing raw audio
- **Architecture:** Vision Transformer backbone operating on mel-spectrogram patches with audio-adapted masking strategies
- **Training:** Self-supervised pre-training on unlabeled AudioSet (10s clips, 32kHz)
- **Key result:** Matches wav2vec 2.0 and data2vec performance while using **<1/5th of their training data** — very data-efficient
- **Evaluation:** Tested across X-ARES benchmark (speech, music, environmental sound)

This is quite relevant to your LeWM TTS work since you're already using a JEPA-based architecture. Audio-JEPA validates that JEPA transfers well to the audio domain, specifically that predicting in latent space (rather than reconstructing spectrograms/waveforms directly) is an effective strategy for learning audio representations. Their data efficiency finding is also notable given your single-speaker dataset setup.

Want me to dig deeper into any specific aspect of the paper — architecture details, masking strategy, or how it might inform your TTS approach?


yeah do


Prompt:
Extract ALL technical details from this paper. Include:

1. Full architecture details - encoder, predictor, target encoder, dimensions, layers, heads, patch sizes
2. Masking strategy - how patches are masked, mask ratios, block masking vs random masking, context vs target selection
3. Loss function - exactly what loss is used (VICReg? MSE? other?), any regularization
4. Training details - optimizer, learning rate schedule, batch size, epochs, EMA schedule for target encoder
5. Mel-spectrogram preprocessing - sample rate, hop size, n_mels, n_fft, window size, normalization
6. How the context encoder and target encoder relate (shared weights? EMA?)
7. Evaluation protocol - how representations are extracted and used for downstream tasks
8. Ablation studies - what was tested and what mattered most
9. Comparison tables with baselines
10. Any discussion of limitations or future work

Be as detailed and technical as possible. Include specific numbers, hyperparameters, and equations where present.


Prompt:
Extract ALL technical details from this paper. Include:

1. Full architecture details - encoder, predictor, target encoder, dimensions, layers, heads, patch sizes
2. Masking strategy - how patches are masked, mask ratios, block masking vs random masking, context vs target selection
3. Loss function - exactly what loss is used, any regularization terms
4. Training details - optimizer, learning rate, batch size, epochs, EMA decay
5. Mel-spectrogram preprocessing details
6. Ablation studies and what mattered
7. All results tables with numbers
8. Equations if present
9. Limitations and future work

Be as exhaustive as possible with specific numbers and hyperparameters.


Request failed with status code 404

Error: Request failed with status code 404


I appreciate your request, but I need to be transparent: the content you've provided appears to be a PDF file encoded in binary/compressed format (starting with `%PDF-1.7` and containing compressed streams). This isn't readable text that I can extract technical details from.

To help you effectively, I would need:

1. **Plain text version** of the paper, or
2. **A readable PDF** that can be processed, or
3. **A direct link** to the paper on arXiv, Papers with Code, or a publisher's website

If you can provide the paper in one of these formats, I'd be happy to extract:
- Architecture specifications (encoder/decoder layers, dimensions, attention heads)
- Masking strategies and ratios
- Loss functions and regularization terms
- Precise hyperparameters (learning rates, batch sizes, schedules)
- Audio preprocessing parameters (sample rates, mel-spectrogram settings)
- Training protocols and evaluation metrics
- Ablation study results with numerical comparisons
- Limitations and future directions

Could you share the paper in a more accessible format?

[Binary content (application/pdf, 671.4KB) also saved to /home/ubuntu/.claude/projects/-home-ubuntu/e792b4a6-7519-4b8f-9922-071bec1295ca/tool-results/webfetch-1774856336725-aq6thu.pdf]


The PDF was downloaded but not parsed inline. Let me read it directly and also try the HTML version.


Error: pdftoppm is not installed. Install poppler-utils (e.g. `brew install poppler` or `apt-get install poppler-utils`) to enable PDF page rendering.


Prompt:
Extract ALL technical details from this paper exhaustively. Include:
1. Architecture details - encoder type, layers, hidden dims, heads, patch sizes, predictor architecture
2. Masking strategy - context vs target selection, mask ratio, block sizes
3. Loss function and regularization
4. Training hyperparameters - optimizer, LR, schedule, batch size, epochs, EMA decay
5. Audio preprocessing - sample rate, n_mels, hop size, n_fft, normalization
6. Ablation results with numbers
7. All benchmark results with numbers
8. Equations
9. Limitations
Be maximally detailed with every number you can find.


Request failed with status code 404


Bash

sudo apt-get install -y poppler-utils 2>&1 | tail -3

Install poppler-utils for PDF reading


No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.


which pdftoppm && pdftoppm -v 2>&1 || echo "not found"

Check pdftoppm installation


/usr/bin/pdftoppm
pdftoppm version 22.02.0
Copyright 2005-2022 The Poppler Developers - http://poppler.freedesktop.org
Copyright 1996-2011 Glyph & Cog, LLC


pdftotext /home/ubuntu/.claude/projects/-home-ubuntu/e792b4a6-7519-4b8f-9922-071bec1295ca/tool-results/webfetch-1774856336725-aq6thu.pdf /home/ubuntu/audio_jepa_paper.txt 2>&1 && wc -l /home/ubuntu/audio_jepa_paper.txt

Convert PDF to text


865 /home/ubuntu/audio_jepa_paper.txt


1	Audio-JEPA: Joint-Embedding Predictive
2	Architecture for Audio Representation Learning
3	Ludovic TUNCAY ¹
4
5	Etienne LABBE ¹
6
7	Emmanouil BENETOS ²
8
9	Thomas PELLEGRINI ¹
10
11	ludovic.tuncay@irit.fr
12
13	etienne.labbe@irit.fr
14
15	emmanouil.benetos@qmul.ac.uk
16
17	thomas.pellegrini@irit.fr
18
19	¹ IRIT, Université de Toulouse, CNRS, Toulouse INP, Toulouse, France
20	² School of Electronic Engineering and Computer Science, Queen Mary University of London, UK
21
22	Abstract—Building on the Joint-Embedding Predictive Architecture (JEPA) paradigm, a recent self-supervised learning
23	framework that predicts latent representations of masked regions in high-level feature spaces, we propose Audio‑JEPA (Audio Joint‑Embedding Predictive Architecture), tailored specifically for audio data. Audio-JEPA uses a simple Vision Transformer backbone to predict latent representations of masked
24	spectrogram patches rather than reconstructing raw audio.
25	We pre‑train on unlabeled AudioSet clips (10s, 32kHz) with
26	random patch masking on mel‑spectrograms. We evaluate on
27	the X‑ARES suite covering speech, music, and environmental‐
28	sound tasks. Although our implementation is a straightforward
29	translation of the original model to audio, the results still show
30	comparable performance to wav2vec 2.0 and data2vec while
31	using less than one‑fifth of their training data and with no
32	hyper‑parameter tuning. All code and pretrained checkpoints
33	will be released on GitHub¹.
34	Index Terms—Self-supervised learning, audio representation,
35	joint-embedding predictive architecture, Audio-JEPA, AudioSet
36
37	I. Introduction
38	Self-Supervised Learning (SSL) has revolutionized representation learning for speech and audio, enabling models to
39	learn from unlabeled data and excel in diverse downstream
40	tasks [1, 2, 3, 4]. Early SSL approaches for audio, such as
41	contrastive predictive coding and wav2vec 2.0, learned latent
42	speech representations by masking the input and solving a
43	contrastive task over latent codes [5]. Follow-up methods like
44	HuBERT [1] introduced offline clustering to generate pseudolabels for masked audio segments and WavLM [6] applied
45	data augmentation and denoising to improve robustness in
46	speech representation learning. More recently, latent prediction approaches have gained traction: data2vec [7] and its
47	efficient successor data2vec 2.0 [8] employ a teacher–student
48	framework to predict contextualized latent representations of
49	the input, achieving strong results across vision, speech, and
50	language tasks. In the audio domain, Niizumi et al. introduced
51	Masked Modeling Duo (M2D) [4], which uses two networks
52	(online and momentum encoder) to predict masked patch
53	embeddings and attained state-of-the-art results on numerous
54	audio benchmarks.
55	In computer vision, a new paradigm called Joint-Embedding Predictive Architecture (JEPA) [9, 10, 11] has been
56	proposed to predict hidden content in a high-level latent space
57	instead of pixel space. Notably, the image-based I-JEPA [10]
58	model demonstrated that predicting representations of masked
59	¹https://github.com/LudovicTuncay/Audio-JEPA
60
61	image regions can yield powerful visual features. The JEPA
62	approach differs from prior masked reconstruction methods by
63	focusing on semantic latent prediction rather than lower-level
64	signal reconstruction. Inspired by these advances, an audio
65	version of JEPA (termed A-JEPA) [12] was recently described
66	by Fei et al. Their A-JEPA encodes a spectrogram “context”
67	part and predicts the latent representations of masked “target”
68	regions using a momentum-updated target encoder. During
69	pre-training, they anneal the mask from fully random toward a
70	spec-augment-style structured scheme. In contrast, our AudioJEPA sticks with purely random masking throughout for
71	simplicity and maximal generality. At the time of writing,
72	no official implementation or checkpoints for A-JEPA are
73	available, motivating our from-scratch development. In the
74	musical domain, Stem-JEPA [13] adapts the JEPA paradigm
75	to multi-track recordings by jointly training an encoder and
76	predictor to forecast embeddings of compatible instrument
77	stems. While utilizing the JEPA backbone, Stem-JEPA differs
78	methodologically as it masks entire instrument stems instead
79	of individual spectrogram patches.
80	Our work targets the ICME 2025 Audio Encoder Capability
81	Challenge, where the goal is to learn general audio representations that perform well across a broad suite of tasks. We
82	present our Audio-JEPA implementation - developed from
83	scratch following the I-JEPA paradigm - and benchmark it on
84	the challenge’s eXtensive Audio Representation and Evaluation Suite (X-ARES). Our contributions include: (1) adapting
85	the JEPA masked latent prediction framework to audio spectrogram inputs using a Vision Transformer (ViT) backbone;
86	(2) an extensive evaluation against prior self-supervised audio
87	models on standard downstream tasks, assessing Audio-JEPA
88	via both linear probing and k-nearest-neighbor evaluation.
89
90	II. Related Work
91	A. Self-Supervised Audio Representation Learning
92	Early work in SSL for audio focused on predicting future or
93	missing parts of the waveform. wav2vec 2.0 [5] pioneered
94	masking in the latent speech representations and training
95	the model to identify the true quantized latent of a masked
96	segment among distractors (a contrastive loss). This approach
97	enabled models to learn rich speech features and achieved
98	remarkable results on speech recognition with limited labeled data. Building on this, HuBERT (Hidden-Unit BERT)
99	[1] introduced a BERT-like masked prediction where the
100	model predicts cluster assignments of masked audio frames.
101
102	Fig. 1. A‑JEPA architecture. Mel‑spectrogram patches are split into visible and masked sets. A context encoder embeds visible patches, a lightweight
103	predictor reconstructs masked‑patch embeddings, and a momentum‑updated target encoder provides targets. Training minimizes average L2 (Euclidean)
104	distance. The dashed arrow denotes a stop‑gradient.
105
106	HuBERT uses an offline k-means on acoustic features to
107	provide target labels, and by iteratively refining these labels, it
108	learns high-level speech units, matching wav2vec 2.0 performance on ASR benchmarks. Facebook’s data2vec presented
109	a modality-general SSL approach: instead of contrastive or
110	classification targets, data2vec [7] trains a student network
111	to regress the contextualized embeddings produced by a
112	teacher network (an Exponential Moving Average (EMA) of
113	the student) for masked portions of the input. data2vec 2.0
114	[8] improved the efficiency of this method by not encoding
115	masked tokens and using a lightweight decoder, achieving
116	similar accuracies to Masked Autoencoders [14] in a fraction
117	of the training time and matching wav2vec 2.0 on speech tasks
118	with over a 10× speedup. These latent regression approaches
119	eliminate the need for discrete targets and have set strong
120	baselines in audio.
121	B. Masked Prediction with Dual Networks
122	The use of two networks (online/target) for masked prediction
123	has also been explored in specialized audio SSL methods.
124	M2D (Masked Modeling Duo) [4] employs an online network
125	that sees the unmasked patches and a momentum target
126	network that encodes only the masked patches. The online
127	network predicts the target network’s representation of the
128	masked region, encouraging both networks to effectively
129	model the input. This design, inspired by Masked Autoencoders but working in representation space, led M2D to
130	state-of-the-art results on a range of audio classification
131	tasks (environmental sound, speaker ID, music genre, etc.).
132	Notably, M2D achieved top performance on datasets like
133	UrbanSound8K [15], VoxCeleb1 [16], GTZAN [17], and
134	SpeechCommands [18] with a single universal model. Such
135	results highlight the power of using latent prediction instead
136	of raw signal reconstruction for learning transferable audio
137	features. Other contemporary models include WavLM [6],
138	which extended HuBERT with simulated noisy inputs and
139	achieved strong results on both speech recognition and classification tasks.
140
141	C. Joint-Embedding Predictive Architectures
142	Rather than predicting low-level details of masked inputs,
143	JEPA methods aim to predict higher-level representations.
144	The image-based I-JEPA [10] demonstrated that a ViT can
145	learn excellent representations by predicting the latent representations of masked image patches, as opposed to generating
146	pixels. By operating in the feature space, I-JEPA forces the
147	model to capture abstract semantic information and ignore
148	minute pixel-level differences. The concept has since been
149	extended to other modalities and combinations (e.g., TIJEPA [19] for text–image, GeoJEPA [20] for geospatial
150	data, etc.), showing JEPA’s flexibility. For audio, Fei et al.
151	recently proposed A-JEPA [12], applying the same principle
152	to spectrogram inputs. While A‑JEPA and M2D both adopt
153	a dual‑network masked‑prediction framework, they differ in
154	how the target encoder is applied: M2D processes only the
155	masked spectrogram patches, whereas JEPA processes the
156	entire spectrogram (context + masked). This richer context
157	enables more detailed representations of the masked regions.
158	Their design uses a context encoder to process unmasked
159	spectrogram patches and a target encoder (the EMA of the
160	context network) to encode masked regions, with a lightweight
161	predictor network aligning the two in latent space. Our work is
162	directly inspired by this approach but contrary to M2D, JEPA
163	does not require data augmentation and the whole spectrogram
164	is seen by the target encoder. We evaluate Audio-JEPA in
165	our experiments, underlining how it bridges the gap between
166	vision-style masked modeling and audio understanding.
167
168	III. Proposed Method: JEPA for Audio
169	In this section, we describe our adaptation of the JEPA paradigm to the audio domain, which we call Audio-JEPA. We first
170	present the overall architecture, then detail the self‑supervised
171	training objectives, and finally highlight the audio‑specific
172	design choices that make A‑JEPA effective on diverse sound
173	data.
174
175	A. Overall architecture
176	As shown in the Fig. 1, our Audio-JEPA model consists of
177	three main modules:
178	1) Context encoder: Processes the “visible” subset of
179	Mel-spectrogram patches
180	2) Target Encoder: Provides stable target embeddings via
181	an Exponential Moving Average (EMA) of the context
182	encoder’s parameters
183	3) Lightweight Predictor Network: Takes context embeddings and predicts latent representations for masked
184	(“target”) patches
185	Upon converting an input waveform to a Mel‑spectrogram and
186	partitioning it into non‑overlapping time–frequency patches,
187	we randomly mask a fixed proportion of patches. The context
188	encoder embeds the remaining visible patches, producing a
189	context representation. The lightweight predictor network then
190	reconstructs embeddings for the masked patches. In parallel,
191	the target encoder (updated by EMA rather than gradient
192	descent) encodes the true masked patches. Training minimizes
193	the average L2 distance between the predictor’s outputs and
194	the target encoder’s embeddings, with stop‑gradient applied
195	between the predictor and the target encoder. This implementation is a direct adaptation of I-JEPA to the audio domain,
196	by considering the spectrogram as a single channel, possibly
197	non-square, image.
198	B. Training objective
199	We train Audio-JEPA using the average 𝐿2 distance between
200	the predicted patch-level representations and the target patchlevel representation in the masked parts. Formally, let
201	𝑐𝑖 = 𝑓ctx (𝑥\𝑀 ) , 𝑐̂𝑗 = 𝑔pred (𝑐),
202	𝑖
203
204	𝑡𝑗 = 𝑓tgt (𝑥)𝑗
205
206	(1)
207
208	where 𝑥 are the Mel-spectrogram patches, 𝑥\𝑀 are the visible
209	patches, 𝑓ctx and 𝑓tgt are the context and target encoders
210	respectively, and 𝑔pred the lightweight predictor. The loss is
211	then
212	1
213	ℒ=
214	∑ ‖̂
215	𝑐 − 𝑡𝑗 ‖22
216	(2)
217	|𝑀 | 𝑗∈𝑀 𝑗
218	We update 𝑓ctx and 𝑔pred parameters via backpropagation,
219	while 𝑓tgt parameters are updated as
220	𝜃tgt ← 𝜏 𝜃tgt + (1 − 𝜏 )𝜃ctx
221	(3)
222	with 𝜏 the EMA decay factor. This design stabilizes target
223	representation and prevents collapse.
224	C. Evaluation
225	Our assessment follows the eXtensive Audio Representation
226	and Evaluation Suite (X-ARES)², which brings together 21
227	publicly available audio datasets spanning a variety of tasks
228	and domains. Using the frozen target encoder in the evaluation, we employ two complementary evaluation strategies
229	drawn from X‑ARES:
230	a) Linear Probing (MLP): For each downstream task, we
231	freeze the pre-trained encoder and attach a single linear layer.
232	This classifier is trained on the task’s labeled data using a
233	²https://github.com/jimbozhang/xares
234
235	fixed set of hyperparameters. By holding the original model
236	weights constant, this procedure reveals how readily the
237	learned representations can be linearly separated and adapted
238	to new tasks.
239	b) k‑Nearest Neighbors (kNN): Without any additional
240	training, we directly apply a kNN classifier to the frozen
241	embeddings. This non‑parametric evaluation highlights the
242	raw discriminative power of the representations. Although it
243	may underperform more sophisticated fine‑tuning methods,
244	kNN offers a strict baseline for the intrinsic quality of the
245	learned features.
246	Due to the architecture and loss, the model’s outputs are
247	not guaranteed to be linearly separable as explained in the VJEPA paper [11]. Therefore, we do not expect good results in
248	that section. However, we should observe decent performance
249	in the kNN task.
250
251	IV. Implementation Details
252	In this section we summarize the key implementation choices
253	and hyperparameters used to train A‑JEPA. Tables 2–3 collect
254	the most important settings; for clarity we defer dataset splitting, and hardware details to Section V.
255	A. Data processing
256	We work with 1921982 AudioSet clips resampled to 32kHz
257	and 10s duration totaling to 5338 hours of audio. Each
258	waveform is converted to a 128‑band Mel‑spectrogram with
259	256 time bins (via a frame size and hop chosen accordingly
260	such that the frame size is 2.5 times the size of the hop).
261	Per example, we randomly sample 40 %–60 % (exact value
262	per batch is uniformly sampled) of the patches indices to
263	be masked. Each batch contains 256 audio clips. Preliminary
264	experiments showed the block masking strategy from I-JEPA
265	yielded lower performance than random masking.
266	B. Model Architecture
267	In Table I you can see the exact ViT hyperparameters used for
268	each module. The context and target encoders share the same
269	ViT configuration with 16×16 patches, a 768‑dimensional
270	embedding, 12 layers, 12 attention heads and an MLP ratio of
271	4.0. The target encoder is kept architecturally identical to the
272	context encoder and updated via EMA where the parameter
273	𝜏 is set in the same way as in BYOL [21]. The predictor uses
274	an embedding size of 384, a head count of 12 and contains 6
275	layers. MLP ratio remains the same at 4.0. The predictor reprojects the embeddings to 768 after going through the ViT
276	so that its outputs can to be compared with those of the target
277	encoder. The total number of trainable parameters during
278	training is 96.7M, with 85.4M parameters used at inference
279	since the predictor is not used.
280	C. Optimization and Scheduling
281	We train using AdamW [22] (𝛽1 = 0.9, 𝛽2 = 0.95, weight
282	decay = 0.05) and an initial learning rate of 3 ⋅ 10−4 . A
283	warmup‑cosine scheduler ramps from 1 ⋅ 10−6 to 3 ⋅ 10−4 over
284	1,000 steps, then anneals to zero.
285
286	TABLE I
287	Encoders and predictor architecture hyperparameters. The total
288	number of trainable parameters during training is 96.7M, with 85.4M
289	used at inference since the predictor is not needed.
290	Context / Target
291	Encoders
292
293	Predictor
294
295	16 × 16
296
297	-
298
299	Embedding dim
300
301	768
302
303	384
304
305	Depth
306
307	12
308
309	6
310
311	Heads
312
313	12
314
315	12
316
317	MLP ratio
318
319	4.0
320
321	4.0
322
323	Hyperparameter
324	Patch size
325
326	# Parameters
327
328	85.4M (each)
329
330	TABLE II
331	Linear-probing evaluation results on the X-ARES evaluation suite,
332	comparing our Audio-JEPA implementation with wav2vec 2.0 and
333	data2vec. Scores are reported as given by X-ARES; bold indicates
334	the best performance, and underline indicates the second-best.
335	Audio-JEPA
336	(ours)
337
338	Wav2Vec2 [5]
339
340	Data2Vec [7]
341
342	ASV2015 [23]
343
344	0.898
345
346	0.924
347
348	0.937
349
350	Clotho [24]
351
352	0.014
353
354	0.014
355
356	0.008
357
358	CREMA-D [25, 26]
359
360	0.427
361
362	0.541
363
364	0.523
365
366	DESED [27, 28]
367
368	0.306
369
370	0.313
371
372	0.136
373
374	ESC-50 [29]
375
376	0.338
377
378	0.510
379
380	0.229
381
382	Fluent Speech
383	Command [18]
384
385	0.025
386
387	0.468
388
389	0.978
390
391	Free Music
392	Archive small [30]
393
394	0.553
395
396	0.469
397
398	0.334
399
400	FSD18-Kaggle [31]
401
402	0.212
403
404	0.241
405
406	0.153
407
408	FSD50k [32]
409
410	0.151
411
412	0.166
413
414	0.085
415
416	GTZAN Genre [17]
417
418	0.628
419
420	0.630
421
422	0.448
423
424	LibriCount [33]
425
426	0.471
427
428	0.583
429
430	0.492
431
432	LibriSpeech-MF [34]
433
434	0.883
435
436	0.948
437
438	0.752
439
440	NSynth-Instruments
441	[35]
442
443	0.404
444
445	0.443
446
447	0.336
448
449	RAVDESS [36]
450
451	0.303
452
453	0.442
454
455	0.467
456
457	Speech
458	V1 [18]
459
460	0.152
461
462	0.714
463
464	0.927
465
466	UrbanSound 8k [15]
467
468	0.585
469
470	0.659
471
472	0.426
473
474	Vocal Imitation [37]
475
476	0.056
477
478	0.147
479
480	0.128
481
482	VocalSound [38]
483
484	0.526
485
486	0.768
487
488	0.803
489
490	VoxCeleb1 [16]
491
492	0.041
493
494	0.340
495
496	0.105
497
498	VoxLingua33 [39]
499
500	0.093
501
502	0.553
503
504	0.620
505
506	Dataset
507
508	11.3M
509
510	V. Experimental Setup
511	We train on 4 NVIDIA V100 GPUs, with a total batch size
512	of 256 clips. In other words, each batch contained somewhere
513	close to 42.7 minutes of audio. Training took place for
514	100,000 steps (~13 epochs) which took 14 hours to complete.
515	This training necessitated significantly less resources than
516	Wav2Vec2 Base [5] and data2vec [7] where each trained for
517	400k steps with bigger batches of 1.6h and 63 minutes of
518	audio respectively.
519
520	VI. Results
521	A. Linear-probe (MLP) performance
522	We first probe Audio-JEPA’s representations with a small
523	MLP head to assess their linear separability across tasks.
524	Table II reports Audio-JEPA’s performance on 20 of the 21 XARES datasets (all except LibriSpeech-100h [34]). The model
525	reaches first or second place on several benchmarks but falls
526	to last on roughly half of them. In particular, Audio-JEPA
527	underperforms significantly on Fluent Speech Commands
528	and Speech Commands V1. As noted in Section III.C.b,
529	Audio-JEPA is at a disadvantage under linear-probe evaluation as Audio-JEPA’s embedding space is not guaranteed to
530	be linearly separable due to the training objective favoring
531	embedding cohesion. As experimented in [11], using attentive
532	pooling might help in that situation.
533	B. kNN performance
534	We next evaluate pure embedding quality by k-nearestneighbor classification on the 16 X-ARES tasks compatible
535	with this probe. Table III shows that Audio-JEPA achieves
536	first place on 3 datasets (ESC-50, FMA-small, GTZAN)
537	and second place on 7 more, outperforming wav2vec 2.0
538	and data2vec despite using far less pre-training data and
539	resources. Conversely, it ranks last on about a third of the
540	tasks, underperforming substantially on those same datasets
541	as when evaluated with linear-probing. Because kNN probes
542	frozen embeddings directly and without extra classifier capacity, these results may better reflect the true representational
543	power of Audio-JEPA across music, environmental sounds
544	and speech domains.
545
546	Commands
547
548	C. Closing summary
549	Overall, Audio-JEPA demonstrates that a simple mask-prediction objective can yield high-quality audio embeddings with
550	far less pre-training data. Under kNN evaluation, AudioJEPA often matches or surpasses baselines on music and
551	general-sound tasks, even though those baselines underwent
552	far more extensive pre-training, confirming the strength of
553	latent-space prediction. Linear-probe results expose the limitations of a single-layer head for Audio-JEPA, but also
554	point to clear remedies (attentive pooling or small multilayer probes). Across both evaluation methods, Audio-JEPA is
555	weakest on tasks requiring fine-grained speech discrimination
556	(e.g. speaker verification and keyword spotting), indicating
557	that specialized data could improve these cases. These results
558	validate Audio-JEPA as a data-efficient foundation for audio
559	representation learning and set the stage for the architectural
560	and tuning improvements detailed in the Conclusion.
561
562	Audio-JEPA
563	(ours)
564
565	Wav2Vec2 [5]
566
567	Data2Vec [7]
568
569	rently untouched, is likely to uncover additional headroom.
570	By open‑sourcing our code and checkpoints, we hope to establish Audio‑JEPA as a solid starting point for the community
571	to explore these directions and to further unify JEPA research
572	across vision and now audio.
573
574	ASV2015 [23]
575
576	0.927
577
578	0.858
579
580	0.942
581
582	Acknowledgement
583
584	CREMA-D [25]
585
586	0.267
587
588	0.221
589
590	0.351
591
592	ESC-50 [29]
593
594	0.140
595
596	0.081
597
598	0.040
599
600	Fluent Speech
601	Command [18]
602
603	0.009
604
605	0.017
606
607	0.630
608
609	Free Music
610	Archive small [30]
611
612	0.449
613
614	0.251
615
616	0.106
617
618	GTZAN Genre [17]
619
620	0.452
621
622	0.303
623
624	0.108
625
626	LibriCount [33]
627
628	0.307
629
630	0.311
631
632	0.176
633
634	LibriSpeech-MF [34]
635
636	0.545
637
638	0.606
639
640	0.724
641
642	NSynth-Instruments
643	[35]
644
645	0.170
646
647	0.251
648
649	0.179
650
651	RAVDESS [36]
652
653	0.215
654
655	0.169
656
657	0.313
658
659	Speech
660	V1 [18]
661
662	0.044
663
664	0.208
665
666	0.852
667
668	UrbanSound 8k [15]
669
670	0.303
671
672	0.339
673
674	0.156
675
676	Vocal Imitation [37]
677
678	0.017
679
680	0.010
681
682	0.018
683
684	VocalSound [38]
685
686	0.256
687
688	0.269
689
690	0.308
691
692	VoxCeleb1 [16]
693
694	0.002
695
696	0.003
697
698	0.033
699
700	VoxLingua33 [39]
701
702	0.057
703
704	0.034
705
706	0.058
707
708	TABLE III
709	kNN evaluation results on the X-ARES evaluation suite, directly
710	comparing frozen embeddings from Audio-JEPA against wav2vec 2.0
711	and data2vec. Scores are reported as given by X-ARES; bold indicates the best performance, and underline indicates the second-best.
712	Dataset
713
714	Commands
715
716	VII. Conclusion
717	We introduced Audio‑JEPA, the first open‑source,
718	from‑scratch adaptation of the Joint‑Embedding Predictive Architecture to audio. Pre‑trained on AudioSet with
719	off‑the‑shelf hyper‑parameters, Audio‑JEPA delivers competitive performance on the X-ARES. The method compares
720	especially well under kNN evaluation, confirming that predicting masked latent targets, rather than the waveform or
721	discrete labels, is a useful inductive bias for general‑purpose
722	audio representation learning.
723	Looking forward, we identify three straightforward upgrades:
724	1) Attention‑pooling head. Replacing the single‑frame
725	MLP used in our linear‑probe evaluation with a
726	lightweight attention‑pooling block, as proposed for
727	V‑JEPA [11], could yield fairer comparisons and narrow the current linear‑probe gap.
728	2) Modern backbones and positional encodings. Swapping the vanilla ViT for recent audio transformers (e.g.
729	ConvFormer [40] or CAFormer [41]) and testing rotary
730	or conditional sine‑cosine encodings should improve
731	modelling of long‑range temporal cues.
732	3) Hyper‑parameter tuning. A systematic sweep of
733	mask ratio, EMA decay and optimizer settings, cur-
734
735	This work was granted access to the HPC resources of
736	IDRIS under the allocation AD011014754 made by GENCI.
737	Support from the ANR-3IA Artificial and Natural Intelligence
738	Toulouse Institute ANITI (ANR-19-PI3A-0004) is gratefully
739	acknowledged
740	References
741	[1]
742
743	W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov,
744	and A. Mohamed, “HuBERT: Self-Supervised Speech Representation
745	Learning by Masked Prediction of Hidden Units.” [Online]. Available:
746	http://arxiv.org/abs/2106.07447
747	[2] H. Dinkel, Y. Wang, Z. Yan, J. Zhang, and Y. Wang, “CED: Consistent
748	ensemble distillation for audio tagging.” [Online]. Available: http://
749	arxiv.org/abs/2308.11957
750	[3] H. Dinkel, Z. Yan, Y. Wang, J. Zhang, Y. Wang, and B. Wang,
751	“Scaling up masked audio encoder learning for general audio classification.” [Online]. Available: http://arxiv.org/abs/2406.06992
752	[4] D. Niizumi, D. Takeuchi, Y. Ohishi, N. Harada, and K. Kashino,
753	“Masked Modeling Duo: Towards a Universal Audio Pre-training
754	Framework.” [Online]. Available: http://arxiv.org/abs/2404.06095
755	[5] A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0:
756	A Framework for Self-Supervised Learning of Speech Representations.” [Online]. Available: http://arxiv.org/abs/2006.11477
757	[6] S. Chen et al., “WavLM: Large-Scale Self-Supervised Pre-Training
758	for Full Stack Speech Processing,” IEEE Journal of Selected Topics
759	in Signal Processing, vol. 16, no. 6, pp. 1505–1518, Oct. 2022, doi:
760	10.1109/JSTSP.2022.3188113.
761	[7] A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli, “data2vec:
762	A General Framework for Self-supervised Learning in Speech, Vision
763	and Language.” [Online]. Available: http://arxiv.org/abs/2202.03555
764	[8] A. Baevski, A. Babu, W.-N. Hsu, and M. Auli, “Efficient Self-supervised Learning with Contextualized Target Representations for Vision,
765	Speech and Language.” [Online]. Available: http://arxiv.org/abs/2212.
766	07525
767	[9] Y. LeCun, “A Path Towards Autonomous Machine Intelligence Version
768	0.9.2, 2022-06-27,” Jun. 2022.
769	[10] M. Assran et al., “Self-Supervised Learning from Images with a JointEmbedding Predictive Architecture.” [Online]. Available: http://arxiv.
770	org/abs/2301.08243
771	[11] A. Bardes et al., “Revisiting Feature Prediction for Learning Visual
772	Representations from Video.” [Online]. Available: http://arxiv.org/abs/
773	2404.08471
774	[12] Z. Fei, M. Fan, and J. Huang, “A-JEPA: Joint-Embedding Predictive
775	Architecture Can Listen.” [Online]. Available: http://arxiv.org/abs/2311.
776	15830
777	[13] A. Riou, S. Lattner, G. Hadjeres, M. Anslow, and G. Peeters,
778	“Stem-JEPA: A Joint-Embedding Predictive Architecture for Musical
779	Stem Compatibility Estimation.” [Online]. Available: http://arxiv.org/
780	abs/2408.02514
781	[14] P.-Y. Huang et al., “Masked Autoencoders that Listen.” [Online]. Available: http://arxiv.org/abs/2207.06405
782	[15] J. Salamon, C. Jacoby, and J. P. Bello, “A Dataset and Taxonomy
783	for Urban Sound Research,” in Proceedings of the 22nd ACM international conference on Multimedia, in MM '14. New York, NY, USA:
784	Association for Computing Machinery, Nov. 2014, pp. 1041–1044. doi:
785	10.1145/2647868.2655045.
786	[16] A. Nagrani, J. S. Chung, and A. Zisserman, “VoxCeleb: a large-scale
787	speaker identification dataset,” in Interspeech 2017, Aug. 2017, pp.
788	2616–2620. doi: 10.21437/Interspeech.2017-950.
789
790	[17] B. L. Sturm, “The GTZAN dataset: Its contents, its faults,
791	their effects on evaluation, and its future use,” Journal of New
792	Music Research, vol. 43, no. 2, pp. 147–172, Apr. 2014, doi:
793	10.1080/09298215.2014.894533.
794	[18] P. Warden, “Speech Commands: A Dataset for Limited-Vocabulary
795	Speech Recognition.” [Online]. Available: http://arxiv.org/abs/1804.
796	03209
797	[19] K. H. N. Vo, D. P. T. Nguyen, T. Nguyen, and T. T. Quan, “TIJEPA: An Innovative Energy-based Joint Embedding Strategy for TextImage Multimodal Systems.” [Online]. Available: http://arxiv.org/abs/
798	2503.06380
799	[20] T. Lundqvist and L. Delvret, “GeoJEPA: Towards Eliminating Augmentation- and Sampling Bias in Multimodal Geospatial Learning.” [Online]. Available: http://arxiv.org/abs/2503.05774
800	[21] J.-B. Grill et al., “Bootstrap your own latent: A new approach to selfsupervised Learning.” [Online]. Available: http://arxiv.org/abs/2006.
801	07733
802	[22] I. Loshchilov and F. Hutter, “Decoupled Weight Decay Regularization.” [Online]. Available: http://arxiv.org/abs/1711.05101
803	[23] Z. Wu, T. Kinnunen, N. Evans, and J. Yamagishi, “Automatic Speaker
804	Verification Spoofing and Countermeasures Challenge (ASVspoof
805	2015) Database.” [Online]. Available: https://datashare.ed.ac.uk/handle/
806	10283/853
807	[24] K. Drossos, S. Lipping, and T. Virtanen, “Clotho: An Audio Captioning
808	Dataset.” [Online]. Available: http://arxiv.org/abs/1910.09387
809	[25] H. Cao, D. G. Cooper, M. K. Keutmann, R. C. Gur, A. Nenkova, and
810	R. Verma, “CREMA-D: Crowd-sourced Emotional Multimodal Actors
811	Dataset,” IEEE transactions on affective computing, vol. 5, no. 4, pp.
812	377–390, 2014, doi: 10.1109/TAFFC.2014.2336244.
813	[26] M. K. Keutmann, S. L. Moore, A. Savitt, and R. C. Gur, “Generating
814	an item pool for translational social cognition research: methodology
815	and initial validation,” Behavior Research Methods, vol. 47, no. 1, pp.
816	228–234, Mar. 2015, doi: 10.3758/s13428-014-0464-0.
817	[27] N. Turpault, R. Serizel, A. P. Shah, and J. Salamon, “Sound event
818	detection in domestic environments with weakly labeled data and
819	soundscape synthesis,” presented at the Workshop on Detection and
820	Classification of Acoustic Scenes and Events, 2019. Accessed: May
821	07, 2025. [Online]. Available: https://inria.hal.science/hal-02160855
822	[28] R. Serizel, N. Turpault, A. Shah, and J. Salamon, “Sound event
823	detection in synthetic domestic environments,” presented at the ICASSP
824	2020 - 45th International Conference on Acoustics, Speech, and Signal
825	Processing, May 2020. Accessed: May 07, 2025. [Online]. Available:
826	https://inria.hal.science/hal-02355573
827	[29] K. J. Piczak, “ESC: Dataset for Environmental Sound Classification,”
828	in Proceedings of the 23rd ACM international conference on Multimedia, in MM '15. New York, NY, USA: Association for Computing
829	Machinery, Oct. 2015, pp. 1015–1018. doi: 10.1145/2733373.2806390.
830	[30] M. Defferrard, K. Benzi, P. Vandergheynst, and X. Bresson, “FMA: A
831	Dataset For Music Analysis.” [Online]. Available: http://arxiv.org/abs/
832	1612.01840
833	[31] E. Fonseca et al., “General-purpose Tagging of Freesound Audio with
834	AudioSet Labels: Task Description, Dataset, and Baseline.” [Online].
835	Available: http://arxiv.org/abs/1807.09902
836	[32] E. Fonseca, X. Favory, J. Pons, F. Font, and X. Serra, “FSD50K: An
837	Open Dataset of Human-Labeled Sound Events.” [Online]. Available:
838	http://arxiv.org/abs/2010.00475
839	[33] F.-R. Stöter, S. Chakrabarty, E. Habets, and B. Edler, “LibriCount,
840	a dataset for speaker count estimation.” [Online]. Available: https://
841	zenodo.org/records/1216072
842	[34] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech:
843	An ASR corpus based on public domain audio books,” in 2015
844	IEEE International Conference on Acoustics, Speech and Signal
845	Processing (ICASSP), Apr. 2015, pp. 5206–5210. doi: 10.1109/
846	ICASSP.2015.7178964.
847	[35] J. Engel et al., “Neural Audio Synthesis of Musical Notes with WaveNet
848	Autoencoders.” [Online]. Available: http://arxiv.org/abs/1704.01279
849	[36] S. R. Livingstone and F. A. Russo, “The Ryerson Audio-Visual Database
850	of Emotional Speech and Song (RAVDESS).” [Online]. Available:
851	https://zenodo.org/records/1188976
852	[37] “Vocal Imitation Set v1.1.3 : Thousands of vocal imitations of hundreds
853	of sounds from the AudioSet ontology.” [Online]. Available: https://
854	zenodo.org/records/1340763
855
856	[38] Y. Gong, J. Yu, and J. Glass, “Vocalsound: A Dataset for Improving Human Vocal Sounds Recognition,” in ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2022, pp. 151–155. doi: 10.1109/
857	ICASSP43922.2022.9746828.
858	[39] N. Yadong, “voxlingua33 in WebDataset Format.” [Online]. Available:
859	https://zenodo.org/records/14723799
860	[40] X. Lin, Z. Yan, X. Deng, C. Zheng, and L. Yu, “ConvFormer:
861	Plug-and-Play CNN-Style Transformers for Improving Medical Image
862	Segmentation.” [Online]. Available: http://arxiv.org/abs/2309.05674
863	[41] W. Yu et al., “MetaFormer Baselines for Vision.” [Online]. Available:
864	http://arxiv.org/abs/2210.13452
865
866

Audio-JEPA: Joint-Embedding Predictive
Architecture for Audio Representation Learning
Ludovic TUNCAY ¹

Etienne LABBE ¹

Emmanouil BENETOS ²

Thomas PELLEGRINI ¹

ludovic.tuncay@irit.fr

etienne.labbe@irit.fr

emmanouil.benetos@qmul.ac.uk

thomas.pellegrini@irit.fr

¹ IRIT, Université de Toulouse, CNRS, Toulouse INP, Toulouse, France
² School of Electronic Engineering and Computer Science, Queen Mary University of London, UK

Abstract—Building on the Joint-Embedding Predictive Architecture (JEPA) paradigm, a recent self-supervised learning
framework that predicts latent representations of masked regions in high-level feature spaces, we propose Audio‑JEPA (Audio Joint‑Embedding Predictive Architecture), tailored specifically for audio data. Audio-JEPA uses a simple Vision Transformer backbone to predict latent representations of masked
spectrogram patches rather than reconstructing raw audio.
We pre‑train on unlabeled AudioSet clips (10s, 32kHz) with
random patch masking on mel‑spectrograms. We evaluate on
the X‑ARES suite covering speech, music, and environmental‐
sound tasks. Although our implementation is a straightforward
translation of the original model to audio, the results still show
comparable performance to wav2vec 2.0 and data2vec while
using less than one‑fifth of their training data and with no
hyper‑parameter tuning. All code and pretrained checkpoints
will be released on GitHub¹.
Index Terms—Self-supervised learning, audio representation,
joint-embedding predictive architecture, Audio-JEPA, AudioSet

I. Introduction
Self-Supervised Learning (SSL) has revolutionized representation learning for speech and audio, enabling models to
learn from unlabeled data and excel in diverse downstream
tasks [1, 2, 3, 4]. Early SSL approaches for audio, such as
contrastive predictive coding and wav2vec 2.0, learned latent
speech representations by masking the input and solving a
contrastive task over latent codes [5]. Follow-up methods like
HuBERT [1] introduced offline clustering to generate pseudolabels for masked audio segments and WavLM [6] applied
data augmentation and denoising to improve robustness in
speech representation learning. More recently, latent prediction approaches have gained traction: data2vec [7] and its
efficient successor data2vec 2.0 [8] employ a teacher–student
framework to predict contextualized latent representations of
the input, achieving strong results across vision, speech, and
language tasks. In the audio domain, Niizumi et al. introduced
Masked Modeling Duo (M2D) [4], which uses two networks
(online and momentum encoder) to predict masked patch
embeddings and attained state-of-the-art results on numerous
audio benchmarks.
In computer vision, a new paradigm called Joint-Embedding Predictive Architecture (JEPA) [9, 10, 11] has been
proposed to predict hidden content in a high-level latent space
instead of pixel space. Notably, the image-based I-JEPA [10]
model demonstrated that predicting representations of masked
¹https://github.com/LudovicTuncay/Audio-JEPA

image regions can yield powerful visual features. The JEPA
approach differs from prior masked reconstruction methods by
focusing on semantic latent prediction rather than lower-level
signal reconstruction. Inspired by these advances, an audio
version of JEPA (termed A-JEPA) [12] was recently described
by Fei et al. Their A-JEPA encodes a spectrogram “context”
part and predicts the latent representations of masked “target”
regions using a momentum-updated target encoder. During
pre-training, they anneal the mask from fully random toward a
spec-augment-style structured scheme. In contrast, our AudioJEPA sticks with purely random masking throughout for
simplicity and maximal generality. At the time of writing,
no official implementation or checkpoints for A-JEPA are
available, motivating our from-scratch development. In the
musical domain, Stem-JEPA [13] adapts the JEPA paradigm
to multi-track recordings by jointly training an encoder and
predictor to forecast embeddings of compatible instrument
stems. While utilizing the JEPA backbone, Stem-JEPA differs
methodologically as it masks entire instrument stems instead
of individual spectrogram patches.
Our work targets the ICME 2025 Audio Encoder Capability
Challenge, where the goal is to learn general audio representations that perform well across a broad suite of tasks. We
present our Audio-JEPA implementation - developed from
scratch following the I-JEPA paradigm - and benchmark it on
the challenge’s eXtensive Audio Representation and Evaluation Suite (X-ARES). Our contributions include: (1) adapting
the JEPA masked latent prediction framework to audio spectrogram inputs using a Vision Transformer (ViT) backbone;
(2) an extensive evaluation against prior self-supervised audio
models on standard downstream tasks, assessing Audio-JEPA
via both linear probing and k-nearest-neighbor evaluation.

II. Related Work
A. Self-Supervised Audio Representation Learning
Early work in SSL for audio focused on predicting future or
missing parts of the waveform. wav2vec 2.0 [5] pioneered
masking in the latent speech representations and training
the model to identify the true quantized latent of a masked
segment among distractors (a contrastive loss). This approach
enabled models to learn rich speech features and achieved
remarkable results on speech recognition with limited labeled data. Building on this, HuBERT (Hidden-Unit BERT)
[1] introduced a BERT-like masked prediction where the
model predicts cluster assignments of masked audio frames.

Fig. 1. A‑JEPA architecture. Mel‑spectrogram patches are split into visible and masked sets. A context encoder embeds visible patches, a lightweight
predictor reconstructs masked‑patch embeddings, and a momentum‑updated target encoder provides targets. Training minimizes average L2 (Euclidean)
distance. The dashed arrow denotes a stop‑gradient.

HuBERT uses an offline k-means on acoustic features to
provide target labels, and by iteratively refining these labels, it
learns high-level speech units, matching wav2vec 2.0 performance on ASR benchmarks. Facebook’s data2vec presented
a modality-general SSL approach: instead of contrastive or
classification targets, data2vec [7] trains a student network
to regress the contextualized embeddings produced by a
teacher network (an Exponential Moving Average (EMA) of
the student) for masked portions of the input. data2vec 2.0
[8] improved the efficiency of this method by not encoding
masked tokens and using a lightweight decoder, achieving
similar accuracies to Masked Autoencoders [14] in a fraction
of the training time and matching wav2vec 2.0 on speech tasks
with over a 10× speedup. These latent regression approaches
eliminate the need for discrete targets and have set strong
baselines in audio.
B. Masked Prediction with Dual Networks
The use of two networks (online/target) for masked prediction
has also been explored in specialized audio SSL methods.
M2D (Masked Modeling Duo) [4] employs an online network
that sees the unmasked patches and a momentum target
network that encodes only the masked patches. The online
network predicts the target network’s representation of the
masked region, encouraging both networks to effectively
model the input. This design, inspired by Masked Autoencoders but working in representation space, led M2D to
state-of-the-art results on a range of audio classification
tasks (environmental sound, speaker ID, music genre, etc.).
Notably, M2D achieved top performance on datasets like
UrbanSound8K [15], VoxCeleb1 [16], GTZAN [17], and
SpeechCommands [18] with a single universal model. Such
results highlight the power of using latent prediction instead
of raw signal reconstruction for learning transferable audio
features. Other contemporary models include WavLM [6],
which extended HuBERT with simulated noisy inputs and
achieved strong results on both speech recognition and classification tasks.

C. Joint-Embedding Predictive Architectures
Rather than predicting low-level details of masked inputs,
JEPA methods aim to predict higher-level representations.
The image-based I-JEPA [10] demonstrated that a ViT can
learn excellent representations by predicting the latent representations of masked image patches, as opposed to generating
pixels. By operating in the feature space, I-JEPA forces the
model to capture abstract semantic information and ignore
minute pixel-level differences. The concept has since been
extended to other modalities and combinations (e.g., TIJEPA [19] for text–image, GeoJEPA [20] for geospatial
data, etc.), showing JEPA’s flexibility. For audio, Fei et al.
recently proposed A-JEPA [12], applying the same principle
to spectrogram inputs. While A‑JEPA and M2D both adopt
a dual‑network masked‑prediction framework, they differ in
how the target encoder is applied: M2D processes only the
masked spectrogram patches, whereas JEPA processes the
entire spectrogram (context + masked). This richer context
enables more detailed representations of the masked regions.
Their design uses a context encoder to process unmasked
spectrogram patches and a target encoder (the EMA of the
context network) to encode masked regions, with a lightweight
predictor network aligning the two in latent space. Our work is
directly inspired by this approach but contrary to M2D, JEPA
does not require data augmentation and the whole spectrogram
is seen by the target encoder. We evaluate Audio-JEPA in
our experiments, underlining how it bridges the gap between
vision-style masked modeling and audio understanding.

III. Proposed Method: JEPA for Audio
In this section, we describe our adaptation of the JEPA paradigm to the audio domain, which we call Audio-JEPA. We first
present the overall architecture, then detail the self‑supervised
training objectives, and finally highlight the audio‑specific
design choices that make A‑JEPA effective on diverse sound
data.

A. Overall architecture
As shown in the Fig. 1, our Audio-JEPA model consists of
three main modules:
1) Context encoder: Processes the “visible” subset of
Mel-spectrogram patches
2) Target Encoder: Provides stable target embeddings via
an Exponential Moving Average (EMA) of the context
encoder’s parameters
3) Lightweight Predictor Network: Takes context embeddings and predicts latent representations for masked
(“target”) patches
Upon converting an input waveform to a Mel‑spectrogram and
partitioning it into non‑overlapping time–frequency patches,
we randomly mask a fixed proportion of patches. The context
encoder embeds the remaining visible patches, producing a
context representation. The lightweight predictor network then
reconstructs embeddings for the masked patches. In parallel,
the target encoder (updated by EMA rather than gradient
descent) encodes the true masked patches. Training minimizes
the average L2 distance between the predictor’s outputs and
the target encoder’s embeddings, with stop‑gradient applied
between the predictor and the target encoder. This implementation is a direct adaptation of I-JEPA to the audio domain,
by considering the spectrogram as a single channel, possibly
non-square, image.
B. Training objective
We train Audio-JEPA using the average 𝐿2 distance between
the predicted patch-level representations and the target patchlevel representation in the masked parts. Formally, let
𝑐𝑖 = 𝑓ctx (𝑥\𝑀 ) , 𝑐̂𝑗 = 𝑔pred (𝑐),
𝑖

𝑡𝑗 = 𝑓tgt (𝑥)𝑗

(1)

where 𝑥 are the Mel-spectrogram patches, 𝑥\𝑀 are the visible
patches, 𝑓ctx and 𝑓tgt are the context and target encoders
respectively, and 𝑔pred the lightweight predictor. The loss is
then
1
ℒ=
∑ ‖̂
𝑐 − 𝑡𝑗 ‖22
(2)
|𝑀 | 𝑗∈𝑀 𝑗
We update 𝑓ctx and 𝑔pred parameters via backpropagation,
while 𝑓tgt parameters are updated as
𝜃tgt ← 𝜏 𝜃tgt + (1 − 𝜏 )𝜃ctx
(3)
with 𝜏 the EMA decay factor. This design stabilizes target
representation and prevents collapse.
C. Evaluation
Our assessment follows the eXtensive Audio Representation
and Evaluation Suite (X-ARES)², which brings together 21
publicly available audio datasets spanning a variety of tasks
and domains. Using the frozen target encoder in the evaluation, we employ two complementary evaluation strategies
drawn from X‑ARES:
a) Linear Probing (MLP): For each downstream task, we
freeze the pre-trained encoder and attach a single linear layer.
This classifier is trained on the task’s labeled data using a
²https://github.com/jimbozhang/xares

fixed set of hyperparameters. By holding the original model
weights constant, this procedure reveals how readily the
learned representations can be linearly separated and adapted
to new tasks.
b) k‑Nearest Neighbors (kNN): Without any additional
training, we directly apply a kNN classifier to the frozen
embeddings. This non‑parametric evaluation highlights the
raw discriminative power of the representations. Although it
may underperform more sophisticated fine‑tuning methods,
kNN offers a strict baseline for the intrinsic quality of the
learned features.
Due to the architecture and loss, the model’s outputs are
not guaranteed to be linearly separable as explained in the VJEPA paper [11]. Therefore, we do not expect good results in
that section. However, we should observe decent performance
in the kNN task.

IV. Implementation Details
In this section we summarize the key implementation choices
and hyperparameters used to train A‑JEPA. Tables 2–3 collect
the most important settings; for clarity we defer dataset splitting, and hardware details to Section V.
A. Data processing
We work with 1921982 AudioSet clips resampled to 32kHz
and 10s duration totaling to 5338 hours of audio. Each
waveform is converted to a 128‑band Mel‑spectrogram with
256 time bins (via a frame size and hop chosen accordingly
such that the frame size is 2.5 times the size of the hop).
Per example, we randomly sample 40 %–60 % (exact value
per batch is uniformly sampled) of the patches indices to
be masked. Each batch contains 256 audio clips. Preliminary
experiments showed the block masking strategy from I-JEPA
yielded lower performance than random masking.
B. Model Architecture
In Table I you can see the exact ViT hyperparameters used for
each module. The context and target encoders share the same
ViT configuration with 16×16 patches, a 768‑dimensional
embedding, 12 layers, 12 attention heads and an MLP ratio of
4.0. The target encoder is kept architecturally identical to the
context encoder and updated via EMA where the parameter
𝜏 is set in the same way as in BYOL [21]. The predictor uses
an embedding size of 384, a head count of 12 and contains 6
layers. MLP ratio remains the same at 4.0. The predictor reprojects the embeddings to 768 after going through the ViT
so that its outputs can to be compared with those of the target
encoder. The total number of trainable parameters during
training is 96.7M, with 85.4M parameters used at inference
since the predictor is not used.
C. Optimization and Scheduling
We train using AdamW [22] (𝛽1 = 0.9, 𝛽2 = 0.95, weight
decay = 0.05) and an initial learning rate of 3 ⋅ 10−4 . A
warmup‑cosine scheduler ramps from 1 ⋅ 10−6 to 3 ⋅ 10−4 over
1,000 steps, then anneals to zero.

TABLE I
Encoders and predictor architecture hyperparameters. The total
number of trainable parameters during training is 96.7M, with 85.4M
used at inference since the predictor is not needed.
Context / Target
Encoders

Predictor

16 × 16

-

Embedding dim

768

384

Depth

12

6

Heads

12

12

MLP ratio

4.0

4.0

Hyperparameter
Patch size

# Parameters

85.4M (each)

TABLE II
Linear-probing evaluation results on the X-ARES evaluation suite,
comparing our Audio-JEPA implementation with wav2vec 2.0 and
data2vec. Scores are reported as given by X-ARES; bold indicates
the best performance, and underline indicates the second-best.
Audio-JEPA
(ours)

Wav2Vec2 [5]

Data2Vec [7]

ASV2015 [23]

0.898

0.924

0.937

Clotho [24]

0.014

0.014

0.008

CREMA-D [25, 26]

0.427

0.541

0.523

DESED [27, 28]

0.306

0.313

0.136

ESC-50 [29]

0.338

0.510

0.229

Fluent Speech
Command [18]

0.025

0.468

0.978

Free Music
Archive small [30]

0.553

0.469

0.334

FSD18-Kaggle [31]

0.212

0.241

0.153

FSD50k [32]

0.151

0.166

0.085

GTZAN Genre [17]

0.628

0.630

0.448

LibriCount [33]

0.471

0.583

0.492

LibriSpeech-MF [34]

0.883

0.948

0.752

NSynth-Instruments
[35]

0.404

0.443

0.336

RAVDESS [36]

0.303

0.442

0.467

Speech
V1 [18]

0.152

0.714

0.927

UrbanSound 8k [15]

0.585

0.659

0.426

Vocal Imitation [37]

0.056

0.147

0.128

VocalSound [38]

0.526

0.768

0.803

VoxCeleb1 [16]

0.041

0.340

0.105

VoxLingua33 [39]

0.093

0.553

0.620

Dataset

11.3M

V. Experimental Setup
We train on 4 NVIDIA V100 GPUs, with a total batch size
of 256 clips. In other words, each batch contained somewhere
close to 42.7 minutes of audio. Training took place for
100,000 steps (~13 epochs) which took 14 hours to complete.
This training necessitated significantly less resources than
Wav2Vec2 Base [5] and data2vec [7] where each trained for
400k steps with bigger batches of 1.6h and 63 minutes of
audio respectively.

VI. Results
A. Linear-probe (MLP) performance
We first probe Audio-JEPA’s representations with a small
MLP head to assess their linear separability across tasks.
Table II reports Audio-JEPA’s performance on 20 of the 21 XARES datasets (all except LibriSpeech-100h [34]). The model
reaches first or second place on several benchmarks but falls
to last on roughly half of them. In particular, Audio-JEPA
underperforms significantly on Fluent Speech Commands
and Speech Commands V1. As noted in Section III.C.b,
Audio-JEPA is at a disadvantage under linear-probe evaluation as Audio-JEPA’s embedding space is not guaranteed to
be linearly separable due to the training objective favoring
embedding cohesion. As experimented in [11], using attentive
pooling might help in that situation.
B. kNN performance
We next evaluate pure embedding quality by k-nearestneighbor classification on the 16 X-ARES tasks compatible
with this probe. Table III shows that Audio-JEPA achieves
first place on 3 datasets (ESC-50, FMA-small, GTZAN)
and second place on 7 more, outperforming wav2vec 2.0
and data2vec despite using far less pre-training data and
resources. Conversely, it ranks last on about a third of the
tasks, underperforming substantially on those same datasets
as when evaluated with linear-probing. Because kNN probes
frozen embeddings directly and without extra classifier capacity, these results may better reflect the true representational
power of Audio-JEPA across music, environmental sounds
and speech domains.

Commands

C. Closing summary
Overall, Audio-JEPA demonstrates that a simple mask-prediction objective can yield high-quality audio embeddings with
far less pre-training data. Under kNN evaluation, AudioJEPA often matches or surpasses baselines on music and
general-sound tasks, even though those baselines underwent
far more extensive pre-training, confirming the strength of
latent-space prediction. Linear-probe results expose the limitations of a single-layer head for Audio-JEPA, but also
point to clear remedies (attentive pooling or small multilayer probes). Across both evaluation methods, Audio-JEPA is
weakest on tasks requiring fine-grained speech discrimination
(e.g. speaker verification and keyword spotting), indicating
that specialized data could improve these cases. These results
validate Audio-JEPA as a data-efficient foundation for audio
representation learning and set the stage for the architectural
and tuning improvements detailed in the Conclusion.

Audio-JEPA
(ours)

Wav2Vec2 [5]

Data2Vec [7]

rently untouched, is likely to uncover additional headroom.
By open‑sourcing our code and checkpoints, we hope to establish Audio‑JEPA as a solid starting point for the community
to explore these directions and to further unify JEPA research
across vision and now audio.

ASV2015 [23]

0.927

0.858

0.942

Acknowledgement

CREMA-D [25]

0.267

0.221

0.351

ESC-50 [29]

0.140

0.081

0.040

Fluent Speech
Command [18]

0.009

0.017

0.630

Free Music
Archive small [30]

0.449

0.251

0.106

GTZAN Genre [17]

0.452

0.303

0.108

LibriCount [33]

0.307

0.311

0.176

LibriSpeech-MF [34]

0.545

0.606

0.724

NSynth-Instruments
[35]

0.170

0.251

0.179

RAVDESS [36]

0.215

0.169

0.313

Speech
V1 [18]

0.044

0.208

0.852

UrbanSound 8k [15]

0.303

0.339

0.156

Vocal Imitation [37]

0.017

0.010

0.018

VocalSound [38]

0.256

0.269

0.308

VoxCeleb1 [16]

0.002

0.003

0.033

VoxLingua33 [39]

0.057

0.034

0.058

TABLE III
kNN evaluation results on the X-ARES evaluation suite, directly
comparing frozen embeddings from Audio-JEPA against wav2vec 2.0
and data2vec. Scores are reported as given by X-ARES; bold indicates the best performance, and underline indicates the second-best.
Dataset

Commands

VII. Conclusion
We introduced Audio‑JEPA, the first open‑source,
from‑scratch adaptation of the Joint‑Embedding Predictive Architecture to audio. Pre‑trained on AudioSet with
off‑the‑shelf hyper‑parameters, Audio‑JEPA delivers competitive performance on the X-ARES. The method compares
especially well under kNN evaluation, confirming that predicting masked latent targets, rather than the waveform or
discrete labels, is a useful inductive bias for general‑purpose
audio representation learning.
Looking forward, we identify three straightforward upgrades:
1) Attention‑pooling head. Replacing the single‑frame
MLP used in our linear‑probe evaluation with a
lightweight attention‑pooling block, as proposed for
V‑JEPA [11], could yield fairer comparisons and narrow the current linear‑probe gap.
2) Modern backbones and positional encodings. Swapping the vanilla ViT for recent audio transformers (e.g.
ConvFormer [40] or CAFormer [41]) and testing rotary
or conditional sine‑cosine encodings should improve
modelling of long‑range temporal cues.
3) Hyper‑parameter tuning. A systematic sweep of
mask ratio, EMA decay and optimizer settings, cur-

This work was granted access to the HPC resources of
IDRIS under the allocation AD011014754 made by GENCI.
Support from the ANR-3IA Artificial and Natural Intelligence
Toulouse Institute ANITI (ANR-19-PI3A-0004) is gratefully
acknowledged
References
[1]

W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov,
and A. Mohamed, “HuBERT: Self-Supervised Speech Representation
Learning by Masked Prediction of Hidden Units.” [Online]. Available:
http://arxiv.org/abs/2106.07447
[2] H. Dinkel, Y. Wang, Z. Yan, J. Zhang, and Y. Wang, “CED: Consistent
ensemble distillation for audio tagging.” [Online]. Available: http://
arxiv.org/abs/2308.11957
[3] H. Dinkel, Z. Yan, Y. Wang, J. Zhang, Y. Wang, and B. Wang,
“Scaling up masked audio encoder learning for general audio classification.” [Online]. Available: http://arxiv.org/abs/2406.06992
[4] D. Niizumi, D. Takeuchi, Y. Ohishi, N. Harada, and K. Kashino,
“Masked Modeling Duo: Towards a Universal Audio Pre-training
Framework.” [Online]. Available: http://arxiv.org/abs/2404.06095
[5] A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0:
A Framework for Self-Supervised Learning of Speech Representations.” [Online]. Available: http://arxiv.org/abs/2006.11477
[6] S. Chen et al., “WavLM: Large-Scale Self-Supervised Pre-Training
for Full Stack Speech Processing,” IEEE Journal of Selected Topics
in Signal Processing, vol. 16, no. 6, pp. 1505–1518, Oct. 2022, doi:
10.1109/JSTSP.2022.3188113.
[7] A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli, “data2vec:
A General Framework for Self-supervised Learning in Speech, Vision
and Language.” [Online]. Available: http://arxiv.org/abs/2202.03555
[8] A. Baevski, A. Babu, W.-N. Hsu, and M. Auli, “Efficient Self-supervised Learning with Contextualized Target Representations for Vision,
Speech and Language.” [Online]. Available: http://arxiv.org/abs/2212.
07525
[9] Y. LeCun, “A Path Towards Autonomous Machine Intelligence Version
0.9.2, 2022-06-27,” Jun. 2022.
[10] M. Assran et al., “Self-Supervised Learning from Images with a JointEmbedding Predictive Architecture.” [Online]. Available: http://arxiv.
org/abs/2301.08243
[11] A. Bardes et al., “Revisiting Feature Prediction for Learning Visual
Representations from Video.” [Online]. Available: http://arxiv.org/abs/
2404.08471
[12] Z. Fei, M. Fan, and J. Huang, “A-JEPA: Joint-Embedding Predictive
Architecture Can Listen.” [Online]. Available: http://arxiv.org/abs/2311.
15830
[13] A. Riou, S. Lattner, G. Hadjeres, M. Anslow, and G. Peeters,
“Stem-JEPA: A Joint-Embedding Predictive Architecture for Musical
Stem Compatibility Estimation.” [Online]. Available: http://arxiv.org/
abs/2408.02514
[14] P.-Y. Huang et al., “Masked Autoencoders that Listen.” [Online]. Available: http://arxiv.org/abs/2207.06405
[15] J. Salamon, C. Jacoby, and J. P. Bello, “A Dataset and Taxonomy
for Urban Sound Research,” in Proceedings of the 22nd ACM international conference on Multimedia, in MM '14. New York, NY, USA:
Association for Computing Machinery, Nov. 2014, pp. 1041–1044. doi:
10.1145/2647868.2655045.
[16] A. Nagrani, J. S. Chung, and A. Zisserman, “VoxCeleb: a large-scale
speaker identification dataset,” in Interspeech 2017, Aug. 2017, pp.
2616–2620. doi: 10.21437/Interspeech.2017-950.

[17] B. L. Sturm, “The GTZAN dataset: Its contents, its faults,
their effects on evaluation, and its future use,” Journal of New
Music Research, vol. 43, no. 2, pp. 147–172, Apr. 2014, doi:
10.1080/09298215.2014.894533.
[18] P. Warden, “Speech Commands: A Dataset for Limited-Vocabulary
Speech Recognition.” [Online]. Available: http://arxiv.org/abs/1804.
03209
[19] K. H. N. Vo, D. P. T. Nguyen, T. Nguyen, and T. T. Quan, “TIJEPA: An Innovative Energy-based Joint Embedding Strategy for TextImage Multimodal Systems.” [Online]. Available: http://arxiv.org/abs/
2503.06380
[20] T. Lundqvist and L. Delvret, “GeoJEPA: Towards Eliminating Augmentation- and Sampling Bias in Multimodal Geospatial Learning.” [Online]. Available: http://arxiv.org/abs/2503.05774
[21] J.-B. Grill et al., “Bootstrap your own latent: A new approach to selfsupervised Learning.” [Online]. Available: http://arxiv.org/abs/2006.
07733
[22] I. Loshchilov and F. Hutter, “Decoupled Weight Decay Regularization.” [Online]. Available: http://arxiv.org/abs/1711.05101
[23] Z. Wu, T. Kinnunen, N. Evans, and J. Yamagishi, “Automatic Speaker
Verification Spoofing and Countermeasures Challenge (ASVspoof
2015) Database.” [Online]. Available: https://datashare.ed.ac.uk/handle/
10283/853
[24] K. Drossos, S. Lipping, and T. Virtanen, “Clotho: An Audio Captioning
Dataset.” [Online]. Available: http://arxiv.org/abs/1910.09387
[25] H. Cao, D. G. Cooper, M. K. Keutmann, R. C. Gur, A. Nenkova, and
R. Verma, “CREMA-D: Crowd-sourced Emotional Multimodal Actors
Dataset,” IEEE transactions on affective computing, vol. 5, no. 4, pp.
377–390, 2014, doi: 10.1109/TAFFC.2014.2336244.
[26] M. K. Keutmann, S. L. Moore, A. Savitt, and R. C. Gur, “Generating
an item pool for translational social cognition research: methodology
and initial validation,” Behavior Research Methods, vol. 47, no. 1, pp.
228–234, Mar. 2015, doi: 10.3758/s13428-014-0464-0.
[27] N. Turpault, R. Serizel, A. P. Shah, and J. Salamon, “Sound event
detection in domestic environments with weakly labeled data and
soundscape synthesis,” presented at the Workshop on Detection and
Classification of Acoustic Scenes and Events, 2019. Accessed: May
07, 2025. [Online]. Available: https://inria.hal.science/hal-02160855
[28] R. Serizel, N. Turpault, A. Shah, and J. Salamon, “Sound event
detection in synthetic domestic environments,” presented at the ICASSP
2020 - 45th International Conference on Acoustics, Speech, and Signal
Processing, May 2020. Accessed: May 07, 2025. [Online]. Available:
https://inria.hal.science/hal-02355573
[29] K. J. Piczak, “ESC: Dataset for Environmental Sound Classification,”
in Proceedings of the 23rd ACM international conference on Multimedia, in MM '15. New York, NY, USA: Association for Computing
Machinery, Oct. 2015, pp. 1015–1018. doi: 10.1145/2733373.2806390.
[30] M. Defferrard, K. Benzi, P. Vandergheynst, and X. Bresson, “FMA: A
Dataset For Music Analysis.” [Online]. Available: http://arxiv.org/abs/
1612.01840
[31] E. Fonseca et al., “General-purpose Tagging of Freesound Audio with
AudioSet Labels: Task Description, Dataset, and Baseline.” [Online].
Available: http://arxiv.org/abs/1807.09902
[32] E. Fonseca, X. Favory, J. Pons, F. Font, and X. Serra, “FSD50K: An
Open Dataset of Human-Labeled Sound Events.” [Online]. Available:
http://arxiv.org/abs/2010.00475
[33] F.-R. Stöter, S. Chakrabarty, E. Habets, and B. Edler, “LibriCount,
a dataset for speaker count estimation.” [Online]. Available: https://
zenodo.org/records/1216072
[34] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech:
An ASR corpus based on public domain audio books,” in 2015
IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP), Apr. 2015, pp. 5206–5210. doi: 10.1109/
ICASSP.2015.7178964.
[35] J. Engel et al., “Neural Audio Synthesis of Musical Notes with WaveNet
Autoencoders.” [Online]. Available: http://arxiv.org/abs/1704.01279
[36] S. R. Livingstone and F. A. Russo, “The Ryerson Audio-Visual Database
of Emotional Speech and Song (RAVDESS).” [Online]. Available:
https://zenodo.org/records/1188976
[37] “Vocal Imitation Set v1.1.3 : Thousands of vocal imitations of hundreds
of sounds from the AudioSet ontology.” [Online]. Available: https://
zenodo.org/records/1340763

[38] Y. Gong, J. Yu, and J. Glass, “Vocalsound: A Dataset for Improving Human Vocal Sounds Recognition,” in ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2022, pp. 151–155. doi: 10.1109/
ICASSP43922.2022.9746828.
[39] N. Yadong, “voxlingua33 in WebDataset Format.” [Online]. Available:
https://zenodo.org/records/14723799
[40] X. Lin, Z. Yan, X. Deng, C. Zheng, and L. Yu, “ConvFormer:
Plug-and-Play CNN-Style Transformers for Improving Medical Image
Segmentation.” [Online]. Available: http://arxiv.org/abs/2309.05674
[41] W. Yu et al., “MetaFormer Baselines for Vision.” [Online]. Available:
http://arxiv.org/abs/2210.13452


Here's the full technical breakdown:

---

## Architecture

**Context / Target Encoders** (identical architecture):
- ViT-Base with **16x16 patches**
- Embedding dim: **768**
- Depth: **12 layers**
- Heads: **12**
- MLP ratio: **4.0**
- **85.4M params** each

**Predictor** (lightweight):
- Embedding dim: **384**
- Depth: **6 layers**
- Heads: **12**
- MLP ratio: **4.0**
- Re-projects output back to 768 for comparison with target encoder
- **11.3M params**

Total trainable: **96.7M** (context encoder + predictor). Target encoder is EMA-only, not gradient-updated. At inference only the target encoder is used (**85.4M params**).

---

## Masking Strategy

- **Random masking** (not block masking — they explicitly tested I-JEPA's block masking and found it hurt performance on audio)
- Mask ratio: **40%–60%**, uniformly sampled per batch
- Applied to mel-spectrogram patches

This is a key divergence from I-JEPA (vision), which uses structured block masking. The prior A-JEPA paper by Fei et al. annealed from random toward SpecAugment-style structured masking; Audio-JEPA keeps it purely random throughout for simplicity.

---

## Loss Function

**Average L2 (Euclidean) distance** between predicted and target patch embeddings:

```
L = (1/|M|) * Σ_{j∈M} ||ĉ_j - t_j||²
```

Where:
- `ĉ_j = g_pred(f_ctx(x\M))` — predictor output for masked patch j
- `t_j = f_tgt(x)_j` — target encoder output for patch j
- Stop-gradient on the target encoder side

No VICReg, no contrastive loss, no auxiliary regularization — just pure L2 regression. Collapse is prevented by the EMA + stop-gradient design (same mechanism as BYOL).

---

## Training Details

- **Optimizer:** AdamW (β₁=0.9, β₂=0.95, weight decay=0.05)
- **LR:** warmup-cosine, ramps from 1e-6 → 3e-4 over 1,000 steps, then cosine anneals to 0
- **Batch size:** 256 clips (~42.7 min of audio per batch)
- **Steps:** 100,000 (~13 epochs)
- **Hardware:** 4x NVIDIA V100
- **Wall time:** ~14 hours
- **EMA schedule:** Same as BYOL (starts lower, ramps up to near 1.0)

For comparison, wav2vec 2.0 and data2vec trained for **400k steps** with much larger effective batches (1.6h and 63min per batch respectively). Audio-JEPA uses **<1/5th the training data**.

---

## Audio Preprocessing

- **Sample rate:** 32kHz
- **Mel bands:** 128
- **Time bins:** 256
- Frame size = **2.5x hop size** (exact values not given, but chosen to yield 128×256 spectrograms from 10s clips)
- **Dataset:** 1,921,982 AudioSet clips, 10s each = **5,338 hours**
- Spectrogram treated as a single-channel (possibly non-square) image

---

## Key Results

**kNN evaluation** (the fairer comparison for JEPA — see below):

| Task | Audio-JEPA | wav2vec 2.0 | data2vec |
|------|-----------|-------------|----------|
| ESC-50 | **0.140** | 0.081 | 0.040 |
| FMA-small | **0.449** | 0.251 | 0.106 |
| GTZAN | **0.452** | 0.303 | 0.108 |
| ASV2015 | 0.927 | 0.858 | **0.942** |
| UrbanSound8k | 0.303 | **0.339** | 0.156 |

Audio-JEPA wins 3/16 kNN tasks, places 2nd on 7 more. It dominates on **music and environmental sound**, but is weak on **fine-grained speech tasks** (speaker verification, keyword spotting).

**Linear probe** results are weaker — the authors explicitly note this is expected because JEPA's latent space is **not guaranteed to be linearly separable** (same finding as V-JEPA). The embeddings cluster well (good for kNN) but don't linearly separate well.

---

## Limitations & Weaknesses

1. **Linear probe performance is mediocre** — embedding space isn't linearly separable. Fix: attention pooling head (as proposed for V-JEPA)
2. **Weak on speech tasks** — especially speaker verification (VoxCeleb1: 0.002 kNN), keyword spotting (Speech Commands V1: 0.044 kNN). Likely needs speech-specific data
3. **No hyperparameter tuning done** — mask ratio, EMA decay, optimizer settings are all off-the-shelf from I-JEPA
4. **Vanilla ViT backbone** — they suggest ConvFormer/CAFormer + rotary/conditional positional encodings would help

---

## Relevance to Your LeWM TTS Work

A few takeaways:

1. **Random masking > block masking for audio** — this is a direct finding. If your JEPA uses block masking, might be worth testing random.

2. **L2 loss in latent space works** — no need for contrastive losses or VICReg-style regularization. EMA + stop-gradient is sufficient to prevent collapse. This validates a simple training objective.

3. **Data efficiency is real** — 5,338 hours with 100k steps competitive with models trained on 5x more. Good signal for your 20-hour single-speaker setup, though you'd likely need to adapt the architecture significantly for that scale.

4. **Speech is the weak spot** — JEPA representations are good for music/environmental sound but struggle with fine-grained speech discrimination. For TTS, you'll want to consider whether your encoder needs to capture speaker identity / phonetic detail that vanilla JEPA might not prioritize.

5. **The predictor architecture matters** — they use a 6-layer, 384-dim predictor that re-projects to 768. The asymmetry (smaller predictor, larger encoder) is deliberate and matches findings from BYOL/I-JEPA.


see only thing i want to do with jepa is i just want to prove ..i can crretae a small model whoch is lesser than 80m or euals 80m and which have decene no of speakrr 10 to 20 ... and it works on edge and it is scaleble in prod ... more concurnecy and minimal ttfb and a good rtf


Got it. So the goal is pragmatic — not research novelty, but a proof-of-concept that JEPA-based TTS can be:

- **≤80M params**
- **10-20 speakers**
- **Edge-deployable** (phone, embedded)
- **Production-scalable** (high concurrency, low TTFB, good RTF)

That's a solid thesis to prove. Here's where Audio-JEPA's findings map to your constraints and where they don't:

## The Budget Problem

Audio-JEPA's encoder alone is 85.4M — already over your limit. You need to shrink. Realistic split for 80M total:

| Component | Rough Budget |
|---|---|
| JEPA encoder (representation) | ~30-35M |
| Decoder (latent → mel/audio) | ~30-35M |
| Speaker embedding table (20 spkrs) | ~negligible |
| Vocoder (if separate) | ~10-15M |

That encoder means something like ViT-Small (384 dim, 6 layers, 6 heads) instead of ViT-Base. Still within the range where JEPA has been shown to work — the Audio-JEPA paper didn't even tune hyperparams, so a well-tuned small model could punch above its weight.

## What Actually Matters for Your Goals

**TTFB** — This is your hardest constraint with JEPA. JEPA encoders process the full context before predicting. For low TTFB you need:
- Chunked/streaming inference — encode text in one shot (cheap), then stream audio chunks
- The JEPA encoder is on the **representation learning** side, not necessarily in the inference hot path. If you use JEPA for pretraining and then distill or use the encoder as a frozen feature extractor at train time only, inference TTFB is just your decoder

**RTF** — ViT-Small with 384 dim is very fast on edge. Quantize to INT8, you're looking at single-digit ms per chunk on modern phone NPUs. The key is keeping the decoder lightweight — something like a small ConvNext or a few WaveRNN-style layers, not a big diffusion model.

**Concurrency** — Small model = small memory footprint = more instances per GPU in prod. 80M params at FP16 = ~160MB. You could run 30+ concurrent streams on a single T4. At INT8, double that.

**Multi-speaker** — 10-20 speakers is trivial. A learned embedding table (256-dim × 20 speakers = 5K params) injected via FiLM conditioning or cross-attention into the decoder. The JEPA encoder stays speaker-agnostic — it learns the audio structure. Speaker identity is the decoder's job.

## The Architecture That Proves Your Point

```
Text → small text encoder (frozen or tiny)
         ↓
   JEPA-pretrained audio encoder (ViT-S, ~30M)
   [used at TRAINING time to provide targets]
         ↓
   Lightweight causal decoder (~35M)
   + speaker embedding (FiLM)
         ↓
   mel chunks (streamed)
         ↓
   small vocoder (~10-15M, e.g. MB-MelGAN / HiFi-GAN-tiny)
```

At **inference**, the JEPA encoder isn't even in the loop if you design it right — it was used to train the decoder's target representations. Inference is just: text encoder → decoder → vocoder. That's your TTFB and RTF story.

## What You'd Actually Need to Show

To prove the thesis, you'd want benchmarks on:

1. **MOS** (or MUSHRA) on 10-20 speakers — "decent quality"
2. **RTF < 1.0** on target edge device (phone CPU/NPU)
3. **TTFB < 200ms** for first audio chunk
4. **Concurrent streams per GPU** in a simulated prod setup
5. **Model size** on disk after quantization (target: <100MB)

That's a compelling demo. The JEPA part is the training story (data-efficient, no need for massive labeled data, learns good representations from unlabeled audio). The edge/prod part is the architecture story (small, streamable, quantization-friendly).

Want me to sketch out a more concrete architecture with exact layer configs that fit the 80M budget?


1 issue i tried jepa lewm i cant produce same speaker sound at all at end of traiing


That's the core problem. JEPA's latent space is optimized for **semantic/structural prediction**, not **speaker fidelity**. Audio-JEPA's own results confirm this — their worst scores are on speaker-related tasks (VoxCeleb1 kNN: 0.002, basically random).

The L2 loss in latent space actively **discards** speaker-specific detail because it's "noise" from the perspective of masked patch prediction. The model learns "what sound goes here" not "what does *this person* sound like."

## Why It Happens

JEPA's training objective:

```
minimize ||predicted_latent - target_latent||²
```

This pushes the encoder to capture **what's being said** and **acoustic structure**, but speaker timbre, pitch contour, vocal tract characteristics — those are consistent across all patches, so they're redundant for patch prediction and get suppressed. The EMA target encoder smooths them out further.

It's the same reason JEPA works great for ESC-50/GTZAN (what sound?) but fails at VoxCeleb (who's speaking?).

## Fixes — Pick One or Stack Them

**1. Speaker-conditioned predictor (cheapest fix)**

Don't let the JEPA predictor ignore speaker. Feed speaker embedding directly into the predictor:

```
predictor_input = [context_patches; speaker_emb]
```

Now the predictor *must* use speaker identity to minimize L2 — because two speakers saying the same word produce different target latents, and the only way to disambiguate is the speaker embedding. This forces speaker info into the representation.

**2. Multi-scale loss**

JEPA's L2 on final-layer features is too abstract. Add a shallow loss:

```
L = L2(pred_deep, target_deep) + λ * L2(pred_shallow, target_shallow)
```

Pull targets from layer 2-3 of the target encoder (not just the last layer). Early layers retain more acoustic/speaker detail. data2vec does this — averages across layers. JEPA only uses the final layer, which is why it loses fine-grained info.

**3. Auxiliary speaker consistency loss**

After the encoder, pool across time and add:

```
L_spk = contrastive_loss(pool(encoder(utt_1)), pool(encoder(utt_2)))
```

Where utt_1 and utt_2 are from the same speaker. This explicitly tells the encoder "preserve who's speaking." Lightweight — just a projection head + InfoNCE on the pooled representation.

**4. Don't use JEPA encoder at inference (cleanest)**

Use JEPA **only** for pretraining. Then:
- Freeze the JEPA encoder
- Train a separate small decoder that takes text + speaker_emb and is supervised by **mel targets directly** (not JEPA latents)
- The JEPA encoder serves as a feature extractor during decoder training (e.g., providing phonetic/prosody targets), but the decoder learns to reconstruct actual speaker-specific mel spectrograms

This way JEPA does what it's good at (structure) and the decoder handles fidelity.

## What I'd Actually Do for Your Case

Given ≤80M, 10-20 speakers, and the speaker reproduction problem:

**Option 4 + Option 2.**

- Pretrain a small JEPA encoder (~30M) with multi-scale L2 loss (layers 3, 6, 9 averaged) so it retains more acoustic detail
- Freeze it
- Train a causal decoder (~35M) conditioned on speaker embeddings, supervised on **real mel spectrograms**, using JEPA features as an auxiliary input (cross-attention or concatenation)
- The decoder owns speaker fidelity. JEPA owns structure/prosody

The JEPA encoder doesn't need to reproduce speaker sound — it just needs to provide useful representations. The decoder does the reconstruction, and it gets speaker identity directly.

Want me to look at your current LeWM architecture to see where exactly the speaker signal is getting lost?


yeah look


Agent

Explore LeWM TTS codebase

Prompt:
Thoroughly explore the LeWM TTS codebase in /home/ubuntu. Find:

1. The main model architecture files — encoder, decoder, predictor, any JEPA-related components
2. How speaker conditioning is implemented (speaker embeddings, FiLM, cross-attention, etc.)
3. The training loop and loss functions
4. How the JEPA encoder/predictor/target encoder interact
5. Any config files showing model dimensions, layers, etc.
6. The inference/synthesis pipeline

Look for directories like lewm*, tts*, model*, src*, etc. Check for Python files with model definitions, training scripts, config files (yaml, json, toml). Be very thorough — check all subdirectories.