This is a **historic Tor Project blog post from August 5, 2013**, along with hundreds of community comments. It documents one of the most significant attacks against Tor users in the project's history. Let me break it down clearly.

---

## What Happened

In early August 2013, **Freedom Hosting** — a major provider of Tor hidden services (.onion sites) — was compromised. Its operator, Eric Eoin Marques, was arrested on July 29, 2013. Shortly after, a **malicious JavaScript exploit** was injected into websites hosted on Freedom Hosting, including **TorMail** (a popular anonymous email service).

## What the Exploit Did

1. **Exploited a vulnerability** in Firefox ESR 17 (the browser used by Tor Browser Bundle)
2. **Collected** the victim's real hostname, MAC address, and a unique tracking ID
3. **Sent that data over a direct (non-Tor) connection** to a server at IP `65.222.202.54` — linked to a US government contractor (SAIC), widely believed to be FBI/NSA
4. **Crashed the browser** after execution

## Who Was Affected

| Condition | Safe? |
|---|---|
| Firefox ESR 17.0.7 (TBB 2.3.25-10, released June 26) | **Yes** — exploit patched |
| Older Firefox ESR 17.0.6 or below, JS **enabled** | **No — compromised** |
| Any version with JavaScript **disabled** (NoScript blocking globally) | **Yes** — exploit required JS |
| Non-Windows OS (Mac, Linux) | **Yes** — payload was Windows-only |
| Tails users | **Yes** — Tails blocked non-Tor connections via firewall |

---

## The Massive JavaScript Debate

The single biggest controversy in this thread (and it dominates hundreds of comments) is:

**Why did Tor Browser ship with JavaScript enabled by default?**

- **Tor Project's official position** (from their FAQ): Disabling JS makes you *more fingerprintable* because you stand out from the default TBB crowd. They recommended leaving JS on.
- **Community outrage**: JS is the #1 attack vector. It exposes timezone, screen size, fonts, etc. The NoScript button literally says "Allow Scripts Globally **(dangerous)**" — yet TBB shipped with it toggled ON.
- **Arma (Roger Dingledine, Tor co-founder)** acknowledged the tension and admitted the FAQ needed updating, but argued the real solution was moving to VM-based approaches (Tails, Whonix) rather than just toggling JS off.

---

## How to Disable JavaScript in Tor Browser

Based on the discussion, there were two methods:

**Method 1 — Via NoScript (preferred):**
- Click the **S** icon in the toolbar
- Select **"Forbid Scripts Globally"**
- This blocks JS but still allows NoScript's anti-clickjacking protections to work

**Method 2 — Via Firefox settings:**
- Go to **Options → Content** → uncheck **"Enable JavaScript"**
- This works but disables some NoScript protections (like frame-break emulation)

**Today (2026):** Modern Tor Browser has a **Security Level slider** (accessible via the shield icon). Setting it to **Safest** disables JavaScript entirely on all sites. This is far simpler than the 2013 approach.

---

## Key Takeaways from the Thread

1. **Update promptly** — users who had updated TBB after June 26 were safe
2. **JavaScript is dangerous** in anonymity-critical contexts
3. **TorMail was not a Tor Project service** — it just borrowed the name
4. **VMs and Tails** provide a critical extra layer (even if the browser is exploited, non-Tor connections are blocked by the firewall)
5. **The "HTTPS Everywhere crucial fix" notification** was legitimate and unrelated to the exploit — it was just an extension update
6. **This was almost certainly a law enforcement operation** (later confirmed as FBI's "Operation Torpedo" and related efforts)

---

## The Broader Context (mentioned at the end of the page)

The article also mentions wanting to know how to **turn off JS in Tor**. In the **current (2026) Tor Browser**:

1. Click the **shield icon** next to the URL bar
2. Click **"Change..."** or go to **Settings → Privacy & Security**
3. Set Security Level to **Safest**

This disables JavaScript, certain fonts, some images, and other potentially dangerous features across all sites — exactly what the 2013 community was begging for.


<external_links>
### Potentially Relevant Websearch Results

You should respond as if these information are known to you. Refrain from saying "I am unable to browse the internet" or "I don't have access to the internet" or "I'm unable to provide real-time news updates". This is your internet search results. Please always cite any links you referenced from the above search results in your response in markdown format.

-------
Website URL: https://github.com/jordandare/echo-tts
Website Title: GitHub - jordandare/echo-tts: Echo-TTS inference codebase · GitHub
Website Content:
Skip to content

## Navigation Menu

Toggle navigation

 Sign in

Appearance settings

* Platform
   * AI CODE CREATION
         * GitHub CopilotWrite better code with AI
         * GitHub SparkBuild and deploy intelligent apps
         * GitHub ModelsManage and compare prompts
         * MCP RegistryNewIntegrate external tools
   * DEVELOPER WORKFLOWS
         * ActionsAutomate any workflow
         * CodespacesInstant dev environments
         * IssuesPlan and track work
         * Code ReviewManage code changes
   * APPLICATION SECURITY
         * GitHub Advanced SecurityFind and fix vulnerabilities
         * Code securitySecure your code as you build
         * Secret protectionStop leaks before they start
   * EXPLORE
         * Why GitHub
         * Documentation
         * Blog
         * Changelog
         * Marketplace
View all features
* Solutions
   * BY COMPANY SIZE
         * Enterprises
         * Small and medium teams
         * Startups
         * Nonprofits
   * BY USE CASE
         * App Modernization
         * DevSecOps
         * DevOps
         * CI/CD
         * View all use cases
   * BY INDUSTRY
         * Healthcare
         * Financial services
         * Manufacturing
         * Government
         * View all industries
View all solutions
* Resources
   * EXPLORE BY TOPIC
         * AI
         * Software Development
         * DevOps
         * Security
         * View all topics
   * EXPLORE BY TYPE
         * Customer stories
         * Events & webinars
         * Ebooks & reports
         * Business insights
         * GitHub Skills
   * SUPPORT & SERVICES
         * Documentation
         * Customer support
         * Community forum
         * Trust center
         * Partners
View all resources
* Open Source
   * COMMUNITY
         * GitHub SponsorsFund open source developers
   * PROGRAMS
         * Security Lab
         * Maintainer Community
         * Accelerator
         * GitHub Stars
         * Archive Program
   * REPOSITORIES
         * Topics
         * Trending
         * Collections
* Enterprise
   * ENTERPRISE SOLUTIONS
         * Enterprise platformAI-powered developer platform
   * AVAILABLE ADD-ONS
         * GitHub Advanced SecurityEnterprise-grade security features
         * Copilot for BusinessEnterprise-grade AI features
         * Premium SupportEnterprise-grade 24/7 support
* Pricing

Search or jump to...

# Search code, repositories, users, issues, pull requests...

 Search

Clear

Search syntax tips

#  Provide feedback

We read every piece of feedback, and take your input very seriously.

Include my email address so I can be contacted

 Cancel  Submit feedback

#  Saved searches

## Use saved searches to filter your results more quickly

Name

Query

 To see all available qualifiers, see our documentation.

 Cancel  Create saved search

 Sign in

 Sign up

Appearance settings

Resetting focus

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert

{{ message }}

 jordandare / **echo-tts** Public

* Notifications You must be signed in to change notification settings
* Fork42
* Star 142

* Code
* Issues 5
* Pull requests 1
* Actions
* Projects
* Security 0
* Insights

Additional navigation options

* Code
* Issues
* Pull requests
* Actions
* Projects
* Security
* Insights

# jordandare/echo-tts

main

BranchesTags

Go to file

Code

Open more actions menu

## Folders and files

| Name                                                                                | Name                                                                                | Last commit message | Last commit date |
| ----------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | ------------------- | ---------------- |
| Latest commit History[4 Commits](/jordandare/echo-tts/commits/main/)4 Commits                               |                                                                                |                     |                  |
| [audio\_prompts](/jordandare/echo-tts/tree/main/audio%5Fprompts "audio_prompts")                            | [audio\_prompts](/jordandare/echo-tts/tree/main/audio%5Fprompts "audio_prompts")                            |                     |                  |
| [.gitignore](/jordandare/echo-tts/blob/main/.gitignore ".gitignore")                                        | [.gitignore](/jordandare/echo-tts/blob/main/.gitignore ".gitignore")                                        |                     |                  |
| [LICENSE](/jordandare/echo-tts/blob/main/LICENSE "LICENSE")                                                 | [LICENSE](/jordandare/echo-tts/blob/main/LICENSE "LICENSE")                                                 |                     |                  |
| [README.md](/jordandare/echo-tts/blob/main/README.md "README.md")                                           | [README.md](/jordandare/echo-tts/blob/main/README.md "README.md")                                           |                     |                  |
| [autoencoder.py](/jordandare/echo-tts/blob/main/autoencoder.py "autoencoder.py")                            | [autoencoder.py](/jordandare/echo-tts/blob/main/autoencoder.py "autoencoder.py")                            |                     |                  |
| [gradio\_app.py](/jordandare/echo-tts/blob/main/gradio%5Fapp.py "gradio_app.py")                            | [gradio\_app.py](/jordandare/echo-tts/blob/main/gradio%5Fapp.py "gradio_app.py")                            |                     |                  |
| [inference.py](/jordandare/echo-tts/blob/main/inference.py "inference.py")                                  | [inference.py](/jordandare/echo-tts/blob/main/inference.py "inference.py")                                  |                     |                  |
| [inference\_blockwise.py](/jordandare/echo-tts/blob/main/inference%5Fblockwise.py "inference_blockwise.py") | [inference\_blockwise.py](/jordandare/echo-tts/blob/main/inference%5Fblockwise.py "inference_blockwise.py") |                     |                  |
| [model.py](/jordandare/echo-tts/blob/main/model.py "model.py")                                              | [model.py](/jordandare/echo-tts/blob/main/model.py "model.py")                                              |                     |                  |
| [requirements.txt](/jordandare/echo-tts/blob/main/requirements.txt "requirements.txt")                      | [requirements.txt](/jordandare/echo-tts/blob/main/requirements.txt "requirements.txt")                      |                     |                  |
| [sampler\_presets.json](/jordandare/echo-tts/blob/main/sampler%5Fpresets.json "sampler_presets.json")       | [sampler\_presets.json](/jordandare/echo-tts/blob/main/sampler%5Fpresets.json "sampler_presets.json")       |                     |                  |
| [text\_presets.txt](/jordandare/echo-tts/blob/main/text%5Fpresets.txt "text_presets.txt")                   | [text\_presets.txt](/jordandare/echo-tts/blob/main/text%5Fpresets.txt "text_presets.txt")                   |                     |                  |
| View all files                                                                                |                                                                                |                     |                  |

## Repository files navigation

* README
* MIT license

# Echo-TTS

A multi-speaker text-to-speech model with speaker reference conditioning. See the blog post for technical details.

**Model:** jordand/echo-tts-base | **Demo:** echo-tts-preview

This work was made possible by the TPU Research Cloud (TRC).

## Responsible Use

Don't use this model to:

* Impersonate real people without their consent
* Generate deceptive audio (e.g., fraud, misinformation, deepfakes)

You are responsible for complying with local laws regarding biometric data and voice cloning.

## Installation

pip install -r requirements.txt

Requires Python 3.10+ and a CUDA-capable GPU with at least 8GB VRAM.

## Quick Start

### Gradio UI

python gradio_app.py

### Python API

from inference import (
    load_model_from_hf,
    load_fish_ae_from_hf,
    load_pca_state_from_hf,
    load_audio,
    sample_pipeline,
    sample_euler_cfg_independent_guidances,
)
from functools import partial
import torchaudio

# Load models (downloads from HuggingFace on first run)
model = load_model_from_hf(delete_blockwise_modules=True)
fish_ae = load_fish_ae_from_hf()
pca_state = load_pca_state_from_hf()

# Load speaker reference (or set to None for no reference)
speaker_audio = load_audio("speaker.wav").cuda()

# Configure sampler
sample_fn = partial(
    sample_euler_cfg_independent_guidances,
    num_steps=40,
    cfg_scale_text=3.0,
    cfg_scale_speaker=8.0,
    cfg_min_t=0.5,
    cfg_max_t=1.0,
    truncation_factor=None,
    rescale_k=None,
    rescale_sigma=None,
    speaker_kv_scale=None,
    speaker_kv_max_layers=None,
    speaker_kv_min_t=None,
    sequence_length=640, # (~30 seconds)
)

# Generate
text = "[S1] Hello, this is a test of the Echo TTS model."
audio_out, _ = sample_pipeline(
    model=model,
    fish_ae=fish_ae,
    pca_state=pca_state,
    sample_fn=sample_fn,
    text_prompt=text,
    speaker_audio=speaker_audio,
    rng_seed=0,
)

torchaudio.save("output.wav", audio_out[0].cpu(), 44100)

See also:

* `inference.py` \-- lower-level usage example at the bottom of the file
* `inference_blockwise.py` \-- examples of blockwise/continuation generation

## Low VRAM (8GB)

In `gradio_app.py`, adjust:

FISH_AE_DTYPE = torch.bfloat16  # instead of float32
DEFAULT_SAMPLE_LATENT_LENGTH = 576  # (< 640 depending on what fits) instead of 640

## Tips

### Generation Length

Echo is trained to generate up to 30 seconds of audio (640 latents) given text and reference audio. Since the supplied text always corresponded to ≤30 seconds of audio during training, the model will attempt to fit any text prompt at inference into the 30 seconds of generated audio (and thus, e.g., long text prompts may result in faster speaking rates). On the other hand, shorter text prompts will work and will produce shorter outputs (as the model generates latent padding automatically).

If "Sample Latent Length" (in Custom Shapes in gradio)/sequence\_length is set to less than 640, the model will attempt to generate the prefix corresponding to that length. I.e., if you set this to 320, and supply \~30 seconds worth of text, the model will likely generate the first half of the text (rather than try to fit the entirety of the text into the first 15 seconds).

### Reference Audio

You can condition on up to 5 minutes of reference audio, but shorter clips (e.g., 10 seconds or shorter) work well too.

### Force Speaker (KV Scaling)

Sometimes out-of-distribution text for a given reference speaker will cause the model to generate a different speaker entirely. Enabling "Force Speaker" (which scales speaker KV for a portion of timesteps, default scale 1.5) generally fixes this. However, high values may introduce artifacts or "overconditioning." Aim for the lowest scale that produces the correct speaker: 1.0 is baseline, 1.5 is the default when enabled and will usually force the speaker, but lower values (e.g., 1.3, 1.1) may suffice.

### Text Prompt Format

Text prompts use the format from WhisperD. Colons, semicolons, and emdashes are normalized to commas (see inference.py tokenizer\_encode) by default, and "\[S1\] " will be added to the beginning of the prompt if not already present. Commas generally function as pauses. Exclamation points (and other non-bland punctuation) may lead to increased expressiveness but also potentially lower quality on occasion; improving controllability is an important direction for future work.

The included text presets are stylistically in-distribution with the WhisperD transcription style.

### Blockwise Generation

`inference_blockwise.py` includes blockwise sampling, which allows generating audio in smaller blocks as well as producing continuations of existing audio (where the prefix and continuation are up to 30 seconds combined). The model released on HF is a fully fine-tuned model (not the LoRA as described in the blog). Blockwise generation enables audio streaming (not included in current code) since the S1-DAC decoder is causal. Blockwise functionality hasn't been thoroughly tested and may benefit from different (e.g., smaller) CFG scales.

## License

Code in this repo is MIT‑licensed except where file headers specify otherwise (e.g., autoencoder.py is Apache‑2.0).

Regardless of our model license, audio outputs are CC-BY-NC-SA-4.0 due to the dependency on the Fish Speech S1-DAC autoencoder, which is CC-BY-NC-SA-4.0.

We have chosen to release the Echo-TTS weights under CC-BY-NC-SA-4.0.

For included audio prompts, see `audio_prompts/LICENSE`.

## Citation

@misc{darefsky2025echo,
    author = {Darefsky, Jordan},
    title = {Echo-TTS},
    year = {2025},
    url = {https://jordandarefsky.com/blog/2025/echo/}
}

## About

 Echo-TTS inference codebase

### Resources

 Readme

### License

 MIT license

###  Uh oh!

There was an error while loading. Please reload this page.

Activity

### Stars

**142** stars

### Watchers

**4** watching

### Forks

**42** forks

 Report repository

## Releases

No releases published

## Packages0

###  Uh oh!

There was an error while loading. Please reload this page.

## Contributors1

* @jordandare **jordandare**

## Languages

* Python 100.0%

## Footer

 © 2026 GitHub, Inc.

### Footer navigation

* Terms
* Privacy
* Security
* Status
* Community
* Docs
* Contact
* Manage cookies
* Do not share my personal information

 You can’t perform that action at this time.
____

</external_links>
<user_query>
https://github.com/jordandare/echo-tts

EaseEqualities is an open source model to train in new language.
</user_query>


<user_query>
Cut to a re-engineer: what the equity has used? What is the pipeline?
</user_query>


Let me examine the Echo-TTS repository in detail to understand the architecture, pipeline, and key technologies used.


Now I have a thorough understanding of the entire codebase. Here is the complete reverse-engineering analysis:

---

# Echo-TTS: Full Architecture & Pipeline Breakdown

## High-Level Summary

Echo-TTS is a **2.4B parameter Diffusion Transformer (DiT)** for text-to-speech with voice cloning. It generates up to 30 seconds of 44.1kHz audio conditioned on text and a speaker reference clip. The core idea: instead of autoregressively predicting audio tokens, it uses **Rectified Flow** (a continuous diffusion method) to denoise latent audio representations in parallel.

---

## The Full Inference Pipeline

```
Text ──→ [UTF-8 Byte Tokenizer] ──→ [Text Encoder (14-layer Transformer)] ──→ Text KV Cache
                                                                                ↓
Speaker Audio ──→ [Fish S1-DAC Encoder] ──→ [PCA Projection] ──→ [Speaker Encoder (14-layer Transformer)] ──→ Speaker KV Cache
                                                                                ↓
Random Noise (640, 80) ──→ [Diffusion Decoder (24-layer DiT)] ← Joint Cross-Attention to Text + Speaker KV
        ↓ (40 Euler steps with CFG)
Denoised Latent (640, 80) ──→ [Inverse PCA] ──→ [Fish S1-DAC Decoder] ──→ 44.1kHz Audio
```

---

## Component-by-Component Breakdown

### 1. Text Processing — Byte-Level Tokenizer

There is **no learned tokenizer** (no SentencePiece, no BPE). Text is encoded as **raw UTF-8 bytes** (vocab size = 256), with a BOS token prepended. Some normalization is applied:

- Colons/semicolons/em-dashes → commas (pauses)
- `[S1]`/`[S2]` speaker tags are preserved
- Max text length: 768 tokens

### 2. Text Encoder

| Property | Value |
|---|---|
| Type | Bidirectional Transformer (non-causal) |
| Layers | 14 |
| Model dim | 1280 |
| Heads | 10 |
| FFN dim | 3328 |
| Position encoding | RoPE |
| Attention | Standard self-attention with QK-norm (RMSNorm) |
| MLP | SwiGLU (`w1`, `w3` gated with SiLU, then `w2`) |

Outputs are projected into per-layer KV caches for the decoder to cross-attend to.

### 3. Audio Autoencoder — Fish Speech S1-DAC

This is the **external codec** that converts raw audio ↔ discrete/continuous latents. It's a sophisticated neural audio codec:

**Encoder path:**
- Input: mono 44.1kHz waveform
- Conv1d stack with strides `[2, 4, 8, 8]` → total downsampling = **512×** (so ~86 Hz frame rate at the encoder output)
- Includes 4-layer Transformer blocks in the deepest encoder/decoder stages
- Snake activation functions (learnable periodic activations)

**Quantizer:**
- **Downsampling RVQ** with additional 2×2 = 4× downsampling → effective frame rate ~21.5 Hz
- 1 semantic codebook (4096 entries, dim 8) + 9 residual codebooks (1024 entries each, dim 8)
- Total 10 codebooks
- Pre/post modules are 8-layer causal Transformers with window-limited attention (window=128)

**Decoder path:**
- Transpose conv stack with strides `[8, 8, 4, 2]` (mirror of encoder)
- **Causal** architecture — enables streaming

**Key insight**: 30 seconds of audio → **640 latent frames**, each with 10 codebook indices.

### 4. PCA Latent Space

The S1-DAC codec produces a 1024-dimensional continuous pre-decoder latent (sum of 10 up-projected codebook embeddings, each 8→1024). Since the actual rank is ≤ 80 (10 codebooks × 8 dim), Echo applies **PCA** to reduce this to an **80-dimensional** latent:

```
z_q (1024-dim) → subtract mean → multiply by PCA components (1024→80) → scale by scalar
```

This 80-dim PCA latent is what the diffusion model actually operates on. The inverse mapping reconstructs the 1024-dim vector for the S1-DAC decoder.

### 5. Speaker Encoder

| Property | Value |
|---|---|
| Type | Causal Transformer |
| Layers | 14 |
| Model dim | 1280 |
| Heads | 10 |
| FFN dim | 3328 |
| Patch size | 4 (groups of 4 latent frames → 1 token) |
| Max input | ~5 min of reference audio (6400 latents, 1600 patched tokens) |

Speaker audio goes through the same S1-DAC encode + PCA pipeline. Then every 4 consecutive latent frames are concatenated (80×4 = 320 dim) and linearly projected to 1280 dim. This is processed by a causal Transformer, and the outputs become per-layer KV caches for the decoder.

### 6. Diffusion Decoder (EchoDiT) — The Core

| Property | Value |
|---|---|
| Type | DiT (Diffusion Transformer) |
| Layers | 24 |
| Model dim | 2048 |
| Heads | 16 |
| FFN dim | 5888 |
| Input sequence | 640 tokens × 80 dim |
| Conditioning | AdaLN (timestep) + Joint cross-attention (text + speaker) |

**Timestep conditioning:**
- Sinusoidal timestep embedding (dim 512) → 3-layer MLP → 3 × 2048 dim (shift, scale, gate)
- Applied via **Low-Rank AdaLN**: each of shift/scale/gate has a down-project (2048→256) then up-project (256→2048) with residual connection
- Gate uses **tanh** (not sigmoid) — this is unusual

**Attention — Joint Self-Cross-Attention:**

Each decoder layer has a single attention operation where the query attends to the **concatenation** of:
1. **Self KV** (from the noised latent sequence itself)
2. **Text KV** (pre-computed from text encoder, per-layer projections)
3. **Speaker KV** (pre-computed from speaker encoder, per-layer projections)
4. **Latent-prefix KV** (for blockwise generation, optional)

Each modality has **separate** `wk` and `wv` projections (so 4 pairs of K/V projection matrices per layer). QK-norm is shared. RoPE is applied to **half the heads only** (first half gets RoPE, second half does not — enables positional and non-positional channels).

**Gated attention**: attention output is element-wise multiplied by `sigmoid(gate(x))` before the output projection.

### 7. Diffusion Process — Rectified Flow

- **Forward**: `x_t = (1-t) * x_0 + t * noise` (linear interpolation)
- **Training objective**: predict velocity `v = noise - x_0`
- **Timestep distribution**: stratified logit-normal
- **Sampling**: Euler method on the ODE, typically 30–40 steps
- Schedule: linear from t=0.999 → 0

### 8. Classifier-Free Guidance (CFG) — Independent Dual Guidance

The model uses **independent guidance** for text and speaker, requiring 3 forward passes per step:

```
v_final = v_cond + w_text × (v_cond - v_uncond_text) + w_speaker × (v_cond - v_uncond_speaker)
```

- `v_cond`: both text and speaker conditioned
- `v_uncond_text`: speaker conditioned, text mask zeroed
- `v_uncond_speaker`: text conditioned, speaker mask zeroed

CFG is only applied within a time window (`cfg_min_t=0.5` to `cfg_max_t=1.0`), saving compute in later denoising steps.

**Optional enhancements:**
- **Temporal score rescaling** (from [arxiv 2510.01184](https://arxiv.org/pdf/2510.01184))
- **Speaker KV scaling**: multiply speaker KV cache values by a scale factor (e.g., 1.5) to force speaker identity when text is out-of-distribution for that voice
- **Truncation factor**: scale initial noise by 0.8–0.9 to reduce artifacts

### 9. Blockwise Generation (for streaming/continuation)

A separate **Latent-Prefix Encoder** (same architecture as Speaker Encoder) processes already-generated clean latents. The decoder can then generate the next block conditioned on this prefix, enabling:

- Streaming with low time-to-first-byte
- Audio continuation from existing clips
- Generating in smaller blocks (e.g., 128, 128, 64 latents)

The prefix encoder uses RoPE with patch-dilated positions for proper temporal alignment.

---

## Training Setup

| Aspect | Detail |
|---|---|
| Hardware | TPU v4-64 pod |
| Framework | JAX/Flax (Linen) — inference code is PyTorch |
| Optimizer | **Muon** (outperformed Adam in ablations) |
| Batch size | 768 |
| Steps | 800K |
| LR schedule | WSD (Warmup-Stable-Decay) |
| Precision | BF16 compute, FP32 weights+optimizer |
| Dataset | ~160K hours podcast-like audio |
| Transcription | WhisperD (custom Whisper with diarization + disfluencies) |
| Condition dropout | 10% text, 10% speaker (independent) |

---

## Dependency Tree

```
Echo-TTS
├── Fish Speech S1-DAC (CC-BY-NC-SA-4.0)
│   ├── Descript Audio Codec (DAC) architecture (MIT)
│   ├── SoundStream-style Residual VQ
│   ├── ConvNeXt blocks
│   └── Window-limited causal Transformer
├── WhisperD (training data transcription)
├── PCA state (pre-computed, stored as safetensors)
├── PyTorch ≥ 2.9.1
├── torchaudio, torchcodec
├── einops
└── HuggingFace Hub (model hosting)
```

---

## Key Design Decisions & What Makes It Work

1. **PCA over concatenation** for the autoencoder latent — preserves geometry, gives varying importance per channel, enables potential future inference tricks
2. **Byte-level text encoding** — no tokenizer dependency, multilingual by default (any UTF-8 text)
3. **Joint attention** instead of separate cross-attention layers — more parameter-efficient, single attention op per layer
4. **Independent CFG** — decouples text adherence from speaker adherence, critical for voice cloning fidelity
5. **Causal S1-DAC decoder** — enables streaming when combined with blockwise generation
6. **Low-rank AdaLN** — reduces parameter count for timestep conditioning while maintaining expressivity
7. **Half-RoPE** — only half the attention heads get positional encoding, letting the other half attend position-independently

---

## For Training in a New Language

Since the tokenizer is **raw UTF-8 bytes** (no language-specific vocabulary), the model architecture is already language-agnostic. To train for a new language, you would need:

1. **Audio data** in the target language (~100K+ hours ideally)
2. **WhisperD-style transcriptions** with speaker diarization tags `[S1]`, `[S2]`
3. The **Fish S1-DAC autoencoder** (unchanged — it's language-independent)
4. **PCA state** (recompute from your data, or reuse the existing one since S1-DAC is language-agnostic)
5. Fine-tune or retrain the **Text Encoder + Diffusion Decoder** on the new data

The Text Encoder would learn the byte patterns of the new language's writing system. The speaker encoder and S1-DAC are already language-independent.


<user_query>
I want you to check the QAN 0.5b real-time TTS model first, because that's an auto-recording model.
</user_query>


Let me look up the Qwen 0.5B real-time TTS model.


Now I have a comprehensive understanding of both models. Here's the complete analysis:

---

# Qwen3-TTS: Full Architecture & Pipeline Breakdown

## The Model You're Asking About

You mentioned "QAN 0.5B" — this is **Qwen3-TTS-12Hz-0.6B-Base**, released by Alibaba's Qwen team in January 2026. It's an **autoregressive** TTS model (not diffusion like Echo-TTS). License: **Apache 2.0**.

- Paper: [arXiv:2601.15621](https://arxiv.org/pdf/2601.15621v1)
- GitHub: [QwenLM/Qwen3-TTS](https://github.com/QwenLM/Qwen3-TTS)
- Model: [Qwen/Qwen3-TTS-12Hz-0.6B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-Base)

---

## High-Level: How It Differs from Echo-TTS

| Aspect | Echo-TTS (2.4B) | Qwen3-TTS-12Hz (0.6B) |
|---|---|---|
| **Paradigm** | Diffusion (Rectified Flow) | **Autoregressive LM** |
| **Backbone** | Custom DiT | **Qwen3 LLM family** |
| **Token rate** | ~21.5 Hz (via Fish S1-DAC) | **12.5 Hz** (custom tokenizer) |
| **Codebooks** | 10 (external codec, PCA'd to 80-dim) | **16** (semantic + 15 RVQ acoustic) |
| **Streaming** | Blockwise (fine-tuned add-on) | **Native dual-track streaming** |
| **First-packet latency** | Not designed for it | **97 ms** |
| **Languages** | English-focused | **10 languages** (zh, en, ja, ko, de, fr, ru, pt, es, it) |
| **Training data** | ~160K hours | **5 million+ hours** |
| **Voice cloning** | Via speaker encoder (up to 5 min ref) | **3-second cloning** (speaker embedding or in-context) |
| **Voice design** | Not supported | **Natural language instructions** |
| **License** | CC-BY-NC-SA-4.0 (due to Fish codec) | **Apache 2.0** |

---

## The Full Qwen3-TTS-12Hz Pipeline

```
┌─────────────────────────────────────────────────────────────────────┐
│                        INFERENCE PIPELINE                          │
│                                                                     │
│  Text ──→ [Qwen Tokenizer] ──┐                                    │
│                                ├──→ [Dual-Track LM Backbone] ──┐   │
│  Speaker Ref ──→ [Speaker     │    (Qwen3 architecture)        │   │
│   Encoder] ──→ embedding ──┘                                   │   │
│                                                                  │   │
│              Codebook 0 (semantic) ←── LM head prediction       │   │
│              Codebooks 1-15 (acoustic) ←── MTP module           │   │
│                                                                  ↓   │
│              [Qwen-TTS-Tokenizer-12Hz Decoder] ──→ 44.1kHz wav │   │
│              (Causal ConvNet — NO diffusion needed)              │   │
└─────────────────────────────────────────────────────────────────────┘
```

---

## Component-by-Component

### 1. Speech Tokenizer: Qwen-TTS-Tokenizer-12Hz

This is the most critical piece — a **12.5 Hz multi-codebook tokenizer** with jointly optimized semantic and acoustic streams.

**Architecture:**
- Based on the **Mimi** architecture (from Meta/Kyutai, the Moshi paper)
- **Encoder**: Fully causal feature encoder, processes audio at 12.5 Hz (80ms per frame)
- **Quantizer**: 16-layer codebook structure
  - **Layer 0**: Semantic codebook (2048 entries) — trained with **WavLM** as a distillation teacher for semantic alignment
  - **Layers 1–15**: Acoustic RVQ (15 layers, 2048 entries each) — progressively captures acoustic detail, prosody, timbre
- **Decoder**: Fully causal ConvNet — reconstructs waveform incrementally, **no diffusion model needed**
- **Training**: GAN-based (generator + discriminator) with multi-scale mel-spectrogram reconstruction loss
- **Output**: 44.1kHz audio

**Why this matters:**
- At 12.5 Hz with 16 codebooks, 30 seconds of audio = **375 frames × 16 codes** = 6,000 total tokens
- The causal decoder means the instant a codec frame is available, audio can be emitted — **no lookahead needed**
- This is what enables the 97ms first-packet latency

**Benchmark results** (on LibriSpeech test-clean):

| Model | FPS | NQ | PESQ_WB | UTMOS | Speaker SIM |
|---|---|---|---|---|---|
| SpeechTokenizer | 50 | 8 | 2.60 | 3.90 | 0.85 |
| Mimi | 12.5 | 16 | 2.88 | 3.87 | 0.87 |
| **Qwen-TTS-Tokenizer-12Hz** | **12.5** | **16** | **3.21** | **4.16** | **0.95** |

State-of-the-art across all metrics, with the highest speaker similarity (0.95) by a wide margin.

### 2. Text Processing: Qwen Tokenizer

Unlike Echo-TTS's raw UTF-8 byte tokenizer, Qwen3-TTS uses the **standard Qwen tokenizer** — the same BPE tokenizer used by Qwen3 LLMs. This means:

- Full vocabulary of ~150K+ tokens
- Excellent multilingual coverage (trained on massive multilingual text corpora)
- Seamless integration with the Qwen3 LLM backbone
- Input is formatted in **ChatML** format for instruction following

### 3. LM Backbone: Dual-Track Autoregressive Architecture

The core model is built on the **Qwen3 LM family** (same architecture as Qwen3 language models). The key innovation is the **dual-track** design:

**Track 1 — Text track:**
- Text tokens flow in as a standard autoregressive sequence

**Track 2 — Audio track:**
- Speech codec tokens are **concatenated along the channel axis** with text tokens
- Upon receiving a text token, the model **immediately** predicts the corresponding audio tokens

**Hierarchical prediction:**
1. The **LM backbone** predicts **Codebook 0** (the semantic codebook) via a linear head
2. An **MTP (Multi-Token Prediction) module** then generates **Codebooks 1–15** (all acoustic residual layers) in one shot from the backbone's hidden states

This is fundamentally different from Echo-TTS: instead of denoising all audio latents in parallel over 40 diffusion steps, Qwen3-TTS generates audio **one codec frame at a time**, autoregressively, but predicts all 16 codebook layers for each frame simultaneously via MTP.

### 4. Speaker Conditioning

Two modes:

**a) Speaker Embedding (real-time cloning):**
- A **learnable speaker encoder** is jointly trained with the backbone
- Takes ~3 seconds of reference audio → speaker embedding vector
- Injected into the LM's conditioning

**b) In-Context Learning (higher-fidelity cloning):**
- Reference text + reference audio tokens are prepended to the input sequence
- The LM learns to continue in the same voice via autoregressive context
- Better preserves prosody and style nuances

### 5. Streaming Architecture

The dual-track design enables **true streaming** with both streaming text input and streaming audio output:

- **LM TTFP (Time-to-First-Packet)**: 93ms for 0.6B model
- **Tokenizer decode time per packet**: 4ms (causal ConvNet is extremely lightweight)
- **Total first-packet latency**: **97ms**
- **Steady-state**: each packet = 4 tokens = 320ms of audio, generated in ~19ms (RTF = 0.288)

Compare this to the 25Hz variant which needs a DiT + BigVGAN pipeline and has 138ms first-packet latency.

### 6. Training

**Three-stage pre-training:**

| Stage | Description | Data |
|---|---|---|
| **S1: General** | Monotonic text→speech mapping, general capabilities | 5M+ hours, 10 languages |
| **S2: High-Quality** | CPT on quality-stratified data, reduces hallucinations | High-quality subset |
| **S3: Long-Context** | Extends max tokens from 8,192 → 32,768, upsamples long speech | Long-form data |

**Three-stage post-training:**

| Stage | Method | Purpose |
|---|---|---|
| **DPO** | Direct Preference Optimization with human feedback | Align with human preferences |
| **GSPO** | Rule-based rewards | Enhance stability across tasks |
| **Speaker SFT** | Lightweight speaker fine-tuning | Adopt specific voices, improve naturalness |

### 7. Model Variants

| Model | Size | Streaming | Voice Clone | Instruction Following |
|---|---|---|---|---|
| 12Hz-0.6B-Base | 0.6B | Yes | Yes | — |
| 12Hz-0.6B-CustomVoice | 0.6B | Yes | Yes | — |
| 12Hz-1.7B-Base | 1.7B | Yes | Yes | — |
| 12Hz-1.7B-VoiceDesign | 1.7B | Yes | Yes | Yes |
| 12Hz-1.7B-CustomVoice | 1.7B | Yes | Yes | — |

---

## Key Benchmark Results (0.6B vs 1.7B)

**Zero-shot voice cloning** (Seed-TTS test set, WER↓):

| Model | Chinese | English |
|---|---|---|
| CosyVoice 3 | 0.71 | 1.45 |
| **Qwen3-TTS-12Hz-0.6B** | **0.92** | **1.32** |
| **Qwen3-TTS-12Hz-1.7B** | **0.77** | **1.24** |

The 0.6B model already outperforms most baselines including F5-TTS, Seed-TTS, and MiniMax-Speech on English.

---

## Echo-TTS vs Qwen3-TTS: Which Approach for What?

| Use Case | Better Choice | Why |
|---|---|---|
| **Real-time streaming / chatbot** | **Qwen3-TTS** | 97ms latency, native streaming, autoregressive |
| **Maximum voice cloning fidelity** | **Qwen3-TTS** | 0.95 speaker SIM, 3-sec cloning, 5M hours training |
| **Multilingual (10 languages)** | **Qwen3-TTS** | Trained on 10 languages with 5M+ hours |
| **Single high-quality batch generation** | **Echo-TTS** | 1.45s for 30s audio on A100, very fast RTF |
| **Creative audio (singing, characters)** | **Echo-TTS** | CFG gives fine-grained control over style |
| **Commercial use** | **Qwen3-TTS** | Apache 2.0 vs Echo's CC-BY-NC-SA-4.0 |
| **Training on new language from scratch** | **Qwen3-TTS** | Proven multilingual architecture, Apache 2.0, Qwen LLM backbone |
| **Smaller GPU (8GB VRAM)** | Either | Echo works at 8GB; Qwen3 0.6B in bf16 ~1.2GB weights |

**Bottom line**: Qwen3-TTS-12Hz-0.6B is the better starting point if you want an autoregressive, real-time, streaming, multilingual TTS model that you can fine-tune for new languages — especially with its Apache 2.0 license and the full Qwen ecosystem behind it.


<user_query>
Forget about echo; we are focusing on qwen 3 0.6 for now. I want to have you see that I have Hindi data. I want to train that on the 0.6 so I can see what is coming and what is not.
</user_query>


Let me understand what you have and plan the approach. First, let me check what data and resources you have available.


Good — you have an A100 80GB, `qwen-tts` and `faster-qwen3-tts` already installed, and two Hindi audio samples. Let me first run a baseline test to see what Qwen3-TTS-12Hz-0.6B produces for Hindi text **out of the box** — since Hindi is not one of the 10 supported languages, this will show us exactly what's missing.


It seems to be stuck loading or on the first generation. Let me check if the process is still alive and see what's happening.


The process died silently. Let me check the full terminal output and see what happened, then run a simpler test.


The process seems to have been killed (likely OOM or the generate call hung). The `generate_voice_clone` API might not work without reference audio, or the API has changed. Let me check the actual `qwen_tts` module and write a more minimal test.


Good, now I can see the full API. Let me also install SoX (it's complaining) and then run a minimal, targeted test.


OK — the Base model **requires** a reference audio for voice cloning (no "no-reference" generation). The two with-reference tests succeeded. Now let me check `generate_custom_voice` (which uses preset speakers and doesn't need a ref audio), run more comprehensive tests, and also check the actual output audio quality.


`generate_custom_voice` requires a "CustomVoice" model variant. The Base model only supports `generate_voice_clone`. Let me now run a proper comprehensive test with the reference audio, testing different scenarios to see how Hindi behaves.


Something odd is happening. Let me check more carefully.


There seems to be a shell issue. Let me try a fresh shell.


The shell seems broken. Let me try with a completely new working directory specification.


Something is fundamentally wrong with shell output capture. Let me try redirecting to a file and reading that.


The shell seems completely dead on output. Let me try a new approach - write a small inline script and execute it.


There's clearly a shell capture issue in this session. Let me check the terminals folder for any output.


The shell commands are not actually executing — they return instantly with exit code 0 and empty output. This is a session-level issue. Let me use a Task subagent which gets its own shell environment.