type: user

role: user

content:
I need to use a model called "SoloSpeech" for target speaker extraction. The user has a reference clip of one speaker and wants to extract that speaker's voice from a mixed audio.

Please research and report concisely (under 300 words):

1. Find the official SoloSpeech GitHub repository (likely the paper "SoloSpeech" 2024/2025 — target speaker extraction via cascaded generative model). Confirm the repo URL.
2. What's the exact installation procedure (pip deps, conda, model checkpoint downloads)?
3. What's the inference command/Python API? Specifically: how to pass a *reference clip of the target speaker* + a *mixed audio* and get back the *extracted target speaker audio*?
4. What input format does it expect (sample rate, mono/stereo, length limits)?
5. Are model weights on HuggingFace? Direct link if so.
6. Any GPU memory / dependency conflicts to watch out for?

Don't write code or install anything — just research and report. Use WebFetch on the repo README and any HF model card. Also try google/web search for "SoloSpeech target speaker extraction github".

userType: external

entrypoint: cli

gitBranch: HEAD

slug: enchanted-strolling-quasar


type: message

role: assistant

type: text

text: I'll research SoloSpeech for target speaker extraction.

service_tier: standard

type: assistant


ToolSearch

query: WebFetch WebSearch

type: direct

speed: standard


tool_name: WebFetch


text: I'll load WebFetch and WebSearch, then research SoloSpeech.


query: select:WebFetch,WebSearch


tool_name: WebSearch


WebSearch

query: SoloSpeech target speaker extraction github cascaded generative


content:
Web search results for query: "SoloSpeech target speaker extraction github cascaded generative"

Links: [{"title":"GitHub - WangHelin1997/SoloSpeech: SoloSpeech: Enhancing Intelligibility and Quality in Target Speech Extraction through a Cascaded Generative Pipeline · GitHub","url":"https://github.com/WangHelin1997/SoloSpeech"},{"title":"SoloSpeech/docs/quick_use.md at main · WangHelin1997/SoloSpeech","url":"https://github.com/WangHelin1997/SoloSpeech/blob/main/docs/quick_use.md"},{"title":"SoloSpeech","url":"https://solospeech.github.io/SoloSpeech-Demo/"},{"title":"Paper page - SoloSpeech: Enhancing Intelligibility and Quality in Target Speech Extraction through a Cascaded Generative Pipeline","url":"https://huggingface.co/papers/2505.19314"},{"title":"SoloSpeech - GitHub Pages","url":"https://wanghelin1997.github.io/SoloSpeech-Demo/"},{"title":"[2505.19314] SoloSpeech: Enhancing Intelligibility and Quality in Target Speech Extraction through a Cascaded Generative Pipeline","url":"https://arxiv.org/abs/2505.19314"},{"title":"SoloSpeech: Enhancing Intelligibility and Quality in Target Speech Extraction through a Cascaded Generative Pipeline","url":"https://arxiv.org/html/2505.19314"},{"title":"(PDF) SoloSpeech: Enhancing Intelligibility and Quality in Target Speech Extraction through a Cascaded Generative Pipeline","url":"https://www.researchgate.net/publication/392105023_SoloSpeech_Enhancing_Intelligibility_and_Quality_in_Target_Speech_Extraction_through_a_Cascaded_Generative_Pipeline"},{"title":"haidog-yaqub (Jiarui Hai) · GitHub","url":"https://github.com/haidog-yaqub"},{"title":"README.md · OpenSound/SoloSpeech at main","url":"https://huggingface.co/spaces/OpenSound/SoloSpeech/blob/main/README.md"}]

Here are the search results for SoloSpeech:

## Overview

SoloSpeech is a novel cascaded generative pipeline that integrates compression, extraction, reconstruction, and correction processes. The project is available on GitHub at: https://github.com/WangHelin1997/SoloSpeech

## What It Does

Target Speech Extraction (TSE) aims to isolate a target speaker's voice from a mixture of multiple speakers by leveraging speaker-specific cues. SoloSpeech presents a novel cascaded generative pipeline that integrates compression, extraction, reconstruction, and correction processes.

## Key Features

SoloSpeech features a speaker-embedding-free target extractor that utilizes conditional information from the cue audio's latent space, aligning it with the mixture audio's latent space to prevent mismatches.

## Performance

Evaluated on the widely-used Libri2Mix dataset, SoloSpeech achieves the new state-of-the-art intelligibility and quality in target speech extraction while demonstrating exceptional generalization on out-of-domain data and real-world scenarios.

## Quick Start

An example to run SoloSpeech involves using the --test-wav flag for the mixture audio, --enroll-wav for the enrollment audio representing the speaker you want to extract, and --output-path for saving the output audio.

REMINDER: You MUST include the sources above in your response to the user using markdown hyperlinks.
[result-id: r1]


WebFetch

prompt: Extract installation steps, dependencies, checkpoint download instructions, inference command/API, input format (sample rate, mono/stereo, length), HuggingFace model links, GPU requirements.


prompt: Extract exact quick-use instructions: install, checkpoint download, inference command line with enroll-wav and test-wav, input audio format requirements.


content:
# SoloSpeech Installation & Usage Guide

## Installation Steps
The README references documentation files for setup. You'll need to check `/docs/quick_use.md` for detailed installation instructions, which aren't fully displayed in the provided content.

## Dependencies
The repository includes a `requirements.txt` file. The project is "Python 99.9%" based on the language breakdown shown.

## Model Downloads
Models are available on HuggingFace: **"🤗 [Models](https://huggingface.co/OpenSound/SoloSpeech-models/)"** - hosted at `huggingface.co/OpenSound/SoloSpeech-models/`

## Inference Access
Multiple demo options are provided:
- **Colab**: "💻 [Colab Demo](https://colab.research.google.com/drive/1cEcyp2rFP2DOLY4BLjaKksF48-xXQAdQ?usp=sharing)"
- **HuggingFace Space**: "🚀 [Space Demo](https://huggingface.co/spaces/OpenSound/SoloSpeech/)"

## Input Specifications
Not explicitly detailed in the provided content. You'll need to review the actual documentation files in the `/docs/` directory.

## GPU Requirements
Not specified in the available README content.

## Key References
- Paper: arxiv.org/abs/2505.19314
- Audio samples: wanghelin1997.github.io/SoloSpeech-Demo/
- License: "CC BY-NC 4.0" (non-commercial use only)

**Note**: The README directs users to separate documentation files for complete setup details.
[result-id: r2]


content:
# SoloSpeech Quick-Use Instructions

## Installation

The setup process for Linux developers involves creating a conda environment and installing dependencies:

```bash
conda create -n solospeech python=3.8.19
conda activate solospeech
git clone https://github.com/WangHelin1997/SoloSpeech
cd SoloSpeech/
pip install -r requirements.txt
pip install .
```

## Inference Command

The basic execution syntax is:

```bash
python scripts/test_v2.py --test-wav "./assets/test2.wav" --enroll-wav "./assets/test2_enroll.wav" --output-path "./demo/test2_solospeech.wav"
```

## Parameter Definitions

- **`--test-wav`**: Path to the mixture audio (input containing multiple speakers)
- **`--enroll-wav`**: Path to enrollment audio representing the target speaker for extraction
- **`--output-path`**: Where to save the extracted audio result

## Notable Gaps

The documentation does not specify:
- Required audio format (sample rate, bit depth, channels)
- Checkpoint download instructions or model weights location
- Minimum enrollment audio duration requirements
[result-id: r3]


prompt: List all model checkpoint files, their sizes, sample rate, download instructions, and any usage notes.


prompt: Extract expected sample rate, mono/stereo, length limits, how checkpoints are loaded (paths), GPU memory usage. Summarize the inference pipeline arguments and defaults.


content:
# SoloSpeech-models - Model Information

Based on the webpage content provided, I can see this is a Hugging Face model repository, but **the specific model checkpoint files, their sizes, and sample rates are not included in the HTML content shared**.

## What I Can Confirm:

**Model Details:**
- **Repository**: OpenSound/SoloSpeech-models
- **License**: CC-BY-NC-SA-4.0
- **Paper**: SoloSpeech: Enhancing Intelligibility and Quality in Target Speech Extraction through a Cascaded Generative Pipeline
- **ArXiv ID**: 2505.19314
- **Published**: May 25, 2025

**Citation:**
```bibtex
@misc{wang2025solospeechenhancingintelligibilityquality,
  title={SoloSpeech: Enhancing Intelligibility and Quality in Target Speech Extraction through a Cascaded Generative Pipeline},
  author={Helin Wang and Jiarui Hai and Dongchao Yang and Chen Chen and Kai Li and Junyi Peng and Thomas Thebaud and Laureano Moro Velazquez and Jesus Villalba and Najim Dehak},
  year={2025},
  eprint={2505.19314},
  archivePrefix={arXiv},
  primaryClass={eess.AS},
  url={https://arxiv.org/abs/2505.19314}
}
```

## To Access Full Model Details:

Visit the model card directly at:
```
https://huggingface.co/OpenSound/SoloSpeech-models
```

The full model files and specifications would be available in the **Files and versions** section of the repository.
[result-id: r4]


content:
# SoloSpeech Inference Pipeline Summary

## Audio Specifications
- **Sample Rate**: 16,000 Hz (fixed)
- **Channel Configuration**: Mono (single waveform processing)
- **Length Limits**: No explicit constraints; code handles variable-length audio with dynamic padding

## Checkpoint Loading
Checkpoints are downloaded automatically from Hugging Face Hub (`OpenSound/SoloSpeech-models`):
- Extractor model: `extractor.pt`
- Compressor (VAE): `compressor.ckpt`
- Corrector (GECO): `corrector.ckpt`
- Config files: `config_extractor.yaml`, `config_compressor.json`

## GPU Memory Considerations
The pipeline uses CUDA when available (`cuda:0`), with fallback to CPU. No explicit memory limits are specified; inference processes batches of up to `num_candidates` samples simultaneously.

## Inference Pipeline Arguments & Defaults

| Argument | Type | Default | Purpose |
|----------|------|---------|---------|
| `output-path` | str | Required | Output audio file destination |
| `test-wav` | str | Required | Mixture audio input |
| `enroll-wav` | str | Required | Reference speaker enrollment |
| `num_infer_steps` | int | 200 | Diffusion sampling iterations |
| `num_candidates` | int | 4 | Parallel extraction candidates for reranking |
| `eta` | int | 0 | DDIM scheduler stochasticity parameter |
| `random-seed` | int | 42 | Reproducibility seed |
| `sample-rate` | int | 16000 | Audio resampling rate |

The pipeline performs speaker extraction via diffusion, speaker identification reranking, and spectral correction sequentially.
[result-id: r5]


text:
I have enough information. Here's the report.

---

## SoloSpeech Research Report

**1. Repository**
Official repo: https://github.com/WangHelin1997/SoloSpeech
Paper: arXiv 2505.19314 (May 2025) — "SoloSpeech: Enhancing Intelligibility and Quality in Target Speech Extraction through a Cascaded Generative Pipeline" by Wang et al. (JHU/CLSP).
License: CC BY-NC 4.0 (non-commercial).

**2. Installation** (from `docs/quick_use.md`)
```
conda create -n solospeech python=3.8.19
conda activate solospeech
git clone https://github.com/WangHelin1997/SoloSpeech
cd SoloSpeech/
pip install -r requirements.txt
pip install .
```
Checkpoints are auto-downloaded from HF Hub (`OpenSound/SoloSpeech-models`) on first run — no manual download required. Files pulled: `extractor.pt`, `compressor.ckpt`, `corrector.ckpt`, plus `config_extractor.yaml` and `config_compressor.json`.

**3. Inference command**
```
python scripts/test_v2.py \
  --test-wav   path/to/mixture.wav \
  --enroll-wav path/to/reference_target_speaker.wav \
  --output-path path/to/extracted.wav
```
`--enroll-wav` is the reference clip of the target speaker; `--test-wav` is the mixed audio; output is the isolated target. Additional args: `--num_infer_steps` (default 200 diffusion steps), `--num_candidates` (default 4, parallel candidates then reranked by a speaker-ID model), `--eta` (0), `--random-seed` (42), `--sample-rate` (16000).

**4. Input format**
- Sample rate: 16 kHz (auto-resampled)
- Channels: mono
- No hard length limit; dynamic padding. Very long clips will increase VRAM since `num_candidates=4` are processed together.

**5. HuggingFace weights**
- Models: https://huggingface.co/OpenSound/SoloSpeech-models (CC-BY-NC-SA-4.0)
- Live demo Space: https://huggingface.co/spaces/OpenSound/SoloSpeech
- Colab: linked from the README.

**6. Gotchas**
- Python pinned to 3.8.19 — will likely conflict with newer torch/torchaudio envs; use an isolated conda env.
- Uses CUDA if available, else CPU (slow — 200 diffusion steps x 4 candidates).
- Pipeline is cascaded (compressor VAE -> diffusion extractor -> corrector/GECO -> SID reranker), so VRAM scales with `num_candidates`; reduce it if you OOM. No published VRAM number, but expect several GB for typical clips on a diffusion model of this size.
- `pip install .` installs the package itself — needed, not just requirements.txt.
- Trained/evaluated on Libri2Mix (English, clean 16 kHz); out-of-domain (music, heavy noise, non-English) may degrade though authors report good generalization.

Sources:
- [SoloSpeech GitHub](https://github.com/WangHelin1997/SoloSpeech)
- [quick_use.md](https://github.com/WangHelin1997/SoloSpeech/blob/main/docs/quick_use.md)
- [HF model weights: OpenSound/SoloSpeech-models](https://huggingface.co/OpenSound/SoloSpeech-models)
- [HF Space demo](https://huggingface.co/spaces/OpenSound/SoloSpeech)
- [arXiv 2505.19314](https://arxiv.org/abs/2505.19314)
- [Demo page](https://wanghelin1997.github.io/SoloSpeech-Demo/)