let's discuss about this dataset. https://huggingface.co/datasets/ai4bharat/IndicVoices, i already have sourced the 11 languages that i need from this list. hi,te,ml,ta,kn,ma,pa,as,or,be,gj(gujarat), and everything that is transcribed is with me. now to explore how much quantity is transcribed and how much is not transcribed, can we get an estimate based on repo's readme, parquets, language based, etc, any estimations ?


Let me fetch the dataset page and README to get estimates on transcribed vs non-transcribed data.


ToolSearch

Question: select:WebFetch


Question: select:WebFetch


WebFetch

Prompt:
Extract ALL information about the dataset, especially:
1. Total hours per language
2. How much is transcribed vs not transcribed per language
3. Any breakdown of "read", "extempore", "conversational" splits
4. The language codes and full names mapping
5. Any statistics tables showing hours, utterances, speakers per language
6. Information about what subset is transcribed and what is not
7. Any mention of IndicVoices-R (the transcribed/read subset)
Extract every table and statistic you can find. Be very thorough.


Prompt: List all files and folders in this dataset repository. I want to see the structure - especially any parquet files, language-specific folders, README, etc. List everything you can see.


# IndicVoices Dataset Structure

Based on the webpage content, here's the complete file and folder structure:

## Language Folders (22 total)
Each language has its own directory containing parquet files:

1. **assamese/**
2. **bengali/**
3. **bodo/**
4. **dogri/**
5. **gujarati/**
6. **hindi/**
7. **kannada/**
8. **kashmiri/**
9. **konkani/**
10. **maithili/**
11. **malayalam/**
12. **manipuri/**
13. **marathi/**
14. **nepali/**
15. **odia/**
16. **punjabi/**
17. **sanskrit/**
18. **santali/**
19. **sindhi/**
20. **tamil/**
21. **telugu/**
22. **urdu/**

## Root Files

- **README.md** (38.8 kB) - Documentation
- **.gitattributes** (2.46 kB) - Git configuration

## Dataset Specifications

| Property | Value |
|----------|-------|
| **Modalities** | Audio, Text |
| **Format** | Parquet |
| **Size Category** | 1M - 10M |
| **License** | CC-BY-4.0 |
| **ArXiv** | 2403.01926 |
| **Libraries** | Datasets, Dask, Polars (+1) |

**Note:** This is a gated dataset requiring login/sign-up to access the actual file contents, though the file structure is visible.


# IndicVoices Dataset - Information Extraction

Based on the webpage content provided, here's all available information about the dataset:

## Overall Dataset Statistics

- **Total Duration**: 23.7K hours of audio
- **Total Speakers**: 51K speakers
- **Geographic Coverage**: 400+ Indian districts
- **Languages Supported**: 22 languages (listed in 8th schedule of Indian Constitution)
- **Transcribed Data**: 11.2K hours (as of December 23, 2025)
- **Untranscribed Data**: ~12.5K hours

## Audio Type Breakdown

The 23.7K hours consists of:
- **Read**: 8%
- **Extempore**: 76%
- **Conversational**: 15%

## Language-Specific Statistics

**⚠ Important Note**: The webpage does NOT contain detailed per-language breakdowns showing:
- Total hours per language
- Transcribed vs. untranscribed hours per language
- Speakers per language
- Utterances per language
- Language codes and full names mapping

These statistics tables that would typically be in a full dataset card are **not present** in the extracted content.

## Available Formats & Size

- **Format**: Parquet
- **Size Category**: 1M - 10M
- **License**: CC BY 4.0
- **ArXiv Paper**: 2403.01926

## Loading the Dataset

```python
from datasets import load_dataset

# Load dataset for a specific language
dataset = load_dataset("ai4bharat/IndicVoices", "assamese", split="valid")

# Stream the dataset
dataset = load_dataset("ai4bharat/IndicVoices", "assamese", split="valid", streaming=True)
```

## Full-Length Audio Access

Full-length audio files are distributed separately in 5 versions (v1-v5) per language:
```
https://iv-release.objectstore.e2enetworks.net/dmu_release/v1_<LANGUAGE>_train.tgz
https://iv-release.objectstore.e2enetworks.net/dmu_release/v2_<LANGUAGE>_train.tgz
...
https://iv-release.objectstore.e2enetworks.net/dmu_release/v5_<LANGUAGE>_train.tgz
```

**Note**: You must download all versions to get complete data for a language.

## What's Missing from This Extraction

To find detailed per-language statistics, you would need to:
1. Visit the full [Dataset Card on Hugging Face](https://huggingface.co/datasets/ai4bharat/IndicVoices)
2. Check the [ArXiv paper (2403.01926)](https://arxiv.org/abs/2403.01926)
3. Review the [GitHub repository](https://github.com/AI4Bharat/IndicVoices)


Good, I have the high-level stats. Let me now fetch the paper and the raw README for detailed per-language breakdowns.


Prompt:
Extract ALL statistics tables from this paper about IndicVoices. I need:
1. Per-language total hours (read, extempore, conversational breakdown)
2. Per-language transcribed vs untranscribed hours
3. Number of speakers per language
4. Any table showing which languages have transcriptions and which don't
5. Any mention of how much data is transcribed for each of the 22 languages
Extract every number and table you can find.


Prompt: Extract the COMPLETE content of this README file. I need every table, every statistic, every per-language breakdown. Do not summarize - extract all data tables with exact numbers for hours, speakers, transcribed status per language.


Request failed with status code 401

Error: Request failed with status code 401


# IndicVoices Dataset Statistics

Based on the provided web page content, I can only extract limited statistics, as the abstract does not contain detailed tables:

## Available Statistics from Abstract:

**Overall Dataset:**
- Total audio: "7348 hours of read (9%), extempore (74%) and conversational (17%) audio"
- Total speakers: 16,237
- Geographic coverage: 145 Indian districts
- Languages covered: 22
- Transcribed hours: "1639 hours have already been transcribed, with a median of 73 hours per language"

## Key Limitation:

The webpage content provided is only the arXiv abstract page. It does not include the actual paper tables with the detailed breakdown you requested, such as:
- Per-language hour distributions
- Per-language speaker counts
- Which specific languages have transcriptions completed
- Transcribed vs. untranscribed hours by language

To obtain these statistics, you would need to access the full PDF or HTML versions of the paper (available via the links provided on the page), which should contain comprehensive statistical tables in the results section.


The README is gated. Let me try the paper's HTML version and the GitHub repo for detailed per-language stats.


Prompt:
Extract ALL tables from this paper. I specifically need:
1. Table showing per-language hours breakdown (read, extempore, conversational)
2. Table showing transcribed hours per language
3. Any table with speaker counts per language
4. Any statistics about Hindi (hi), Telugu (te), Malayalam (ml), Tamil (ta), Kannada (kn), Marathi (ma), Punjabi (pa), Assamese (as), Odia (or), Bengali (be), Gujarati (gj/gu)
Extract EVERY table with ALL numbers. Do not summarize.


Prompt: Extract ALL information, tables, and statistics about the IndicVoices dataset. I need per-language breakdowns of hours, transcribed vs untranscribed data, speaker counts, etc. Extract every table and number you find.


# IndicVoices Dataset: Information Extraction

## Overall Dataset Statistics

The README provides limited granular breakdowns. Here's what's available:

**Total Dataset Composition:**
- "12000 hours of read (8%), extempore (76%) and conversational (15%) audio from 22563 speakers covering 208 Indian districts and 22 languages"
- Transcribed portion: "3200 hours have already been transcribed, with a median of 122 hours per language"

## Key Findings

**What the content DOES NOT provide:**
- Per-language hour breakdowns
- Speaker counts by language
- Individual language transcription percentages
- District-level distribution details
- Detailed untranscribed vs. transcribed comparisons by language

## Available Resources

The document directs readers to access detailed data at: "https://ai4bharat.iitm.ac.in/datasets/indicvoices" where comprehensive language-specific statistics likely exist.

**Related Materials Referenced:**
- Research paper: https://arxiv.org/abs/2403.01926
- Data collection platforms: Kathbath (audio collection) and Shoonya (annotation/labeling)

## Conclusion

The GitHub repository README serves as a high-level overview rather than a detailed statistical breakdown. Comprehensive per-language metrics, speaker distributions, and transcription status by language are not included in this web content.


# IndicVoices Dataset Tables Extraction

## Table 1: Dataset Comparison (Existing Datasets vs IndicVoices)

| Dataset | #L | #Hours | #Sp | #D | Type | Channel | Un | Lb | R | E | C | WB | NB |
|---------|----|----|----|----|------|---------|----|----|---|---|---|----|----|
| CommonVoice | 8 | - | 373 | - | 4 | ✗ | ✗ | ✗ | ✓ | ✗ | - | - | - |
| FLEURS | 13 | - | 163 | - | - | ✓ | ✗ | ✗ | ✓ | ✗ | - | - | - |
| MSR | 3 | - | 150 | 1286 | 1 | ✗ | ✗ | ✓ | ✗ | ✓ | - | - | - |
| OpenSLR | 3 | - | 618 | 1513 | - | ✓ | ✗ | ✗ | ✓ | ✗ | - | - | - |
| CMS | 6 | - | 35 | 243 | 1 | ✓ | ✗ | ✗ | ✓ | ✗ | - | - | - |
| MUCS | 3 | - | 351 | 158 | 4 | ✓ | ✗ | ✗ | ✓ | ✗ | - | - | - |
| Kathbath | 12 | - | 1684 | 1218 | 3 | ✓ | ✗ | ✗ | ✓ | ✗ | - | - | - |
| Shrutilipi | 12 | - | 6457 | - | - | ✓ | ✗ | ✗ | ✓ | ✗ | - | - | - |
| Graamvaani | 1 | 1000 | 108 | - | - | ✗ | ✗ | ✓ | ✗ | ✓ | - | - | - |
| IISc-Mile | 2 | - | 500 | 1446 | - | ✓ | ✗ | ✗ | ✓ | ✗ | - | - | - |
| KDC | 1 | - | 1 | 43 | - | ✗ | ✗ | ✓ | ✗ | ✓ | - | - | - |
| Vāksañcayah | 1 | - | 78 | 27 | 8 | ✗ | ✗ | ✗ | ✓ | ✗ | - | - | - |
| IIITH-ISD | 7 | - | 11 | 35 | 1 | ✓ | ✗ | ✗ | ✓ | ✗ | - | - | - |
| IITB-MSC | 1 | - | 109 | 36 | 1 | ✓ | ✗ | ✗ | ✓ | ✗ | - | - | - |
| SMC-MSC | 1 | - | 2 | 75 | 4 | ✗ | ✗ | ✗ | ✗ | ✗ | - | - | - |
| IITM | 3 | - | 690 | - | - | ✓ | ✗ | ✗ | ✓ | ✗ | - | - | - |
| NPTEL | 8 | - | 857 | - | 1 | ✓ | ✗ | ✗ | ✓ | ✗ | - | - | - |
| IndicTTS | 13 | - | 225 | 25 | 4 | ✓ | ✗ | ✗ | ✓ | ✗ | - | - | - |
| Svarah | 1 | - | 10 | 117 | 37 | ✓ | ✓ | ✓ | ✓ | ✗ | - | - | - |
| SPRING-INX | 10 | - | 2005 | 7609 | 16 | ✓ | ✓ | ✓ | ✓ | ✓ | - | - | - |
| SPIRE-SIES | 1 | 171 | 23 | 1607 | - | ✗ | ✓ | ✗ | ✓ | ✗ | - | - | - |
| **IndicVoices** | **22** | **7348** | **16237** | **145** | **52** | **✓** | **✓** | **✓** | **✓** | **✓** | - | - | - |

---

## Table 2: Prompts and Sentences Count by Language

| Type | as | bn | brx | doi | gu | hi | kn | kok | ks | mai | ml | mni | mr | ne | or | pa | sa | sat | sd | ta | te | ur |
|------|----|----|-----|-----|----|----|----|----|----|----|-----|----|----|----|----|----|----|-----|----|----|----|-----|
| Sentences for read speech | 9146 | 9200 | 9199 | 8976 | 9200 | 9040 | 9006 | 9119 | 9135 | 9196 | 9200 | 8444 | 9065 | 9200 | 9199 | 8982 | 9199 | 9152 | 5036 | 9200 | 7721 | 9068 |
| Everyday tasks | 9330 | 9359 | 9359 | 8000 | 8958 | 6363 | 9354 | 9093 | 5297 | 7962 | 9357 | 9356 | 9354 | 9350 | 9355 | 8171 | 9342 | 9346 | 7988 | 9359 | 9345 | 9302 |
| Digital financial transactions | 9448 | 9448 | 9448 | 9448 | 9448 | 9448 | 9448 | 9447 | 9448 | 9448 | 9448 | 9448 | 9448 | 9448 | 9448 | 9448 | 9448 | 9448 | 9444 | 9448 | 9448 | 9448 |
| Online grocery transactions | 9335 | 9335 | 9335 | 9335 | 9335 | 9335 | 9335 | 9334 | 9335 | 9335 | 9335 | 9335 | 9335 | 9334 | 9335 | 9335 | 9335 | 9335 | 9331 | 9335 | 9334 | 9335 |
| Digital govt. services | 9274 | 9274 | 9274 | 9274 | 9274 | 9274 | 9274 | 9273 | 9274 | 9274 | 9274 | 9274 | 9274 | 9274 | 9274 | 9274 | 9274 | 9274 | 9274 | 9274 | 9274 | 9272 |
| Customer care interactions | 103 | 103 | 103 | 103 | 103 | 103 | 103 | 103 | 103 | 103 | 103 | 103 | 103 | 103 | 103 | 103 | 103 | 103 | 103 | 103 | 103 | 103 |
| Extempore questions | 2401 | 2401 | 2401 | 2401 | 2401 | 2401 | 2401 | 2401 | 2401 | 2401 | 2401 | 2401 | 2401 | 2401 | 2401 | 2401 | 1216 | 2401 | 2401 | 2401 | 2401 | 2401 |
| Icebreaker questions | 152 | 152 | 152 | 152 | 152 | 152 | 152 | 152 | 152 | 152 | 152 | 152 | 152 | 152 | 152 | 152 | 145 | 152 | 152 | 152 | 152 | 152 |
| Questions about named entities | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 |
| Role-play scenarios | 1185 | 1911 | 201 | 196 | 887 | 3256 | 1355 | 448 | 423 | 439 | 662 | 199 | 1488 | 426 | 2700 | 329 | 0000 | 245 | 425 | 1882 | 1804 | 2722 |

---

## Table 6: Train-Test-Valid Splits (Hours and Speakers)

| Language | Train #sp | Train #h | Test #sp | Test #h | Valid #sp | Valid #h | Language | Train #sp | Train #h | Test #sp | Test #h | Valid #sp | Valid #h |
|----------|-----------|---------|---------|---------|-----------|---------|----------|-----------|---------|---------|---------|-----------|---------|
| as | 684 | 100 | 163 | 5 | 1 | - | mni | 88 | 15 | 19 | 4 | 1 | - |
| bn | 590 | 100 | 125 | 5 | 1 | - | mr | 407 | 85 | 100 | 5 | 1 | - |
| brx | 694 | 100 | 164 | 5 | 1 | - | ne | 496 | 100 | 114 | 5 | 1 | - |
| doi | 321 | 65 | 84 | 5 | 1 | - | or | 391 | 75 | 92 | 5 | 1 | - |
| gu | 71 | 15 | 14 | 5 | 1 | - | pa | 274 | 71 | 61 | 5 | 0.4 | - |
| hi | 287 | 65 | 63 | 5 | 1 | - | sa | 176 | 45 | 33 | 5 | 1 | - |
| kn | 390 | 50 | 112 | 5 | 1 | - | sat | 237 | 86 | 53 | 5 | 1 | - |
| kok | 234 | 55 | 54 | 5 | 1 | - | sd | 108 | 10 | 30 | 5 | 1 | - |
| ks | 317 | 50 | 86 | 5 | 1 | - | ta | 863 | 100 | 195 | 5 | 1 | - |
| mai | 421 | 100 | 101 | 5 | 1 | - | te | 477 | 102 | 107 | 5 | 1 | - |
| ml | 332 | 60 | 67 | 5 | 1 | - | ur | 491 | 65 | 99 | 5 | 1 | - |

---

## Table 7: IndicASR Model Performance Comparison (WER)

| Model | as | bn | brx | doi | gu | hi | kn | kok | ks | mai | ml | mni | mr | ne | or | pa | sa | sat | sd | ta | te | ur |
|-------|----|----|-----|-----|----|----|----|----|----|----|-----|----|----|----|----|----|----|-----|----|----|----|-----|
| USM | 74.8 | - | - | - | 22.4 | 20.5 | 46.5 | - | - | - | 78.2 | - | 30.7 | 38.8 | - | 35.7 | - | - | 99.8 | 58.9 | 40.3 | 29.1 |
| Azure | - | 33.6 | - | - | 54.7 | 27.2 | 57.3 | - | - | - | 62.3 | - | 44.6 | 57.0 | - | 39.9 | - | - | - | 61.2 | 53.3 | 31.3 |
| Whisper | 127.2 | 93.2 | - | - | 60.5 | 34.0 | 96.5 | - | - | - | 148.6 | - | 95.2 | 100.8 | - | 89.1 | 99.0 | - | 92.6 | 78.4 | 151.9 | 29.7 |
| MMS | 51.0 | 44.4 | - | 77.6 | 42.1 | 38.9 | 70.7 | - | - | 77.1 | 76.8 | - | 52.7 | - | 54.6 | 44.0 | - | - | - | 75.4 | 62.4 | 43.6 |
| **IndicASR** | **20.4** | **15.9** | **25.9** | **33.3** | **21.4** | **15.0** | **30.3** | **31.4** | **39.3** | **33.6** | **40.5** | **26.1** | **18.2** | **16.4** | **23.4** | **12.9** | **22.8** | **35.4** | **29.6** | **31.2** | **26.8** | **14.4** |

---

## Key Regional Languages Data Summary

**Hindi (hi)**: 287 train speakers, 65 train hours, 63 test speakers, 5 test hours; IndicASR WER: 15.0

**Telugu (te)**: 477 train speakers, 102 train hours, 107 test speakers, 5 test hours; IndicASR WER: 26.8

**Malayalam (ml)**: 332 train speakers, 60 train hours, 67 test speakers, 5 test hours; IndicASR WER: 40.5

**Tamil (ta)**: 863 train speakers, 100 train hours, 195 test speakers, 5 test hours; IndicASR WER: 31.2

**Kannada (kn)**: 390 train speakers, 50 train hours, 112 test speakers, 5 test hours; IndicASR WER: 30.3

**Marathi (mr)**: 407 train speakers, 85 train hours, 100 test speakers, 5 test hours; IndicASR WER: 18.2

**Punjabi (pa)**: 274 train speakers, 71 train hours, 61 test speakers, 5 test hours; IndicASR WER: 12.9

**Assamese (as)**: 684 train speakers, 100 train hours, 163 test speakers, 5 test hours; IndicASR WER: 20.4

**Odia (or)**: 391 train speakers, 75 train hours, 92 test speakers, 5 test hours; IndicASR WER: 23.4

**Bengali (bn)**: 590 train speakers, 100 train hours, 125 test speakers, 5 test hours; IndicASR WER: 15.9

**Gujarati (gu)**: 71 train speakers, 15 train hours, 14 test speakers, 5 test hours; IndicASR WER: 21.4


Excellent, I got detailed data from the paper. Let me also check the IndicVoices-R subset and the ai4bharat page for updated numbers.


Prompt: Extract ALL information about IndicVoices-R dataset. This should be the read/transcribed subset. I need per-language hours, total hours, what languages are included, and any statistics tables. Extract everything.


Request failed with status code 401


Prompt: Extract ALL per-language statistics from this page. I need total hours per language, transcribed hours per language, speaker counts, any breakdown of read/extempore/conversational per language. Extract every table and number.


# Analysis of AI4Bharat Datasets Page

I cannot extract the requested per-language statistics from this page because the content provided is primarily CSS styling code and Next.js framework configuration.

The page content shows:

1. **Navigation structure** mentioning research areas (Transliteration, Machine Translation, AST, Speech Synthesis, LLMs)
2. **A prominent notice**: "We are currently moving all our datasets to HuggingFace, Datasets will be accesible soon."
3. **A call-to-action button** linking to HuggingFace

**No actual data tables or language-specific statistics are present** in the extracted HTML. The page appears to be a dataset landing page that is currently under migration, with actual dataset details not yet loaded or accessible in this page capture.

To obtain the language statistics you need, you would need to:
- Visit the HuggingFace organization directly (https://huggingface.co/ai4bharat)
- Check if there's a datasets page on the main AI4Bharat site with fully loaded content
- Contact the organization directly


Now let me compile all the information. Here's what I've gathered:

---

## IndicVoices Dataset — Transcribed vs Untranscribed Estimates

### Evolution of the Dataset (it's been growing)

| Source | Total Hours | Transcribed | Speakers | Districts |
|--------|------------|-------------|----------|-----------|
| **Paper (2024)** | 7,348 h | 1,639 h (22%) | 16,237 | 145 |
| **GitHub README** | 12,000 h | 3,200 h (27%) | 22,563 | 208 |
| **HF Page (latest)** | 23,700 h | 11,200 h (47%) | 51,000 | 400+ |

The dataset has grown ~3x since the paper. Transcription has grown from ~22% to ~47%.

### Audio Type Distribution
- **Read**: 8% (~1,896 h) — these are most likely to be transcribed (text is known a priori)
- **Extempore**: 76% (~18,012 h) — spontaneous speech, harder to transcribe
- **Conversational**: 15% (~3,555 h) — multi-speaker, hardest to transcribe

### Per-Language Transcribed Hours (from the paper's Table 6)

These are the **transcribed train hours** for your 11 languages. The paper had 7.3K total; the dataset has since grown ~3x, so I'm providing both the paper numbers and a scaled estimate:

| Language | Code | Paper Train (h) | Paper Test (h) | Paper Valid (h) | Est. Current Transcribed (h) |
|----------|------|-----------------|----------------|-----------------|------------------------------|
| Hindi | hi | 65 | 5 | 1 | ~195–210 |
| Telugu | te | 102 | 5 | 1 | ~306–340 |
| Malayalam | ml | 60 | 5 | 1 | ~180–200 |
| Tamil | ta | 100 | 5 | 1 | ~300–330 |
| Kannada | kn | 50 | 5 | 1 | ~150–170 |
| Marathi | mr | 85 | 5 | 1 | ~255–280 |
| Punjabi | pa | 71 | 5 | 0.4 | ~213–230 |
| Assamese | as | 100 | 5 | 1 | ~300–330 |
| Odia | or | 75 | 5 | 1 | ~225–250 |
| Bengali | bn | 100 | 5 | 1 | ~300–330 |
| Gujarati | gu | 15 | 5 | 1 | ~45–65 |
| **Total (11 langs)** | | **~823 h** | **55 h** | **~9.4 h** | **~2,470–2,735 h** |

### Key Observations

1. **Gujarati is an outlier** — only 15h train / 71 speakers in the paper, by far the smallest among your 11 languages. All others have 50–102h.

2. **~53% of the total dataset is still untranscribed** (12,500h of 23,700h on HF). For your 11 languages, if total hours scale proportionally (~12 of 22 languages), you likely have:
   - **~12,000–13,000 h total** across your 11 languages
   - **~5,500–6,000 h transcribed**
   - **~6,500–7,000 h untranscribed**

3. **The untranscribed portion is mostly extempore (76%) and conversational (15%)** — spontaneous speech that requires manual annotation. The "read" portion (8%) is likely mostly transcribed since the prompts serve as ground truth.

4. **What you already have**: You mentioned you have "everything that is transcribed" for these 11 languages. Based on HF's 11.2K total transcribed across 22 languages, your share is roughly **5,000–6,000 hours** of transcribed audio.

### To get exact numbers from your local data

If you want precise counts from what you've already downloaded:

```bash
# Count total hours from your parquet files
python -c "
import pandas as pd
from pathlib import Path
# Adjust path to your data
for lang_dir in sorted(Path('YOUR_DATA_PATH').iterdir()):
    if lang_dir.is_dir():
        total_dur = 0
        for pq in lang_dir.glob('*.parquet'):
            df = pd.read_parquet(pq, columns=['duration'])
            total_dur += df['duration'].sum()
        print(f'{lang_dir.name}: {total_dur/3600:.1f} hours')
"
```

Would you like me to run something like this on your actual downloaded data to get exact numbers?


no. can you somehow based on available parquets size, or somehow, recursively, or with atleast one parquet or few parquets estimate, get the available language distribution of untranscribed data distribution amopng the 11 languages ?


Let me check what you have locally and also query HF's API for parquet metadata.


Bash

find /home/ubuntu -maxdepth 4 -name "*.parquet" -path "*IndicVoices*" 2>/dev/null | head -30

Find IndicVoices parquet files locally


(Bash completed with no output)


find /home/ubuntu -maxdepth 4 -type d -iname "*indicvoice*" 2>/dev/null; find /home/ubuntu -maxdepth 4 -type d -iname "*indic_voice*" 2>/dev/null; find /home/ubuntu -maxdepth 3 -type d -iname "*ai4bharat*" 2>/dev/null

Find IndicVoices directories


WebFetch

Prompt: Extract ALL information. List every language subset, every split (train/test/valid), every parquet file URL and its size. I need to see the complete structure of all parquet files available. List everything.


Gated dataset. Let me try the HF dataset viewer API and search more broadly for your local data.


Prompt: Extract ALL information. I need: for each language config/subset, the number of rows, splits, features, dataset size in bytes, download size. Extract every number for every language.


Prompt: Extract ALL size information. For each language/config, I need: number of rows, dataset size in bytes, download size. List EVERYTHING for every language subset.


Bash

find /home/ubuntu -maxdepth 5 -name "*.parquet" 2>/dev/null | grep -i -E "hindi|telugu|malayalam|tamil|kannada|marathi|punjabi|assamese|odia|bengali|gujarati|indic" | head -30

Search for language-specific parquet files


All gated. Let me try the HF tree API for file sizes and also the dataset viewer's different endpoints.


WebFetch

Prompt: List ALL files with their sizes. I need the complete file tree with file sizes in bytes for every file in this repository.


Prompt: List ALL parquet files in the hindi folder with their sizes. Extract every file name and size.


Prompt: Extract all splits information - every config name, split name, and number of rows/examples for each.


# Repository File Tree

Based on the web page content provided, here are **all files with their sizes**:

## Files (2 total)
| Path | Size (bytes) |
|------|-------------|
| `.gitattributes` | 2,461 |
| `README.md` | 38,801 |

## Directories (22 total)
The repository also contains 22 language-specific directories (all empty, size 0):
- assamese/
- bengali/
- bodo/
- dogri/
- gujarati/
- hindi/
- kannada/
- kashmiri/
- konkani/
- maithili/
- malayalam/
- manipuri/
- marathi/
- nepali/
- odia/
- punjabi/
- sanskrit/
- santali/
- sindhi/
- tamil/
- telugu/
- urdu/

**Note:** The provided content only shows the top-level directory structure. To get a complete file tree with sizes for files nested within the 22 language directories, you would need to access the full repository contents, as the current data only displays the root level.


# Hindi Folder Parquet Files

Based on the webpage content, here are all the parquet files in the hindi folder with their sizes:

| File Name | Size |
|-----------|------|
| train-00000-of-00064.parquet | 489 MB |
| train-00001-of-00064.parquet | 533 MB |
| train-00002-of-00064.parquet | 493 MB |
| train-00003-of-00064.parquet | 522 MB |
| train-00004-of-00064.parquet | 482 MB |
| train-00005-of-00064.parquet | 493 MB |
| train-00006-of-00064.parquet | 546 MB |
| train-00007-of-00064.parquet | 540 MB |
| train-00008-of-00064.parquet | 518 MB |
| train-00009-of-00064.parquet | 483 MB |
| train-00010-of-00064.parquet | 509 MB |
| train-00011-of-00064.parquet | 556 MB |
| train-00012-of-00064.parquet | 556 MB |
| train-00013-of-00064.parquet | 546 MB |
| train-00014-of-00064.parquet | 521 MB |
| train-00015-of-00064.parquet | 541 MB |
| train-00016-of-00064.parquet | 540 MB |
| train-00017-of-00064.parquet | 509 MB |
| train-00018-of-00064.parquet | 546 MB |
| train-00019-of-00064.parquet | 517 MB |
| train-00020-of-00064.parquet | 547 MB |
| train-00021-of-00064.parquet | 560 MB |
| train-00022-of-00064.parquet | 540 MB |
| train-00023-of-00064.parquet | 671 MB |
| train-00024-of-00064.parquet | 631 MB |
| train-00025-of-00064.parquet | 606 MB |
| train-00026-of-00064.parquet | 629 MB |
| train-00027-of-00064.parquet | 677 MB |
| train-00028-of-00064.parquet | 649 MB |
| train-00029-of-00064.parquet | 646 MB |
| train-00030-of-00064.parquet | 689 MB |
| train-00031-of-00064.parquet | 687 MB |
| train-00032-of-00064.parquet | 705 MB |
| train-00033-of-00064.parquet | 745 MB |
| train-00034-of-00064.parquet | 714 MB |
| train-00035-of-00064.parquet | 728 MB |
| train-00036-of-00064.parquet | 634 MB |
| train-00037-of-00064.parquet | 593 MB |
| train-00038-of-00064.parquet | 643 MB |
| train-00039-of-00064.parquet | 651 MB |
| train-00040-of-00064.parquet | 619 MB |
| train-00041-of-00064.parquet | 638 MB |
| train-00042-of-00064.parquet | 685 MB |
| train-00043-of-00064.parquet | 639 MB |
| train-00044-of-00064.parquet | 640 MB |
| train-00045-of-00064.parquet | 652 MB |
| train-00046-of-00064.parquet | 674 MB |
| train-00047-of-00064.parquet | 627 MB |
| train-00048-of-00064.parquet | 668 MB |
| train-00049-of-00064.parquet | 644 MB |

**Note:** The page indicates "Load more files" at the end, suggesting there are additional files (up to train-00063-of-00064.parquet based on the naming convention), but they are not shown in the provided content.


Now I can see the parquet structure. Hindi has 64 train shards with embedded audio. Let me get file counts/sizes for all your 11 languages to estimate the distribution.


Prompt: List ALL parquet files with sizes. How many total train files are there (e.g. train-00000-of-00XXX)? What is the total number of shards and approximate total size? Also list any test/valid parquet files.


Prompt: List ALL parquet files with sizes. How many total train shards? Total size estimate?


Prompt: List ALL parquet files with sizes. How many total train shards? Total size estimate?


Prompt: List ALL parquet files with sizes. How many total train shards? Total size estimate?


Prompt: List ALL parquet files with sizes. How many total train shards? Total size estimate?


# IndicVoices Dataset - Parquet Files Summary

## Train Files Listed
Based on the content shown, there are **56 total train files** (train-00000-of-00056 through train-00055-of-00056).

### Files Shown (00000-00049):
- **train-00000 to train-00022**: ~454-509 MB each (23 files)
- **train-00023 to train-00049**: ~558-807 MB each (27 files)

**Approximate sizes of visible files:**
- Files 00000-00022: ~11.2 GB
- Files 00023-00049: ~18.5 GB
- **Visible subtotal: ~29.7 GB** (for 50 of 56 files)

## Total Dataset Estimate

| Metric | Value |
|--------|-------|
| **Total Shards** | 56 |
| **Estimated Total Size** | **~26-30 GB** (based on 1M-10M size category) |
| **Language** | Telugu |
| **Format** | Parquet |
| **License** | CC-BY-4.0 |

## Test/Validation Files
**None listed** in the provided content. Only training files (train-00000-of-00056) are shown.

## Additional Details
- **Modalities**: Audio + Text
- **Status**: Gated dataset (requires login to access files)
- **ArXiv Reference**: 2403.01926
- **Last Updated**: 3 months ago


# IndicVoices Dataset - Parquet Files Analysis

## Total Train Shards
**88 shards total** (indicated by "train-00000-of-00088" through "train-00087-of-00088")

## Listed Parquet Files (First 50 shown)
The page displays files 00000-00049 with sizes ranging from **429 MB to 485 MB**:

| Shard | Size |
|-------|------|
| 00000-00049 | 429-485 MB each |

## Total Size Estimate

**Visible files (00000-00049): ~23 GB**
- 50 shards × average ~460 MB = ~23 GB

**Complete dataset (all 88 shards): ~40.5 GB estimate**
- 88 shards × average ~460 MB ≈ **40-41 GB total**

## Key Details
- **Format**: Parquet
- **Language**: Malayalam (shown in file paths)
- **Size Category**: 1M-10M rows
- **License**: CC-BY-4.0
- **Status**: Gated dataset (requires login to access files)
- **ArXiv**: 2403.01926

The dataset is split into 88 parquet shards, each approximately 430-485 MB, with consistent file sizes across the distribution.


# IndicVoices Dataset - Parquet Files Summary

## Train Shards
**Total: 32 shards** (train-00000-of-00032 through train-00031-of-00032)

## All Parquet Files with Sizes

### Training Files (32 shards)
| File | Size |
|------|------|
| train-00000-of-00032.parquet | 457 MB |
| train-00001-of-00032.parquet | 440 MB |
| train-00002-of-00032.parquet | 474 MB |
| train-00003-of-00032.parquet | 481 MB |
| train-00004-of-00032.parquet | 483 MB |
| train-00005-of-00032.parquet | 456 MB |
| train-00006-of-00032.parquet | 491 MB |
| train-00007-of-00032.parquet | 458 MB |
| train-00008-of-00032.parquet | 419 MB |
| train-00009-of-00032.parquet | 369 MB |
| train-00010-of-00032.parquet | 352 MB |
| train-00011-of-00032.parquet | 278 MB |
| train-00012-of-00032.parquet | 286 MB |
| train-00013-of-00032.parquet | 397 MB |
| train-00014-of-00032.parquet | 437 MB |
| train-00015-of-00032.parquet | 452 MB |
| train-00016-of-00032.parquet | 411 MB |
| train-00017-of-00032.parquet | 436 MB |
| train-00018-of-00032.parquet | 448 MB |
| train-00019-of-00032.parquet | 430 MB |
| train-00020-of-00032.parquet | 418 MB |
| train-00021-of-00032.parquet | 406 MB |
| train-00022-of-00032.parquet | 435 MB |
| train-00023-of-00032.parquet | 436 MB |
| train-00024-of-00032.parquet | 439 MB |
| train-00025-of-00032.parquet | 454 MB |
| train-00026-of-00032.parquet | 429 MB |
| train-00027-of-00032.parquet | 415 MB |
| train-00028-of-00032.parquet | 458 MB |
| train-00029-of-00032.parquet | 431 MB |
| train-00030-of-00032.parquet | 430 MB |
| train-00031-of-00032.parquet | 392 MB |

### Validation File
| File | Size |
|------|------|
| valid-00000-of-00001.parquet | 203 MB |

## Total Size Estimate

**Training Data:** ~13.1 GB (32 shards × ~410 MB average)
**Validation Data:** 203 MB
**Total Dataset:** **~13.3 GB**


# IndicVoices Dataset - Parquet Files Analysis

## Total Train Shards
**73 shards** (indicated by `train-00000-of-00073` through `train-00072-of-00073`)

## Listed Parquet Files (First 50 of 73)

| File | Size |
|------|------|
| train-00000-of-00073.parquet | 519 MB |
| train-00001-of-00073.parquet | 505 MB |
| train-00002-of-00073.parquet | 541 MB |
| train-00003-of-00073.parquet | 500 MB |
| train-00004-of-00073.parquet | 541 MB |
| train-00005-of-00073.parquet | 472 MB |
| train-00006-of-00073.parquet | 478 MB |
| train-00007-of-00073.parquet | 446 MB |
| train-00008-of-00073.parquet | 451 MB |
| train-00009-of-00073.parquet | 457 MB |
| train-00010-of-00073.parquet | 428 MB |
| train-00011-of-00073.parquet | 420 MB |
| train-00012-of-00073.parquet | 417 MB |
| train-00013-of-00073.parquet | 405 MB |
| train-00014-of-00073.parquet | 435 MB |
| train-00015-of-00073.parquet | 463 MB |
| train-00016-of-00073.parquet | 462 MB |
| train-00017-of-00073.parquet | 459 MB |
| train-00018-of-00073.parquet | 423 MB |
| train-00019-of-00073.parquet | 454 MB |
| train-00020-of-00073.parquet | 510 MB |
| train-00021-of-00073.parquet | 474 MB |
| train-00022-of-00073.parquet | 475 MB |
| train-00023-of-00073.parquet | 456 MB |
| train-00024-of-00073.parquet | 480 MB |
| train-00025-of-00073.parquet | 473 MB |
| train-00026-of-00073.parquet | 436 MB |
| train-00027-of-00073.parquet | 455 MB |
| train-00028-of-00073.parquet | 470 MB |
| train-00029-of-00073.parquet | 461 MB |
| train-00030-of-00073.parquet | 450 MB |
| train-00031-of-00073.parquet | 490 MB |
| train-00032-of-00073.parquet | 466 MB |
| train-00033-of-00073.parquet | 511 MB |
| train-00034-of-00073.parquet | 514 MB |
| train-00035-of-00073.parquet | 483 MB |
| train-00036-of-00073.parquet | 419 MB |
| train-00037-of-00073.parquet | 499 MB |
| train-00038-of-00073.parquet | 476 MB |
| train-00039-of-00073.parquet | 428 MB |
| train-00040-of-00073.parquet | 413 MB |
| train-00041-of-00073.parquet | 405 MB |
| train-00042-of-00073.parquet | 405 MB |
| train-00043-of-00073.parquet | 409 MB |
| train-00044-of-00073.parquet | 399 MB |
| train-00045-of-00073.parquet | 412 MB |
| train-00046-of-00073.parquet | 436 MB |
| train-00047-of-00073.parquet | 405 MB |
| train-00048-of-00073.parquet | 398 MB |
| train-00049-of-00073.parquet | 408 MB |

## Total Size Estimate

**First 50 shards average:** ~456 MB per file

**Estimated total size for all 73 shards:** **~33.3 GB**

*(Based on average of 456 MB × 73 shards)*

The remaining 23 shards (00050-00072) are not shown in the page preview but follow the similar size pattern of 390-540 MB each.


# IndicVoices Dataset - Parquet Files Summary

## Total Train Shards
**78 shards** (indicated by "train-00000-of-00078" through "train-00077-of-00078")

## Listed Parquet Files (0-49 shown)
Here are all 50 files visible in the preview:

| File | Size |
|------|------|
| train-00000-of-00078.parquet | 545 MB |
| train-00001-of-00078.parquet | 550 MB |
| train-00002-of-00078.parquet | 541 MB |
| train-00003-of-00078.parquet | 560 MB |
| train-00004-of-00078.parquet | 610 MB |
| train-00005-of-00078.parquet | 545 MB |
| train-00006-of-00078.parquet | 555 MB |
| train-00007-of-00078.parquet | 590 MB |
| train-00008-of-00078.parquet | 577 MB |
| train-00009-of-00078.parquet | 553 MB |
| train-00010-of-00078.parquet | 547 MB |
| train-00011-of-00078.parquet | 587 MB |
| train-00012-of-00078.parquet | 556 MB |
| train-00013-of-00078.parquet | 525 MB |
| train-00014-of-00078.parquet | 546 MB |
| train-00015-of-00078.parquet | 561 MB |
| train-00016-of-00078.parquet | 552 MB |
| train-00017-of-00078.parquet | 540 MB |
| train-00018-of-00078.parquet | 557 MB |
| train-00019-of-00078.parquet | 558 MB |
| train-00020-of-00078.parquet | 565 MB |
| train-00021-of-00078.parquet | 577 MB |
| train-00022-of-00078.parquet | 596 MB |
| train-00023-of-00078.parquet | 536 MB |
| train-00024-of-00078.parquet | 546 MB |
| train-00025-of-00078.parquet | 561 MB |
| train-00026-of-00078.parquet | 583 MB |
| train-00027-of-00078.parquet | 582 MB |
| train-00028-of-00078.parquet | 572 MB |
| train-00029-of-00078.parquet | 513 MB |
| train-00030-of-00078.parquet | 565 MB |
| train-00031-of-00078.parquet | 577 MB |
| train-00032-of-00078.parquet | 555 MB |
| train-00033-of-00078.parquet | 677 MB |
| train-00034-of-00078.parquet | 798 MB |
| train-00035-of-00078.parquet | 818 MB |
| train-00036-of-00078.parquet | 777 MB |
| train-00037-of-00078.parquet | 786 MB |
| train-00038-of-00078.parquet | 782 MB |
| train-00039-of-00078.parquet | 800 MB |
| train-00040-of-00078.parquet | 818 MB |
| train-00041-of-00078.parquet | 794 MB |
| train-00042-of-00078.parquet | 839 MB |
| train-00043-of-00078.parquet | 809 MB |
| train-00044-of-00078.parquet | 705 MB |
| train-00045-of-00078.parquet | 655 MB |
| train-00046-of-00078.parquet | 626 MB |
| train-00047-of-00078.parquet | 660 MB |
| train-00048-of-00078.parquet | 666 MB |
| train-00049-of-00078.parquet | 669 MB |

## Total Size Estimate

**Shown files (0-49): ~31.8 GB**

**Full dataset estimate (all 78 shards): ~49-50 GB**

The page shows "Load more files" indicating shards 50-77 are not displayed. Based on the pattern where shards 0-32 average ~560 MB and shards 33-49 average ~750 MB, the remaining shards likely continue this higher size trend.


Good data coming in. Let me get the remaining 6 languages.


Prompt: How many total train shards (train-XXXXX-of-NNNNN)? What's the average file size? List any test/valid files too.


Prompt: How many total train shards (train-XXXXX-of-NNNNN)? What's the average file size? List any test/valid files too.


Prompt: How many total train shards (train-XXXXX-of-NNNNN)? What's the average file size? List any test/valid files too.


Prompt: How many total train shards (train-XXXXX-of-NNNNN)? What's the average file size? List any test/valid files too.


Prompt: How many total train shards (train-XXXXX-of-NNNNN)? What's the average file size? List any test/valid files too.


# IndicVoices Dataset Structure

## Total Train Shards
**75 shards** (train-00000-of-00075 through train-00074-of-00075)

## Average File Size
Based on the visible files (train-00000 through train-00049):
- **~500 MB per file** (ranging from 376 MB to 559 MB)
- Total visible: ~25 GB for first 50 shards
- **Estimated total dataset size: ~37.5 GB** (75 shards × ~500 MB average)

## Test/Validation Files
**No test or validation files are shown** in the provided file listing. Only training files are visible in the Marathi language directory.

**Note:** This is a **gated dataset** requiring login/signup to access the actual file contents. The page shows file listings but access to download is restricted.


# IndicVoices Dataset Analysis

## Total Train Shards
Based on the file naming pattern `train-XXXXX-of-00086`, there are **86 total train shards**.

## Average File Size
From the visible files (train-00000 through train-00049):
- **Range**: 364 MB to 507 MB
- **Average**: Approximately **410-420 MB per shard**
- **Estimated total dataset size**: ~35-36 GB (86 shards × ~410 MB)

## Test/Valid Files
The webpage content shows **no test or validation split files** listed. Only training data files (parquet format) are displayed for the Punjabi language subset.

## Summary
- **Format**: Parquet
- **Language shown**: Punjabi
- **Total shards**: 86
- **File size range**: 364-507 MB
- **No separate test/validation sets visible** in the current view

*Note: This is a gated dataset requiring authentication to access the actual files.*


# IndicVoices Dataset Analysis

## Total Train Shards

Based on the naming convention `train-XXXXX-of-00125`, there are **125 total train shards** in this dataset.

## Average File Size

From the visible files (train-00000 through train-00049):
- **Sum of shown files**: ~19,240 MB (19.24 GB for 50 files)
- **Average file size**: **~385 MB per shard**
  - Range: 261 MB (train-00038) to 510 MB (train-00041)

Extrapolating across all 125 shards: **~48 GB total for training data**

## Test/Validation Files

**None listed** in the provided content. The preview only shows:
- Train files from the `assamese/` directory
- The naming pattern indicates only training data (`train-XXXXX-of-00125`)
- No separate `test-*` or `valid-*` files are visible

The dataset appears to contain only training splits across multiple languages (at minimum Assamese, based on the directory structure shown).


# IndicVoices Dataset Shards Analysis

## Train Shards
**Total: 96 shards** (as indicated by the `of-00096` suffix in filenames)

The page shows files `train-00000-of-00096` through `train-00049-of-00096` (50 files visible), with more available via the "Load more files" option.

## Average File Size

Based on the visible train files (train-00000 through train-00049):

**~400 MB average per file**

Sample sizes:
- Files 00000-00014: ~450-490 MB range
- Files 00015-00035: ~330-370 MB range
- Files 00036-00049: ~380-445 MB range

With 96 total shards at ~400 MB average, the complete dataset is approximately **~38 GB**

## Test/Validation Files

**None listed.** The preview only shows training files. No separate `test-` or `valid-` partitions are displayed in the file listing.

---

**Note:** This is a gated dataset requiring authentication to access the actual files. The dataset contains audio and text modalities in Parquet format, with a CC-BY-4.0 license.


# IndicVoices Dataset Analysis

## Total Train Shards
Based on the filenames shown, there are **89 total train shards** (as indicated by the `of-00089` pattern in files like `train-00000-of-00089.parquet`).

The page displays files 00000 through 00049 (50 files shown), with a "Load more files" option indicating additional shards exist.

## Average File Size

From the 50 visible files:
- **Range**: 370 MB to 605 MB
- **Approximate total**: ~22,500 MB (22.5 GB for the 50 visible files)
- **Average**: **~450 MB per file**

This suggests the complete dataset is approximately **40 GB** in total (89 shards × ~450 MB average).

## Test/Validation Files

**None are visible in the current preview.** The page only shows training files from the `odia/` directory with the naming pattern `train-XXXXX-of-00089.parquet`.

No separate `test` or `valid` split files are listed in the displayed content. To confirm whether these splits exist, you would need to:
- Authenticate and access the full file browser
- Check the dataset card documentation
- Explore other language directories (this shows only Odia files)


Now I have all the data. Let me compile this into a comprehensive estimate.

Here's the full picture based on HuggingFace parquet shard counts and sizes for your 11 languages:

## Total Data per Language (from parquet sizes — includes audio blobs)

| Language | Code | Train Shards | Avg Size/Shard | Est. Total Size (GB) | Paper Transcribed (h) |
|----------|------|-------------|----------------|----------------------|----------------------|
| **Assamese** | as | 125 | ~385 MB | **~48 GB** | 100 |
| **Bengali** | bn | 96 | ~400 MB | **~38 GB** | 100 |
| **Punjabi** | pa | 86 | ~415 MB | **~36 GB** | 71 |
| **Malayalam** | ml | 88 | ~460 MB | **~40 GB** | 60 |
| **Odia** | or | 89 | ~450 MB | **~40 GB** | 75 |
| **Tamil** | ta | 78 | ~620 MB | **~48 GB** | 100 |
| **Marathi** | mr | 75 | ~500 MB | **~37 GB** | 85 |
| **Kannada** | kn | 73 | ~456 MB | **~33 GB** | 50 |
| **Hindi** | hi | 64 | ~590 MB | **~38 GB** | 65 |
| **Telugu** | te | 56 | ~530 MB | **~30 GB** | 102 |
| **Gujarati** | gu | 32 | ~425 MB | **~13 GB** | 15 |

### Estimated Hours (using ~80 MB/hour for compressed parquet with embedded audio)

| Language | Est. Total Hours | Transcribed (paper) | Est. Untranscribed | % Untranscribed |
|----------|-----------------|--------------------|--------------------|-----------------|
| **Assamese** | ~600 h | ~100 h | **~500 h** | 83% |
| **Bengali** | ~475 h | ~100 h | **~375 h** | 79% |
| **Tamil** | ~600 h | ~100 h | **~500 h** | 83% |
| **Malayalam** | ~500 h | ~60 h | **~440 h** | 88% |
| **Odia** | ~500 h | ~75 h | **~425 h** | 85% |
| **Punjabi** | ~450 h | ~71 h | **~379 h** | 84% |
| **Hindi** | ~475 h | ~65 h | **~410 h** | 86% |
| **Marathi** | ~462 h | ~85 h | **~377 h** | 82% |
| **Kannada** | ~412 h | ~50 h | **~362 h** | 88% |
| **Telugu** | ~375 h | ~102 h | **~273 h** | 73% |
| **Gujarati** | ~163 h | ~15 h | **~148 h** | 91% |
| **TOTAL** | **~5,012 h** | **~823 h** | **~4,189 h** | **~84%** |

### Key Takeaways

1. **~84% of the data across your 11 languages is untranscribed** — roughly 4,200 hours without transcriptions vs 823 hours with.

2. **Largest untranscribed pools**: Assamese (~500h), Tamil (~500h), Malayalam (~440h), Odia (~425h) — these are your biggest opportunities for pseudo-labeling or manual transcription.

3. **Gujarati is severely underrepresented** — only ~163h total and just 15h transcribed. It's an outlier among your 11 languages.

4. **Telugu has the best transcription ratio** (73% untranscribed) — it has the most transcribed hours (102h) relative to its total.

5. **Since the paper, the dataset has grown ~3x** (from 7.3K to 23.7K total). The transcribed portion has grown too (1.6K to 11.2K), but the per-language numbers from the paper are the floor — actual current transcribed hours are likely 2-3x higher.

**Caveat**: The MB/hour ratio (~80 MB/h) is a rough estimate. The actual ratio depends on audio sample rate, encoding, and parquet compression. The shard counts and GB numbers are solid; the hour estimates have ~20-30% margin of error. Want me to download a single parquet shard to calibrate the ratio more precisely?


let's not trust the numbers from paper to be GT as transcribed content. as they said in the repo, 11k is transcribed.


You're right. Let me redo this properly. The repo says **11.2K transcribed** out of **23.7K total**. The paper's per-language numbers are stale.

The problem: we know total data per language (from parquet sizes), but we don't know how the 11.2K transcribed hours are distributed across languages. Two approaches:

1. **Assume proportional** — each language has ~47% transcribed (11.2K/23.7K)
2. **Peek at a parquet** — check if there's a `transcription` column that's null/empty for untranscribed rows, giving us exact counts

Let me try option 2 first — it'll give us ground truth if the parquets have that info. I'll check the dataset viewer API for the schema.


Prompt: Extract the COMPLETE schema/features of this dataset. What columns/fields does each row have? Specifically look for any column related to transcription, transcript, text, sentence, label, etc. Also tell me if any text/transcription field is null or empty in the sample rows shown. List ALL column names and their types.


Request failed with status code 401


Prompt: Extract ALL column names and types. Look for transcription/text fields. Are any rows missing transcriptions (null/empty)? Show sample values for text-related columns.


Gated. Let me check the README's feature description and the GitHub repo for schema info.


Prompt: Extract the dataset schema/columns. What fields does each row in the parquet files have? Look for column names like: audio, transcription, text, sentence, duration, language, speaker, type, etc. Also look for any mention of how transcribed vs untranscribed data is differentiated in the parquets. Extract ALL column/field information.


Prompt: Extract the column names shown in the dataset viewer table. What are all the field/column names? Are there any rows where transcription or text is empty/null? List all column names you can see.


Based on the provided GitHub repository content, **no parquet file schema or column definitions are documented**.

The README mentions:
- "3200 hours have already been transcribed" and references "extempore (76%) and conversational (15%) audio"
- Data preparation steps involving audio downsampling and manifest file creation
- Model training with language IDs (`langid=$LANGID`)

However, the actual parquet column names, field structures, and how transcribed versus untranscribed data is differentiated are **not specified in this documentation**. The repository appears to focus on data collection infrastructure (Kathbath platform), transcription tools (Shoonya), and ASR model training rather than schema documentation.

To find this information, you would need to:
1. Examine the actual parquet files in the downloaded dataset
2. Check the AI4Bharat IndicVoices download portal directly
3. Review the arxiv paper (2403.01926) referenced in the README


The APIs are all gated. Let me check what data you actually have locally — you said you have the transcribed portion already.


Bash

find /home/ubuntu -maxdepth 5 -type d -name "*indic*" -o -name "*ai4bharat*" -o -name "*voice*" 2>/dev/null | grep -iv node_modules | head -30

Find IndicVoices data directories


ls /home/ubuntu/training/ 2>/dev/null | head -40

List training directory contents


api_results
api_results.zip
asr-benchmark-dashboard
benchmark_gemma3n_asr.py
benchmark_maya_asr_tdt.py
benchmark_outputs
benchmark_qwen3_asr.py
benchmark_results
benchmark_results_gemma3n
benchmark_schema
benchmark_tokenizers.py
checkpoints
convert_to_schema_v1.py
datasets
gemma3n-E2B-asr-ft
generate_schema_outputs.py
indic-asr-benchmark-6k
maya-asr-hybrid-fastconformer-rnnt-stage1
maya-tts-llama32-1b-pretraining
__pycache__
qwen3-asr-1.7b-phase2-sft
test_audio


ls /home/ubuntu/training/datasets/ 2>/dev/null | head -40

List datasets directory


find /home/ubuntu -maxdepth 6 -name "*.parquet" 2>/dev/null | grep -iv benchmark | head -30

Find non-benchmark parquet files


You don't have the IndicVoices parquets locally. Let me get the remaining 11 languages' parquet sizes so I can compute each language's share of the full 23.7K/11.2K pool properly.


WebFetch

Prompt: How many total train shards (look at the "of-NNNNN" in filename)? What's the average file size in MB?


Prompt: How many total train shards (look at the "of-NNNNN" in filename)? What's the average file size in MB?


Prompt: How many total train shards (look at the "of-NNNNN" in filename)? What's the average file size in MB?


Prompt: How many total train shards (look at the "of-NNNNN" in filename)? What's the average file size in MB?


Prompt: How many total train shards (look at the "of-NNNNN" in filename)? What's the average file size in MB?


# IndicVoices Dataset Analysis

## Total Train Shards
The dataset contains **56 total train shards**, as indicated by the "of-00056" pattern in the filenames (e.g., `train-00000-of-00056.parquet`).

## Average File Size
Based on the visible files listed (train-00000 through train-00049), the file sizes range from approximately 389 MB to 562 MB.

**Calculating the average from the 50 visible files:**
- Sum of all visible file sizes: ~25,305 MB
- Number of visible files: 50
- **Average file size: ~506 MB**

This is a reasonable estimate for all 56 shards, as the visible files appear to be representative of the full dataset's distribution.


# IndicVoices Dataset Analysis

Based on the web page content, here are the answers:

## Total Train Shards
**56 shards** - This is evident from the filenames which all follow the pattern `train-XXXXX-of-00056.parquet`

## Average File Size
Looking at the 50 files displayed on the page:

- **Sum of visible files**: ~19,445 MB
- **Number of visible files**: 50
- **Average of visible files**: ~388.9 MB

Given that all 56 shards appear to be consistently sized (ranging from ~313 MB to ~518 MB with most clustering in the 350-450 MB range), the **estimated average across all 56 shards is approximately 390-400 MB**.

A reasonable estimate for the complete dataset would be:
- **56 shards × ~390 MB ≈ 21.8 TB total** (approximate dataset size)


# IndicVoices Dataset Analysis

## Total Train Shards
Based on the filenames, there are **44 total train shards** (as indicated by `train-00000-of-00044` through `train-00043-of-00044`).

## Average File Size
Summing all 44 train parquet files:

- Files 0-14: ~6,893 MB
- Files 15-27: ~15,087 MB
- Files 28-43: ~19,434 MB
- **Total: ~21,414 MB**

**Average file size: ~486 MB per shard**

(Approximately 21,414 MB ÷ 44 shards = 486.6 MB)


# IndicVoices Dataset Analysis

## Total Train Shards
Based on the filenames shown, there are **59 total train shards** (indicated by `train-XXXXX-of-00059.parquet`).

## Average File Size

From the 50 files listed in the preview, the file sizes range from 400 MB to 608 MB.

**Calculation:**
- Sum of all 50 visible files: ~23,555 MB
- Average: **~471 MB per file**

Using this average across all 59 shards:
- **Estimated total dataset size: ~27,789 MB (~27.8 GB)**

The file sizes are fairly consistent (mostly 450-500 MB), with some variation in the later shards (files 44-49 are notably larger at 558-608 MB).


# IndicVoices Dataset Analysis

## Total Train Shards
Based on the filenames, there are **41 total train shards** (as indicated by "of-00041" in the filenames like `train-00000-of-00041.parquet`).

## Average File Size

Looking at the 41 training files listed:

**Sum of all train file sizes:**
- Files 00000-00008: 489 + 481 + 494 + 483 + 475 + 490 + 461 + 362 + 277 = 3,912 MB
- Files 00009-00023: 330 + 321 + 273 + 331 + 321 + 308 + 284 + 361 + 357 + 343 + 369 + 337 + 367 + 339 + 350 = 4,971 MB
- Files 00024-00040: 471 + 488 + 470 + 463 + 414 + 418 + 398 + 444 + 398 + 410 + 411 + 407 + 399 + 405 + 410 + 399 + 389 = 6,872 MB

**Total: ~15,755 MB**

**Average file size: 15,755 ÷ 41 ≈ 384 MB per shard**


Prompt: How many total train shards (look at the "of-NNNNN" in filename)? What's the average file size in MB?


Prompt: How many total train shards (look at the "of-NNNNN" in filename)? What's the average file size in MB?


Prompt: How many total train shards (look at the "of-NNNNN" in filename)? What's the average file size in MB?


Prompt: How many total train shards (look at the "of-NNNNN" in filename)? What's the average file size in MB?


Prompt: How many total train shards (look at the "of-NNNNN" in filename)? What's the average file size in MB?


# IndicVoices Dataset Analysis

## Total Train Shards
Looking at the filenames, the train shards are numbered from `train-00000-of-00036` to `train-00035-of-00036`, which indicates there are **36 total train shards**.

## Average File Size

Adding up all the visible train file sizes:
- Files 00000-00025: 535 + 521 + 522 + 482 + 496 + 509 + 510 + 490 + 468 + 490 + 522 + 495 + 451 + 477 + 518 + 502 + 509 + 532 + 529 + 521 + 532 + 509 + 527 + 505 + 530 + 434 = **12,896 MB**
- Files 00026-00035: 283 + 284 + 272 + 283 + 292 + 292 + 266 + 298 + 276 + 277 = **2,783 MB**

**Total: 15,679 MB**

**Average file size: 15,679 ÷ 36 = ~435.5 MB per shard**


Prompt: How many total train shards (look at the "of-NNNNN" in filename)? What's the average file size in MB?


# IndicVoices Dataset Analysis

## Total Train Shards
Based on the filenames, there are **50 total train shards** (files are numbered from `train-00000-of-00049.parquet` to `train-00048-of-00049.parquet`, where "of-00049" indicates 50 total shards: 0-49).

## Average File Size
Looking at the listed files (train shards 0-48 are shown):

- **Shards 0-20**: Average ~460 MB
- **Shards 21-48**: Average ~560 MB
- **Overall average**: Approximately **520 MB per shard**

There's a noticeable size increase starting from shard 21, with later shards being roughly 100 MB larger than earlier ones.


# IndicVoices Dataset Analysis

Based on the file listing in the webpage:

## Total Train Shards
**50 shards** (evident from filenames like `train-00000-of-00050` through `train-00049-of-00050`)

## Average File Size
**~492 MB**

**Calculation:**
- Total visible files shown: 50 shards
- Sum of all file sizes: ~24,600 MB
- Average: 24,600 ÷ 50 = **492 MB per shard**

**Total dataset size (Sanskrit split):** Approximately **24.6 GB** (50 shards × ~492 MB)


# IndicVoices Dataset Analysis

## Total Train Shards
Looking at the filenames, the train shards are numbered from `train-00000-of-00044` through `train-00043-of-00044`.

**Total train shards: 44**

## Average File Size

Summing all 44 train file sizes:
- Files 00000-00018: ~514-545 MB each (avg ~520 MB)
- Files 00019-00027: ~459-498 MB each (avg ~480 MB)
- Files 00028-00043: ~495-544 MB each (avg ~515 MB)

Total size: ~22,095 MB
**Average file size: ~502 MB per shard**

(Plus 1 validation shard of 218 MB)


# IndicVoices Dataset Analysis

## Total Train Shards
Based on the filenames, there are **58 total train shards** (as indicated by `of-00058` in the filenames).

## Average File Size

From the visible files listed (train-00000 through train-00049), the file sizes range from approximately 499 MB to 605 MB.

**Average file size: ~545 MB**

This is calculated from the 50 visible files totaling approximately 27,250 MB (27.25 GB), divided by 50 files.

Given the consistent file sizes throughout the dataset, the full 58 shards would total approximately **31.6 GB** (58 shards × 545 MB average).


# IndicVoices Dataset Analysis

Based on the file listing shown, here are the answers:

## Total Train Shards
**59 shards** - as indicated by the "of-00059" pattern in the filenames (e.g., `train-00000-of-00059.parquet`)

## Average File Size
Calculating from the visible files (00000-00049, which is 50 files shown):

Sum of file sizes: 415 + 417 + 427 + 450 + 439 + 432 + 451 + 420 + 408 + 460 + 435 + 434 + 415 + 402 + 426 + 408 + 390 + 392 + 365 + 391 + 364 + 388 + 356 + 386 + 409 + 374 + 399 + 377 + 398 + 384 + 384 + 408 + 406 + 381 + 432 + 436 + 459 + 448 + 450 + 432 + 474 + 453 + 451 + 477 + 468 + 464 + 447 + 454 + 434 + 472 = **20,569 MB**

**Average: ~411.4 MB per shard**

(Note: Files 00050-00058 are not shown in the preview, but based on the pattern, the average should remain similar)


Now I have all 22 languages. Let me compute the full picture.


Bash

python3 << 'EOF'
# All 22 languages: shards, avg_size_MB
data = {
    # Your 11 languages
    "Hindi":      (64,  590),
    "Telugu":     (56,  530),
    "Malayalam":  (88,  460),
    "Tamil":      (78,  620),
    "Kannada":    (73,  456),
    "Marathi":    (75,  500),
    "Punjabi":    (86,  415),
    "Assamese":   (125, 385),
    "Odia":       (89,  450),
    "Bengali":    (96,  400),
    "Gujarati":   (32,  425),
    # Other 11 languages
    "Bodo":       (56,  506),
    "Dogri":      (56,  390),
    "Kashmiri":   (44,  486),
    "Konkani":    (41,  384),
    "Maithili":   (59,  471),
    "Manipuri":   (36,  435),
    "Nepali":     (50,  520),
    "Sanskrit":   (50,  492),
    "Santali":    (44,  502),
    "Sindhi":     (59,  411),
    "Urdu":       (58,  545),
}

TOTAL_HOURS = 23700
TRANSCRIBED_HOURS = 11200
UNTRANSCRIBED_HOURS = TOTAL_HOURS - TRANSCRIBED_HOURS

# Compute estimated GB per language
for lang in data:
    shards, avg_mb = data[lang]
    data[lang] = (shards, avg_mb, shards * avg_mb / 1024)  # add GB

total_gb = sum(v[2] for v in data.values())

your_langs = ["Hindi", "Telugu", "Malayalam", "Tamil", "Kannada", "Marathi",
              "Punjabi", "Assamese", "Odia", "Bengali", "Gujarati"]

print("=" * 90)
print(f"{'Language':<12} {'Shards':>7} {'Avg MB':>7} {'Est GB':>8} {'% of Total':>10} {'Est Total h':>12} {'Est Trans h':>12} {'Est Untrans h':>14}")
print("=" * 90)

your_total_gb = 0
your_total_h = 0
your_trans_h = 0
your_untrans_h = 0

for lang in your_langs:
    shards, avg_mb, gb = data[lang]
    pct = gb / total_gb * 100
    est_hours = TOTAL_HOURS * (gb / total_gb)
    est_trans = TRANSCRIBED_HOURS * (gb / total_gb)
    est_untrans = UNTRANSCRIBED_HOURS * (gb / total_gb)
    your_total_gb += gb
    your_total_h += est_hours
    your_trans_h += est_trans
    your_untrans_h += est_untrans
    print(f"{lang:<12} {shards:>7} {avg_mb:>7} {gb:>8.1f} {pct:>9.1f}% {est_hours:>11.0f} {est_trans:>11.0f} {est_untrans:>13.0f}")

print("-" * 90)
print(f"{'YOUR 11':<12} {'':>7} {'':>7} {your_total_gb:>8.1f} {your_total_gb/total_gb*100:>9.1f}% {your_total_h:>11.0f} {your_trans_h:>11.0f} {your_untrans_h:>13.0f}")
print()

other_total_gb = 0
for lang in data:
    if lang not in your_langs:
        other_total_gb += data[lang][2]

print(f"{'OTHER 11':<12} {'':>7} {'':>7} {other_total_gb:>8.1f} {other_total_gb/total_gb*100:>9.1f}%")
print(f"{'ALL 22':<12} {'':>7} {'':>7} {total_gb:>8.1f} {'100.0':>9}%")
print()

print("=" * 90)
print("UNTRANSCRIBED DATA DISTRIBUTION (your 11 languages)")
print("=" * 90)
print(f"{'Language':<12} {'Est Untrans h':>14} {'% of Your Untrans':>18}")
print("-" * 50)
for lang in sorted(your_langs, key=lambda l: UNTRANSCRIBED_HOURS * (data[l][2] / total_gb), reverse=True):
    gb = data[lang][2]
    est_untrans = UNTRANSCRIBED_HOURS * (gb / total_gb)
    pct_of_yours = est_untrans / your_untrans_h * 100
    print(f"{lang:<12} {est_untrans:>13.0f} {pct_of_yours:>17.1f}%")
print("-" * 50)
print(f"{'TOTAL':<12} {your_untrans_h:>13.0f} {'100.0':>17}%")

EOF

Compute per-language hour estimates from parquet sizes


==========================================================================================
Language      Shards  Avg MB   Est GB % of Total  Est Total h  Est Trans h  Est Untrans h
==========================================================================================
Hindi             64     590     36.9       5.7%        1351         639           713
Telugu            56     530     29.0       4.5%        1062         502           560
Malayalam         88     460     39.5       6.1%        1449         685           764
Tamil             78     620     47.2       7.3%        1731         818           913
Kannada           73     456     32.5       5.0%        1191         563           628
Marathi           75     500     36.6       5.7%        1342         634           708
Punjabi           86     415     34.9       5.4%        1277         604           674
Assamese         125     385     47.0       7.3%        1722         814           908
Odia              89     450     39.1       6.0%        1433         677           756
Bengali           96     400     37.5       5.8%        1374         649           725
Gujarati          32     425     13.3       2.1%         487         230           257
------------------------------------------------------------------------------------------
YOUR 11                         393.5      60.8%       14420        6815          7606

OTHER 11                        253.2      39.2%
ALL 22                          646.7     100.0%

==========================================================================================
UNTRANSCRIBED DATA DISTRIBUTION (your 11 languages)
==========================================================================================
Language      Est Untrans h  % of Your Untrans
--------------------------------------------------
Tamil                  913              12.0%
Assamese               908              11.9%
Malayalam              764              10.0%
Odia                   756               9.9%
Bengali                725               9.5%
Hindi                  713               9.4%
Marathi                708               9.3%
Punjabi                674               8.9%
Kannada                628               8.3%
Telugu                 560               7.4%
Gujarati               257               3.4%
--------------------------------------------------
TOTAL                 7606             100.0%


Here's the complete picture using **repo-stated 11.2K transcribed / 23.7K total**, distributed proportionally by parquet size:

## Estimated Distribution Across Your 11 Languages

| Language | Est. Total (h) | Est. Transcribed (h) | Est. Untranscribed (h) | % of Untrans Pool |
|----------|:-:|:-:|:-:|:-:|
| **Tamil** | 1,731 | 818 | **913** | 12.0% |
| **Assamese** | 1,722 | 814 | **908** | 11.9% |
| **Malayalam** | 1,449 | 685 | **764** | 10.0% |
| **Odia** | 1,433 | 677 | **756** | 9.9% |
| **Bengali** | 1,374 | 649 | **725** | 9.5% |
| **Hindi** | 1,351 | 639 | **713** | 9.4% |
| **Marathi** | 1,342 | 634 | **708** | 9.3% |
| **Punjabi** | 1,277 | 604 | **674** | 8.9% |
| **Kannada** | 1,191 | 563 | **628** | 8.3% |
| **Telugu** | 1,062 | 502 | **560** | 7.4% |
| **Gujarati** | 487 | 230 | **257** | 3.4% |
| **TOTAL** | **14,420** | **6,815** | **7,606** | **100%** |

### Key takeaways

1. **Your 11 languages hold ~61% of the full dataset** (~14.4K of 23.7K hours), which makes sense for the major Indic languages.

2. **~7,600 hours of untranscribed data** across your 11 languages — that's a massive pool for pseudo-labeling with your ASR models.

3. **Distribution is fairly uniform** (7–12% each) except Gujarati which is a clear outlier at 3.4%. The top 4 untranscribed pools are **Tamil, Assamese, Malayalam, Odia**.

4. **Caveat**: This assumes transcription effort was distributed proportionally to total data per language. In reality, some languages may have been prioritized — but without accessing the actual parquet rows (gated), proportional is our best estimate. The actual per-language transcription rate could vary from ~40% to ~55%.


let's not trust the numbers from paper to be GT as transcribed content. as they said in the repo, 11k is transcribed.