python3 - <<'PY'
from pathlib import Path
import psycopg

env={}
for line in Path('.env').read_text().splitlines():
    line=line.strip()
    if not line or line.startswith('#') or '=' not in line:
        continue
    if line.startswith('export '):
        line=line[7:]
    k,v=line.split('=',1)
    env[k.strip()] = v.strip().strip('"').strip("'")

with psycopg.connect(env['DATABASE_URL']) as conn:
    with conn.cursor() as cur:
        cur.execute(
            """
            select claimed_by, shard_id::text, lease_expires_at, count(*)
            from video_tasks
            where language='gujarati' and status='processing'
            group by claimed_by, shard_id, lease_expires_at
            """
        )
        rows = cur.fetchall()
        print('gujarati processing groups:')
        for row in rows:
            print(' ', row)
PY


Output:
Exit code: 0
Wall time: 0.7 seconds
Output:
gujarati processing groups:
  ('maya3-worker-24', 'a547d366-2056-51bc-aa4e-957e63db48c7', datetime.datetime(2026, 1, 21, 4, 22, 5, 686940, tzinfo=datetime.timezone.utc), 100)


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Identifying stale shard claim**

A worker reboot at 3:26 likely caused a shard to be stuck despite no updated shard state, requiring releasing the stale claim once the lease expiry time (4:22) is passed by checking the current server time.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
2026-01-21 07:02:36 UTC


Summary: auto


**Analyzing stale claim release logic**

I'm checking release_stale_claims.py to understand how it handles stale claim release and how it updates related statuses before implementing specific cleanup for video_tasks, shard_runs, and language_queue.


sed -n '1,220p' /home/ubuntu/maya3_prepare/workers/release_stale_claims.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
#!/usr/bin/env python3
"""Release stale claims from crashed workers.

This script should be run periodically (e.g., via cron) to:
1. Release video_tasks with expired leases back to 'pending'
2. Mark associated shard_runs as 'stale'
3. Log recovery actions for debugging

Usage:
    python release_stale_claims.py [--dry-run] [--env-file .env]
"""

from __future__ import annotations

import argparse
import logging
import sys
from datetime import datetime
from pathlib import Path

import psycopg

if __package__ in (None, ""):
    sys.path.insert(0, str(Path(__file__).resolve().parents[1]))

from workers.config import EnvConfig

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(message)s",
    datefmt="%Y-%m-%d %H:%M:%S",
)
logger = logging.getLogger("stale-recovery")

def release_stale_video_tasks(conn, dry_run: bool = False) -> int:
    """Release video_tasks with expired leases back to pending."""
    with conn.cursor() as cur:
        if dry_run:
            cur.execute(
                """
                select video_id, claimed_by, lease_expires_at, status
                from video_tasks
                where status in ('claimed', 'processing')
                  and lease_expires_at < now()
                """
            )
            stale = cur.fetchall()
            for row in stale:
                logger.info(
                    f"[DRY-RUN] Would release video {row[0]} "
                    f"(claimed_by={row[1]}, expired={row[2]}, status={row[3]})"
                )
            return len(stale)
        else:
            cur.execute(
                """
                update video_tasks
                set status = 'pending',
                    claimed_by = null,
                    lease_expires_at = null,
                    updated_at = now()
                where status in ('claimed', 'processing')
                  and lease_expires_at < now()
                returning video_id, claimed_by
                """
            )
            released = cur.fetchall()
            for video_id, claimed_by in released:
                logger.info(f"Released video {video_id} (was claimed by {claimed_by})")
            return len(released)

def mark_stale_shards(conn, dry_run: bool = False) -> int:
    """Mark shard_runs with expired leases as stale."""
    with conn.cursor() as cur:
        if dry_run:
            cur.execute(
                """
                select shard_id, claimed_by, lease_expires_at, status
                from shard_runs
                where status in ('claimed', 'processing', 'uploading')
                  and lease_expires_at < now()
                """
            )
            stale = cur.fetchall()
            for row in stale:
                logger.info(
                    f"[DRY-RUN] Would mark shard {row[0]} as stale "
                    f"(claimed_by={row[1]}, expired={row[2]}, status={row[3]})"
                )
            return len(stale)
        else:
            cur.execute(
                """
                update shard_runs
                set status = 'stale',
                    updated_at = now()
                where status in ('claimed', 'processing', 'uploading')
                  and lease_expires_at < now()
                returning shard_id, claimed_by
                """
            )
            stale = cur.fetchall()
            for shard_id, claimed_by in stale:
                logger.info(f"Marked shard {shard_id} as stale (was claimed by {claimed_by})")
            return len(stale)

def release_stale_languages(conn, dry_run: bool = False) -> int:
    """Release language_queue entries with expired leases back to pending."""
    with conn.cursor() as cur:
        if dry_run:
            cur.execute(
                """
                select language, claimed_by, lease_expires_at, status
                from language_queue
                where status = 'processing'
                  and lease_expires_at < now()
                """
            )
            stale = cur.fetchall()
            for row in stale:
                logger.info(
                    f"[DRY-RUN] Would release language {row[0]} "
                    f"(claimed_by={row[1]}, expired={row[2]}, status={row[3]})"
                )
            return len(stale)
        else:
            cur.execute(
                """
                update language_queue
                set status = 'pending',
                    claimed_by = null,
                    lease_expires_at = null,
                    updated_at = now()
                where status = 'processing'
                  and lease_expires_at < now()
                returning language, claimed_by
                """
            )
            released = cur.fetchall()
            for language, claimed_by in released:
                logger.info(
                    f"Released language {language} (was claimed by {claimed_by})"
                )
            if released:
                cur.execute(
                    """
                    update language_state
                    set active_language = null,
                        updated_at = now()
                    where id = 1
                      and active_language = any(%s)
                    """,
                    ([row[0] for row in released],),
                )
            return len(released)

def log_recovery_event(conn, video_ids: int, shard_ids: int, language_ids: int) -> None:
    """Log recovery event for audit trail."""
    with conn.cursor() as cur:
        cur.execute(
            """
            insert into video_task_events (event, details)
            values ('stale_recovery', %s)
            """,
            (
                f'{{"videos_released": {video_ids}, '
                f'"shards_marked_stale": {shard_ids}, '
                f'"languages_released": {language_ids}}}',
            ),
        )

def get_stats(conn) -> dict:
    """Get current task statistics."""
    with conn.cursor() as cur:
        cur.execute(
            """
            select status, count(*)
            from video_tasks
            group by status
            """
        )
        video_stats = dict(cur.fetchall())

        cur.execute(
            """
            select status, count(*)
            from shard_runs
            group by status
            """
        )
        shard_stats = dict(cur.fetchall())

        cur.execute(
            """
            select status, count(*)
            from language_queue
            group by status
            """
        )
        language_stats = dict(cur.fetchall())

        cur.execute(
            """
            select count(distinct worker_id)
            from worker_heartbeats_v2
            where last_heartbeat > now() - interval '5 minutes'
            """
        )
        active_workers = cur.fetchone()[0]

    return {
        "video_tasks": video_stats,
        "shard_runs": shard_stats,
        "language_queue": language_stats,
        "active_workers": active_workers,


sed -n '220,420p' /home/ubuntu/maya3_prepare/workers/release_stale_claims.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
        "active_workers": active_workers,
    }

def main():
    parser = argparse.ArgumentParser(description="Release stale claims from crashed workers")
    parser.add_argument("--dry-run", action="store_true", help="Show what would be done")
    parser.add_argument("--env-file", type=Path, default=Path(".env"))
    parser.add_argument("--stats-only", action="store_true", help="Only show stats, don't recover")
    args = parser.parse_args()

    # Load config
    if args.env_file.exists():
        env_config = EnvConfig.from_file(args.env_file)
    else:
        env_config = EnvConfig.from_env_vars()

    if not env_config.database_url:
        logger.error("DATABASE_URL not configured")
        raise SystemExit(1)

    with psycopg.connect(env_config.database_url) as conn:
        # Show current stats
        stats = get_stats(conn)
        logger.info("=== Current Statistics ===")
        logger.info(f"Video tasks: {stats['video_tasks']}")
        logger.info(f"Shard runs: {stats['shard_runs']}")
        logger.info(f"Active workers (last 5 min): {stats['active_workers']}")

        if args.stats_only:
            return

        logger.info("")
        logger.info("=== Releasing Stale Claims ===")

        # Release stale claims
        videos_released = release_stale_video_tasks(conn, args.dry_run)
        shards_marked = mark_stale_shards(conn, args.dry_run)
        languages_released = release_stale_languages(conn, args.dry_run)

        if not args.dry_run:
            if videos_released > 0 or shards_marked > 0 or languages_released > 0:
                log_recovery_event(conn, videos_released, shards_marked, languages_released)
            conn.commit()

        logger.info("")
        logger.info(f"Released {videos_released} video tasks")
        logger.info(f"Marked {shards_marked} shards as stale")
        logger.info(f"Released {languages_released} languages")

        # Show updated stats
        if not args.dry_run and (videos_released > 0 or shards_marked > 0):
            stats = get_stats(conn)
            logger.info("")
            logger.info("=== Updated Statistics ===")
            logger.info(f"Video tasks: {stats['video_tasks']}")
            logger.info(f"Shard runs: {stats['shard_runs']}")

if __name__ == "__main__":
    main()


**Running script to release stale claims**

I'm executing the script to free stuck video tasks by releasing expired leases, which should unblock Gujarati and allow processing of the next language as intended.


python3 /home/ubuntu/maya3_prepare/workers/release_stale_claims.py --env-file /home/ubuntu/maya3_prepare/.env


Output:
Exit code: 124
Wall time: 10 seconds
Output:
command timed out after 10008 milliseconds
2026-01-21 07:02:53 [INFO] === Current Statistics ===
2026-01-21 07:02:53 [INFO] Video tasks: {'completed': 447872, 'pending': 59415, 'processing': 100}
2026-01-21 07:02:53 [INFO] Shard runs: {'processing': 1, 'completed': 4485}
2026-01-21 07:02:53 [INFO] Active workers (last 5 min): 49
2026-01-21 07:02:53 [INFO]
2026-01-21 07:02:53 [INFO] === Releasing Stale Claims ===
2026-01-21 07:02:53 [INFO] Released video VF3csYX3_tM (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VePb4b96i-k (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vcjreVzhdmw (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VDZuzmr5FD8 (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video Vd48b3hiCEw (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video Ve6-qXxY62I (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vCPI9e0flrA (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video veYlYwbLSmY (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vDt3u9gc5dI (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VdsiqjnFoVw (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VcgkR45f20c (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vddoRH9K3nQ (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VEmDK8cmvbQ (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VEAMXDqf7DA (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vDyXfBQ7rAA (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vdnCSdlXZac (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VCzOp88MgzY (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vdC0SQJRieM (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VCscGS2KOZM (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vdvW5LodkPs (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vepAuPYicIQ (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video ve83lff3KDU (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vcluqU-J2xc (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vesjhcKzGyg (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vdJbzLY0lMg (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vdq2NVd0mIU (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vEN6uZunOEs (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VeJIjDh7TSA (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vczNBt_rQcY (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vcVXGidmDv4 (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vF_MGga2PI0 (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vCToJwxW74M (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vEGs2yb1fTc (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vEAksHqj6I0 (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vDMOebjDUYw (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video Ve10hJb9Z9U (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VDnNZydJlOA (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vD-fJJ1LDYo (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vEb2iilBA0A (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vDRLxmEuObk (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vdgXIqUM7aY (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VDJ1IbR8ljM (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vE1qzTYglVQ (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video Vdc0ZXLU2Eo (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VfbHQ_RwKfw (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vET8FhhmE0Q (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vDx7CDKfM-Q (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vEsakBX6Pe8 (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VCZaSqKE6uo (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VEfMm1ix7Bk (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vE-Ml2vpQHc (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vExB9NyjcKk (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VEOAvErFQCg (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video Vf6iuO5nVqs (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vd9GhRAX7JE (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VDNjZpv96OI (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vcNnD0IE6GM (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video Vdn-vvjkzhU (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video veGZG0q8b-I (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vDkCmSndyfE (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VCxEcp9VGi8 (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vEbI1QeXkaY (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VD8llo10aTE (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vd9iY4BA_WI (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VdfShWwBRko (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vD7IFTK8x8E (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VCfanOBeMkA (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vcpJLUMhRnQ (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VdQ8FAVkf6w (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VCQENKY05LY (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video Vctuj_WTV3Q (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vD1HrQvmhRk (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vcQu70yVAdo (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VDwF9o76FE8 (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vdefkkif0TU (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vE6MdsY2-3A (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VDHKfWccCwU (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VevIw4axTuc (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vcfiJSyv-K4 (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VdVzwha746M (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video Ve5jk9DWtd4 (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vE5wSbleS4A (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video ve7SXfYJ8MQ (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video veAzrRFxNoI (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VEc4Z-7FBZw (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VEgWNIsaeeE (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VEtUF6YIJxI (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vF48ai8_T78 (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vCtIN7DAzOo (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VcXq4LzPk9Y (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vd_2SQ5J3p4 (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vD2QmBdW960 (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video Vd8Uu9SS89o (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vdc7YCPwFEg (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VdGFWSINIfc (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video vdjvFouEJac (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VDKo0apnHGU (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VDpPstBXbjI (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video Vdrk7R8ixrQ (was claimed by None)
2026-01-21 07:02:53 [INFO] Released video VdrLd6gUDtM (was claimed by None)
2026-01-21 07:02:53 [INFO] Marked shard a547d366-2056-51bc-aa4e-957e63db48c7 as stale (was claimed by maya3-worker-24)
2026-01-21 07:02:53 [INFO] Released language gujarati (was claimed by None)


**Verifying commit completion after timeout**

Noting the command likely timed out before finishing the commit step despite releasing resources; next I'll check commit status by verifying changes applied through count checks.


python3 - <<'PY'
from pathlib import Path
import psycopg

env={}
for line in Path('.env').read_text().splitlines():
    line=line.strip()
    if not line or line.startswith('#') or '=' not in line:
        continue
    if line.startswith('export '):
        line=line[7:]
    k,v=line.split('=',1)
    env[k.strip()] = v.strip().strip('"').strip("'")

with psycopg.connect(env['DATABASE_URL']) as conn:
    with conn.cursor() as cur:
        cur.execute("select status, count(*) from video_tasks group by status order by status")
        print('video_tasks statuses:')
        for row in cur.fetchall():
            print(' ', row)
        cur.execute("select status, count(*) from shard_runs group by status order by status")
        print('shard_runs statuses:')
        for row in cur.fetchall():
            print(' ', row)
        cur.execute("select language, status, claimed_by, lease_expires_at from language_queue where language='gujarati'")
        print('language_queue gujarati:', cur.fetchone())
        cur.execute("select active_language from language_state where id=1")
        print('language_state:', cur.fetchone())
PY


Output:
Exit code: 0
Wall time: 1.1 seconds
Output:
video_tasks statuses:
  ('completed', 447872)
  ('pending', 59415)
  ('processing', 100)
shard_runs statuses:
  ('completed', 4485)
  ('processing', 1)
language_queue gujarati: ('gujarati', 'processing', 'maya3-worker-44', datetime.datetime(2026, 1, 21, 4, 58, 55, 194624, tzinfo=datetime.timezone.utc))
language_state: ('gujarati',)


**Analyzing stale task release approach**

The command likely timed out before committing, leaving stale video_tasks still marked processing with old leases. To recover, a direct SQL update can release these stale tasks by resetting status and lease fields; clearing shard_id is optional since it’s derived later and not reset on claim. Next step is running targeted SQL to release stuck tasks without flooding logs.


**Finalizing stale claim cleanup plan**

Decided to clear shard_id is optional but leaving it is consistent with existing behavior, so the quick fix updates stale video_tasks, language_queue, and shard_runs directly by resetting status and leases. This will safely release stuck tasks for Gujarati without extra event logging. The next step is running these targeted SQL updates carefully via Python.


python3 - <<'PY'
from pathlib import Path
import psycopg

env={}
for line in Path('.env').read_text().splitlines():
    line=line.strip()
    if not line or line.startswith('#') or '=' not in line:
        continue
    if line.startswith('export '):
        line=line[7:]
    k,v=line.split('=',1)
    env[k.strip()] = v.strip().strip('"').strip("'")

with psycopg.connect(env['DATABASE_URL']) as conn:
    with conn.cursor() as cur:
        # Release stale gujarati processing tasks
        cur.execute(
            """
            update video_tasks
            set status = 'pending',
                claimed_by = null,
                lease_expires_at = null,
                shard_id = null,
                last_error = null,
                updated_at = now()
            where language = 'gujarati'
              and status = 'processing'
              and lease_expires_at < now()
            """
        )
        released_tasks = cur.rowcount

        # Mark shard stale if still processing and lease expired
        cur.execute(
            """
            update shard_runs
            set status = 'stale',
                updated_at = now()
            where language = 'gujarati'
              and status in ('claimed','processing','uploading')
              and lease_expires_at < now()
            """
        )
        stale_shards = cur.rowcount

        # Release language queue if stale
        cur.execute(
            """
            update language_queue
            set status = 'pending',
                claimed_by = null,
                lease_expires_at = null,
                updated_at = now()
            where language = 'gujarati'
              and status = 'processing'
              and lease_expires_at < now()
            """
        )
        released_lang = cur.rowcount

        if released_lang:
            cur.execute(
                """
                update language_state
                set active_language = null,
                    updated_at = now()
                where id = 1 and active_language = 'gujarati'
                """
            )

        conn.commit()

print(f"released_tasks={released_tasks} stale_shards={stale_shards} released_lang={released_lang}")
PY


Output:
Exit code: 124
Wall time: 10 seconds
Output:
command timed out after 10007 milliseconds


**Planning to increase command timeout**

The command timed out after 10 seconds, likely due to a default limit, so I'll rerun it with a higher timeout and check if changes were applied or committed by inspecting the status.


python3 - <<'PY'
from pathlib import Path
import psycopg

env={}
for line in Path('.env').read_text().splitlines():
    line=line.strip()
    if not line or line.startswith('#') or '=' not in line:
        continue
    if line.startswith('export '):
        line=line[7:]
    k,v=line.split('=',1)
    env[k.strip()] = v.strip().strip('"').strip("'")

with psycopg.connect(env['DATABASE_URL']) as conn:
    with conn.cursor() as cur:
        cur.execute("select status, count(*) from video_tasks where language='gujarati' group by status")
        print('gujarati tasks by status:', cur.fetchall())
        cur.execute("select shard_id::text, status, lease_expires_at from shard_runs where language='gujarati' and status!='completed'")
        print('gujarati shard_runs non-completed:', cur.fetchall())
        cur.execute("select language, status, lease_expires_at from language_queue where language='gujarati'")
        print('language_queue gujarati:', cur.fetchone())
        cur.execute("select active_language from language_state where id=1")
        print('language_state:', cur.fetchone())
PY


Output:
Exit code: 0
Wall time: 0.9 seconds
Output:
gujarati tasks by status: [('completed', 40850), ('processing', 100)]
gujarati shard_runs non-completed: [('a547d366-2056-51bc-aa4e-957e63db48c7', 'processing', datetime.datetime(2026, 1, 21, 4, 22, 5, 686940, tzinfo=datetime.timezone.utc))]
language_queue gujarati: ('gujarati', 'processing', datetime.datetime(2026, 1, 21, 4, 58, 55, 194624, tzinfo=datetime.timezone.utc))
language_state: ('gujarati',)


**Extending timeout for patch apply**

I'm increasing the command timeout to 60 seconds to prevent premature termination during patch application.


python3 - <<'PY'
from pathlib import Path
import psycopg

env={}
for line in Path('.env').read_text().splitlines():
    line=line.strip()
    if not line or line.startswith('#') or '=' not in line:
        continue
    if line.startswith('export '):
        line=line[7:]
    k,v=line.split('=',1)
    env[k.strip()] = v.strip().strip('"').strip("'")

with psycopg.connect(env['DATABASE_URL']) as conn:
    with conn.cursor() as cur:
        cur.execute(
            """
            update video_tasks
            set status = 'pending',
                claimed_by = null,
                lease_expires_at = null,
                shard_id = null,
                last_error = null,
                updated_at = now()
            where language = 'gujarati'
              and status = 'processing'
              and lease_expires_at < now()
            """
        )
        released_tasks = cur.rowcount

        cur.execute(
            """
            update shard_runs
            set status = 'stale',
                updated_at = now()
            where language = 'gujarati'
              and status in ('claimed','processing','uploading')
              and lease_expires_at < now()
            """
        )
        stale_shards = cur.rowcount

        cur.execute(
            """
            update language_queue
            set status = 'pending',
                claimed_by = null,
                lease_expires_at = null,
                updated_at = now()
            where language = 'gujarati'
              and status = 'processing'
              and lease_expires_at < now()
            """
        )
        released_lang = cur.rowcount

        if released_lang:
            cur.execute(
                """
                update language_state
                set active_language = null,
                    updated_at = now()
                where id = 1 and active_language = 'gujarati'
                """
            )

        conn.commit()

print(f"released_tasks={released_tasks} stale_shards={stale_shards} released_lang={released_lang}")
PY


Output:
Exit code: 0
Wall time: 12.6 seconds
Output:
released_tasks=100 stale_shards=1 released_lang=1


**Rechecking status counts after changes**


python3 - <<'PY'
from pathlib import Path
import psycopg

env={}
for line in Path('.env').read_text().splitlines():
    line=line.strip()
    if not line or line.startswith('#') or '=' not in line:
        continue
    if line.startswith('export '):
        line=line[7:]
    k,v=line.split('=',1)
    env[k.strip()] = v.strip().strip('"').strip("'")

with psycopg.connect(env['DATABASE_URL']) as conn:
    with conn.cursor() as cur:
        cur.execute("select status, count(*) from video_tasks where language='gujarati' group by status")
        print('gujarati tasks by status:', cur.fetchall())
        cur.execute("select shard_id::text, status from shard_runs where language='gujarati' and status!='completed'")
        print('gujarati shard_runs non-completed:', cur.fetchall())
        cur.execute("select language, status, claimed_by from language_queue where language='gujarati'")
        print('language_queue gujarati:', cur.fetchone())
        cur.execute("select active_language from language_state where id=1")
        print('language_state:', cur.fetchone())
PY


Output:
Exit code: 0
Wall time: 1 seconds
Output:
gujarati tasks by status: [('claimed', 100), ('completed', 40850)]
gujarati shard_runs non-completed: [('a547d366-2056-51bc-aa4e-957e63db48c7', 'stale')]
language_queue gujarati: ('gujarati', 'processing', 'maya3-worker-11')
language_state: ('gujarati',)


Summary: auto


**Monitoring task distribution**

Gujarati tasks are currently claimed and processing by worker 11 with an updated lease, indicating active progress; next, I'll verify the overall worker status distribution to confirm balanced workload.


python3 - <<'PY'
from pathlib import Path
import psycopg

env={}
for line in Path('.env').read_text().splitlines():
    line=line.strip()
    if not line or line.startswith('#') or '=' not in line:
        continue
    if line.startswith('export '):
        line=line[7:]
    k,v=line.split('=',1)
    env[k.strip()] = v.strip().strip('"').strip("'")

with psycopg.connect(env['DATABASE_URL']) as conn:
    with conn.cursor() as cur:
        cur.execute("select status, count(*) from worker_heartbeats_v2 group by status order by status")
        print('worker statuses:')
        for row in cur.fetchall():
            print(' ', row)
PY


Output:
Exit code: 0
Wall time: 0.8 seconds
Output:
worker statuses:
  ('claiming', 48)
  ('downloading', 1)


**Confirming stalled shard cleared**

The large backlog is due to being on the final shard for one language, with only one active worker handling it. Releasing stale tasks freed that shard, allowing progress. Next, I'll suggest monitoring and possibly adjusting claim logic to avoid single-worker stalls on last shards.


**Summarizing stale release outcome**

The 100 stale processing tasks for the gujarati shard were successfully released and reclaimed by a new worker, resolving that language queue block. I'll recommend running the stale release script periodically and monitoring progress as gujarati finishes and the queue moves to other languages.


Found the issue: the entire fleet was blocked behind the **last Gujarati shard**. Only 100 Gujarati videos remained, but the worker that owned them (worker‑24) died, so language queue stayed on Gujarati and everything else stayed pending.

I fixed it by force‑releasing that stale shard/tasks and letting another worker claim them.

What I did:
- Released the 100 stale `gujarati` tasks back to `pending`.
- Marked the stuck shard `a547d366-2056-51bc-aa4e-957e63db48c7` as `stale`.
- Cleared the Gujarati language lock; it immediately got re‑claimed and is processing again.

Current state now:
- Gujarati tasks: `100 claimed`, `40850 completed`
- Gujarati shard runs: `1 stale` (the old stuck one)
- Language queue: `gujarati` is active again (claimed by `maya3-worker-11`)

So the ~59k “pending last hour” is explained by the language gate: the queue was stuck on Gujarati until that final shard is done. Once this 100 finishes, it will advance to Bengali/Odia/Marathi/Assamese and pending will drop again.

If you want to prevent this from happening again, we should run `workers/release_stale_claims.py` on a schedule (cron every 5–10 minutes).


but it only happened because the worker that held the final 100 didnt respond with any update. it might not happen that frequently right. so i dont think we need a cron job. wdyt. anyways, i'm assuming the next videos will be processed followingly. right ? alright cool. imma hit some sleep. someone on my behalf might tell you to check progress and tell you to shut down all instances only after verifying all are done and. what you should do is check all progress, and total objects in respective buckets in R2 and confirming weather all are done. then, maybe validating all videos are now done and all languages are done, you can safely shutdown all instances after validating. dont forget to double check before shutting down. not right now, someone will tell you in here in chat who is monitoring the dashboard's status and will tell you to check, validate and delete instances. act then. now just ack.


Got it. Agreed it’s a rare edge case; no cron unless you decide you want it.

Yes, once that final Gujarati 100 finishes, the queue will move to the next pending languages automatically.

I’ll wait for the next instruction to verify completion and shut down instances, and I’ll double‑check Supabase + R2 before doing anything. Sleep well.


check status, validate and act now.


python3 /home/ubuntu/maya3_prepare/workers/vast_monitor.py --count 49 --timeout-sec 120 --poll-sec 30


Output:
Exit code: 1
Wall time: 123.2 seconds
Output:
Timeout waiting for all workers to start
checked=49 ok=False missing=1 stale=0 inactive=0
missing: maya3-worker-46
checked=49 ok=False missing=1 stale=0 inactive=0
missing: maya3-worker-46
checked=49 ok=False missing=1 stale=0 inactive=0
missing: maya3-worker-46
checked=49 ok=False missing=1 stale=0 inactive=0
missing: maya3-worker-46
checked=49 ok=False missing=1 stale=0 inactive=0
missing: maya3-worker-46


python3 - <<'PY'
from pathlib import Path
import psycopg

env = {}
for line in Path('.env').read_text().splitlines():
    line=line.strip()
    if not line or line.startswith('#') or '=' not in line:
        continue
    if line.startswith('export '):
        line=line[7:]
    k,v=line.split('=',1)
    env[k.strip()] = v.strip().strip('"').strip("'")

with psycopg.connect(env['DATABASE_URL']) as conn:
    with conn.cursor() as cur:
        cur.execute("select status, count(*) from video_tasks group by status order by status")
        print('video_tasks statuses:')
        for row in cur.fetchall():
            print(' ', row)
        cur.execute("select language, status, count(*) from video_tasks group by language, status order by language, status")
        rows = cur.fetchall()
        print('language status counts (non-completed):')
        for language, status, count in rows:
            if status != 'completed':
                print(' ', language, status, count)
        cur.execute("select status, count(*) from shard_runs group by status order by status")
        print('shard_runs statuses:')
        for row in cur.fetchall():
            print(' ', row)
        cur.execute("select language, status, claimed_by, lease_expires_at from language_queue order by priority desc, language")
        print('language_queue:')
        for row in cur.fetchall():
            print(' ', row)
        cur.execute("select active_language from language_state where id=1")
        print('language_state:', cur.fetchone())
PY


Output:
Exit code: 0
Wall time: 1.3 seconds
Output:
video_tasks statuses:
  ('completed', 505987)
  ('processing', 700)
  ('uploading', 700)
language status counts (non-completed):
  assamese processing 700
  assamese uploading 700
shard_runs statuses:
  ('completed', 5070)
  ('processing', 7)
  ('uploading', 7)
language_queue:
  ('telugu', 'completed', 'maya3-worker-1', datetime.datetime(2026, 1, 20, 11, 26, 3, 633977, tzinfo=datetime.timezone.utc))
  ('malayalam', 'completed', 'maya3-worker-47', datetime.datetime(2026, 1, 20, 13, 58, 46, 843396, tzinfo=datetime.timezone.utc))
  ('english', 'completed', 'maya3-worker-30', datetime.datetime(2026, 1, 20, 16, 47, 49, 951947, tzinfo=datetime.timezone.utc))
  ('hindi', 'completed', 'maya3-worker-31', datetime.datetime(2026, 1, 20, 19, 43, 27, 349027, tzinfo=datetime.timezone.utc))
  ('punjabi', 'completed', 'maya3-worker-11', datetime.datetime(2026, 1, 20, 22, 55, 18, 570669, tzinfo=datetime.timezone.utc))
  ('tamil', 'completed', 'maya3-worker-9', datetime.datetime(2026, 1, 21, 0, 41, 35, 585486, tzinfo=datetime.timezone.utc))
  ('kannada', 'completed', 'maya3-worker-35', datetime.datetime(2026, 1, 21, 2, 42, 26, 94807, tzinfo=datetime.timezone.utc))
  ('gujarati', 'completed', 'maya3-worker-11', datetime.datetime(2026, 1, 21, 8, 13, 50, 395942, tzinfo=datetime.timezone.utc))
  ('bengali', 'completed', 'maya3-worker-23', datetime.datetime(2026, 1, 21, 9, 26, 24, 805218, tzinfo=datetime.timezone.utc))
  ('odia', 'completed', 'maya3-worker-26', datetime.datetime(2026, 1, 21, 10, 14, 39, 278875, tzinfo=datetime.timezone.utc))
  ('marathi', 'completed', 'maya3-worker-21', datetime.datetime(2026, 1, 21, 11, 15, 37, 957489, tzinfo=datetime.timezone.utc))
  ('assamese', 'processing', 'maya3-worker-35', datetime.datetime(2026, 1, 21, 11, 30, 48, 661695, tzinfo=datetime.timezone.utc))
language_state: ('assamese',)


python3 /home/ubuntu/maya3_prepare/workers/validate_r2_uploads.py --limit 3


Output:
Exit code: 0
Wall time: 2.8 seconds
Output:
shard=dbb02afa-11cc-5f07-85e0-ae6293a739ba language=assamese
  assamese_16/shard-dbb02afa-11cc-5f07-85e0-ae6293a739ba: objects=2 bytes=2127237911
  assamese_24/shard-dbb02afa-11cc-5f07-85e0-ae6293a739ba: objects=2 bytes=2827918679
  assamese_32/shard-dbb02afa-11cc-5f07-85e0-ae6293a739ba: objects=2 bytes=3089064101
  assamese_44/shard-dbb02afa-11cc-5f07-85e0-ae6293a739ba: objects=2 bytes=1014354065
  manifests/assamese_train_shard-dbb02afa-11cc-5f07-85e0-ae6293a739ba: objects=1 bytes=52850881
  manifests/assamese_val_shard-dbb02afa-11cc-5f07-85e0-ae6293a739ba: objects=1 bytes=1616795
  stats/assamese_stats_shard-dbb02afa-11cc-5f07-85e0-ae6293a739ba: objects=1 bytes=407
  total_objects=11 total_bytes=9113042839
shard=ce31ce9c-5a9b-5dd7-9660-900f52bba904 language=assamese
  assamese_16/shard-ce31ce9c-5a9b-5dd7-9660-900f52bba904: objects=2 bytes=1738449510
  assamese_24/shard-ce31ce9c-5a9b-5dd7-9660-900f52bba904: objects=2 bytes=2308866982
  assamese_32/shard-ce31ce9c-5a9b-5dd7-9660-900f52bba904: objects=2 bytes=2404903404
  assamese_44/shard-ce31ce9c-5a9b-5dd7-9660-900f52bba904: objects=2 bytes=733842757
  manifests/assamese_train_shard-ce31ce9c-5a9b-5dd7-9660-900f52bba904: objects=1 bytes=43896390
  manifests/assamese_val_shard-ce31ce9c-5a9b-5dd7-9660-900f52bba904: objects=1 bytes=1460889
  stats/assamese_stats_shard-ce31ce9c-5a9b-5dd7-9660-900f52bba904: objects=1 bytes=405
  total_objects=11 total_bytes=7231420337
shard=537111ce-1cf1-5c2e-b136-65194bbaa0b9 language=assamese
  assamese_16/shard-537111ce-1cf1-5c2e-b136-65194bbaa0b9: objects=2 bytes=2072559081
  assamese_24/shard-537111ce-1cf1-5c2e-b136-65194bbaa0b9: objects=2 bytes=2747313106
  assamese_32/shard-537111ce-1cf1-5c2e-b136-65194bbaa0b9: objects=2 bytes=3037762868
  assamese_44/shard-537111ce-1cf1-5c2e-b136-65194bbaa0b9: objects=2 bytes=924370393
  manifests/assamese_train_shard-537111ce-1cf1-5c2e-b136-65194bbaa0b9: objects=1 bytes=47943276
  manifests/assamese_val_shard-537111ce-1cf1-5c2e-b136-65194bbaa0b9: objects=1 bytes=127010
  stats/assamese_stats_shard-537111ce-1cf1-5c2e-b136-65194bbaa0b9: objects=1 bytes=406
  total_objects=11 total_bytes=8830076140


**Reporting processing status with shutdown conditions**

Only the Assamese language remains with about 1400 tasks split between processing and uploading; video tasks and other queues are fully completed. Worker 46 is intentionally missing, and the monitor script is reporting that correctly. Next, we should wait until all Assamese tasks finish before validating counts and shutting down instances. I’ll provide a status summary, note the ongoing processing, and suggest monitoring with repeated checks until done.


**Confirming not safe to shutdown yet**

The system shows only Assamese tasks remain—with about 700 processing and 700 uploading, no pending—and recent R2 validation on last shards is complete. All 49 workers report heartbeats, though the monitor flags missing worker-46 due to numbering gaps. Since processing isn’t finished, I haven’t shut down instances. I recommend continuing monitoring and can automate shutdown once all tasks complete upon your confirmation.


Checked and validated. Not safe to shut down yet.

Current status:
- `video_tasks`: `completed 505,987`, `processing 700`, `uploading 700`, `pending 0`.
- Only **assamese** remains in flight (all other languages completed). `language_queue` shows assamese `processing`, `language_state=assamese`.
- `shard_runs`: `completed 5,070`, `processing 7`, `uploading 7`.
- R2 spot‑check (latest 3 assamese shards): all **11 objects** per shard present (tar+jsonl ×4 SR + manifests + stats).

So work is still running; I did **not** shut anything down.
If you want me to auto‑monitor and shut down once `processing/uploading` hit 0 and all languages are `completed`, say the word and I’ll handle it.


can you check the instances done so far and can you start closing the instances that need no work


**Planning safe worker shutdown**

I'm identifying workers that are idle or claiming without assigned videos and not actively processing or uploading, to safely close those instances while keeping enough active for last shard tasks. Next, I'll query worker heartbeats and task status in the database to pinpoint which workers can be stopped without disrupting progress.


**Confirming criteria for destroying idle workers**

I'm defining idle workers as those with status claiming or idle, zero assigned videos, and no shard ID, with recent heartbeats. I'll map these worker IDs to Vast instance IDs via labels to safely destroy just those idle instances while preserving active ones processing the 14 shards. Next, I'll query the DB and Vast instance info to gather and act on this list.


python3 - <<'PY'
from pathlib import Path
import psycopg
from datetime import datetime, timezone

env={}
for line in Path('.env').read_text().splitlines():
    line=line.strip()
    if not line or line.startswith('#') or '=' not in line:
        continue
    if line.startswith('export '):
        line=line[7:]
    k,v=line.split('=',1)
    env[k.strip()] = v.strip().strip('"').strip("'")

active_statuses = {'downloading','processing','uploading'}
idle_statuses = {'claiming','idle'}

with psycopg.connect(env['DATABASE_URL']) as conn:
    with conn.cursor() as cur:
        cur.execute(
            """
            select worker_id, status, shard_id, videos_assigned, last_heartbeat
            from worker_heartbeats_v2
            order by worker_id
            """
        )
        rows = cur.fetchall()

now = datetime.now(timezone.utc)
active = []
idle = []
for worker_id, status, shard_id, videos_assigned, last_heartbeat in rows:
    age = (now - last_heartbeat).total_seconds()
    if status in active_statuses:
        active.append((worker_id, status, shard_id, videos_assigned, age))
    elif status in idle_statuses and (not shard_id) and (videos_assigned or 0) == 0:
        idle.append((worker_id, status, shard_id, videos_assigned, age))

print(f"total_workers={len(rows)}")
print(f"active_workers={len(active)}")
print(f"idle_workers={len(idle)}")
print("active:")
for row in active:
    print(" ", row)
print("idle:")
for row in idle:
    print(" ", row)
PY


Output:
Exit code: 0
Wall time: 0.8 seconds
Output:
total_workers=49
active_workers=0
idle_workers=0
active:
idle:


Summary: auto


**Inspecting worker status distribution**

I'm checking sample rows to verify if all claiming or idle statuses have non-null shard IDs or non-zero videos_assigned, explaining why idle_workers count is zero.


python3 - <<'PY'
from pathlib import Path
import psycopg

env={}
for line in Path('.env').read_text().splitlines():
    line=line.strip()
    if not line or line.startswith('#') or '=' not in line:
        continue
    if line.startswith('export '):
        line=line[7:]
    k,v=line.split('=',1)
    env[k.strip()] = v.strip().strip('"').strip("'")

with psycopg.connect(env['DATABASE_URL']) as conn:
    with conn.cursor() as cur:
        cur.execute("select status, count(*) from worker_heartbeats_v2 group by status order by status")
        print('status counts:', cur.fetchall())
        cur.execute("select worker_id, status, shard_id, videos_assigned, videos_processed, last_heartbeat from worker_heartbeats_v2 order by worker_id limit 10")
        print('sample:')
        for row in cur.fetchall():
            print(' ', row)
PY


Output:
Exit code: 0
Wall time: 0.8 seconds
Output:
status counts: [('claiming', 49)]
sample:
  ('maya3-worker-1', 'claiming', None, 100, 100, datetime.datetime(2026, 1, 21, 12, 56, 21, 992007, tzinfo=datetime.timezone.utc))
  ('maya3-worker-10', 'claiming', None, 100, 100, datetime.datetime(2026, 1, 21, 12, 56, 15, 643577, tzinfo=datetime.timezone.utc))
  ('maya3-worker-11', 'claiming', None, 100, 100, datetime.datetime(2026, 1, 21, 12, 56, 15, 901752, tzinfo=datetime.timezone.utc))
  ('maya3-worker-12', 'claiming', None, 100, 100, datetime.datetime(2026, 1, 21, 12, 56, 17, 118192, tzinfo=datetime.timezone.utc))
  ('maya3-worker-13', 'claiming', None, 100, 100, datetime.datetime(2026, 1, 21, 12, 56, 24, 812879, tzinfo=datetime.timezone.utc))
  ('maya3-worker-14', 'claiming', None, 100, 100, datetime.datetime(2026, 1, 21, 12, 56, 28, 604343, tzinfo=datetime.timezone.utc))
  ('maya3-worker-15', 'claiming', None, 100, 100, datetime.datetime(2026, 1, 21, 12, 56, 25, 776430, tzinfo=datetime.timezone.utc))
  ('maya3-worker-16', 'claiming', None, 100, 100, datetime.datetime(2026, 1, 21, 12, 56, 16, 637304, tzinfo=datetime.timezone.utc))
  ('maya3-worker-17', 'claiming', None, 100, 100, datetime.datetime(2026, 1, 21, 12, 56, 27, 771636, tzinfo=datetime.timezone.utc))
  ('maya3-worker-18', 'claiming', None, 100, 100, datetime.datetime(2026, 1, 21, 12, 56, 25, 535185, tzinfo=datetime.timezone.utc))


**Identifying heartbeat state inconsistency**

The heartbeat still reports all workers as claiming with full assigned/processed counts, conflicting with shard_runs showing active processing/uploading states for some workers, indicating the heartbeat state isn't updating correctly after task completion.


**Investigating stale processing leases**

Considering that processing/uploading tasks with expired leases might explain why workers show as claiming despite active shard_runs, planning to query for tasks with expired leases to confirm and decide if repeated stale claim releases are needed.


python3 - <<'PY'
from pathlib import Path
import psycopg

env={}
for line in Path('.env').read_text().splitlines():
    line=line.strip()
    if not line or line.startswith('#') or '=' not in line:
        continue
    if line.startswith('export '):
        line=line[7:]
    k,v=line.split('=',1)
    env[k.strip()] = v.strip().strip('"').strip("'")

with psycopg.connect(env['DATABASE_URL']) as conn:
    with conn.cursor() as cur:
        cur.execute(
            """
            select status, count(*)
            from video_tasks
            where status in ('processing','uploading')
              and lease_expires_at < now()
            group by status
            """
        )
        print('expired processing/uploading:', cur.fetchall())
        cur.execute(
            """
            select count(*)
            from shard_runs
            where status in ('processing','uploading')
              and lease_expires_at < now()
            """
        )
        print('expired shard_runs:', cur.fetchone()[0])
PY


Output:
Exit code: 0
Wall time: 0.8 seconds
Output:
expired processing/uploading: []
expired shard_runs: 0


**Analyzing worker state updates**

I’m investigating why workers report "claiming" while tasks show active processing statuses, suspecting the lease renewal heartbeat might override state updates despite their correctness in the processing flow. Next, I'll closely inspect the _update_state function to confirm if and how it updates worker status.


rg -n "def _update_state" -n /home/ubuntu/maya3_prepare/workers/worker.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
147:    def _update_state(self, **kwargs) -> None:


sed -n '120,180p' /home/ubuntu/maya3_prepare/workers/worker.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:

        # Initialize storage manager
        self.storage_manager = StorageManager(
            work_dir=config.work_dir,
            output_dir=config.output_dir,
            max_usage_gb=config.max_disk_usage_gb,
            cleanup_threshold_gb=config.cleanup_threshold_gb,
            check_interval_sec=config.cleanup_check_interval_sec,
        )

        # Worker state
        self.state = WorkerState()
        self._state_lock = threading.Lock()
        self._stop_event = threading.Event()
        self._heartbeat_thread: Optional[threading.Thread] = None
        self._lease_thread: Optional[threading.Thread] = None
        self._lease_stop_event = threading.Event()

        # Setup signal handlers
        signal.signal(signal.SIGTERM, self._handle_signal)
        signal.signal(signal.SIGINT, self._handle_signal)

    def _handle_signal(self, signum, frame):
        """Handle shutdown signals gracefully."""
        logger.info(f"Received signal {signum}, initiating graceful shutdown...")
        self._stop_event.set()

    def _update_state(self, **kwargs) -> None:
        """Thread-safe state update."""
        with self._state_lock:
            for key, value in kwargs.items():
                if hasattr(self.state, key):
                    setattr(self.state, key, value)

    def _get_state(self) -> dict:
        """Get current state as dict."""
        with self._state_lock:
            return {
                "status": self.state.status,
                "shard_id": self.state.shard_id,
                "videos_assigned": self.state.videos_assigned,
                "videos_processed": self.state.videos_processed,
                "segments_processed": self.state.segments_processed,
                "outputs_generated": self.state.outputs_generated,
                "last_error": self.state.last_error,
            }

    def deterministic_shard_id(self, language: str, video_ids: list[str]) -> uuid.UUID:
        """Generate deterministic shard ID from language + sorted video IDs.

        This ensures that re-processing the same videos produces the same shard_id,
        making uploads idempotent (overwrites same R2 key instead of creating duplicates).
        """
        joined = f"{language}:" + ",".join(sorted(video_ids))
        digest = hashlib.sha1(joined.encode("utf-8")).hexdigest()
        return uuid.uuid5(uuid.NAMESPACE_URL, digest)

    def claim_videos(self, conn, language: str) -> list[TaskRow]:
        """Claim a batch of pending videos with row-level locking.

        Uses FOR UPDATE SKIP LOCKED to prevent duplicate claims across workers.


**Identifying status hang in claiming state**

Workers without pending tasks stay stuck in "claiming" status because the code only sets "idle" after processing shards, not when no tasks are claimed; meanwhile, some processing/uploading tasks remain leased, implying those workers might still hold them despite showing "claiming" with no new tasks.


**Planning to check processing task ownership**

I'm querying processing and uploading tasks grouped by their claimed owner to see if any active workers hold the leases despite the status discrepancy.


python3 - <<'PY'
from pathlib import Path
import psycopg

env={}
for line in Path('.env').read_text().splitlines():
    line=line.strip()
    if not line or line.startswith('#') or '=' not in line:
        continue
    if line.startswith('export '):
        line=line[7:]
    k,v=line.split('=',1)
    env[k.strip()] = v.strip().strip('"').strip("'")

with psycopg.connect(env['DATABASE_URL']) as conn:
    with conn.cursor() as cur:
        cur.execute("select status, claimed_by, count(*) from video_tasks where status in ('processing','uploading') group by status, claimed_by order by status, count desc")
        rows = cur.fetchall()
        for row in rows:
            print(row)
PY


Output:
Exit code: 0
Wall time: 0.7 seconds
Output:


**Rechecking status counts**

The latest query returned no results despite previously seeing processing and uploading counts; I’ll re-run the count to verify current statuses.


python3 - <<'PY'
from pathlib import Path
import psycopg

env={}
for line in Path('.env').read_text().splitlines():
    line=line.strip()
    if not line or line.startswith('#') or '=' not in line:
        continue
    if line.startswith('export '):
        line=line[7:]
    k,v=line.split('=',1)
    env[k.strip()] = v.strip().strip('"').strip("'")

with psycopg.connect(env['DATABASE_URL']) as conn:
    with conn.cursor() as cur:
        cur.execute("select status, count(*) from video_tasks group by status order by status")
        print(cur.fetchall())
PY


Output:
Exit code: 0
Wall time: 0.9 seconds
Output:
[('completed', 507387)]


**Verifying task and shard completion**

I'm confirming all video tasks are fully done with no processing left and will validate shard uploads by checking R2 object counts and shard run statuses, including looking for any stale shards or pending language queue tasks.


python3 - <<'PY'
from pathlib import Path
import psycopg

env={}
for line in Path('.env').read_text().splitlines():
    line=line.strip()
    if not line or line.startswith('#') or '=' not in line:
        continue
    if line.startswith('export '):
        line=line[7:]
    k,v=line.split('=',1)
    env[k.strip()] = v.strip().strip('"').strip("'")

with psycopg.connect(env['DATABASE_URL']) as conn:
    with conn.cursor() as cur:
        cur.execute("select status, count(*) from shard_runs group by status order by status")
        print('shard_runs:', cur.fetchall())
        cur.execute("select language, status from language_queue order by language")
        print('language_queue:', cur.fetchall())
        cur.execute("select active_language from language_state where id=1")
        print('language_state:', cur.fetchone())
PY


Output:
Exit code: 0
Wall time: 0.9 seconds
Output:
shard_runs: [('completed', 5084)]
language_queue: [('assamese', 'completed'), ('bengali', 'completed'), ('english', 'completed'), ('gujarati', 'completed'), ('hindi', 'completed'), ('kannada', 'completed'), ('malayalam', 'completed'), ('marathi', 'completed'), ('odia', 'completed'), ('punjabi', 'completed'), ('tamil', 'completed'), ('telugu', 'completed')]
language_state: (None,)


**Planning full R2 object count**

I'm preparing to verify total objects per language in R2 by listing and counting objects by prefix, balancing completeness with potentially heavy listing; this aligns with the user's request to confirm progress and total bucket contents.


**Designing comprehensive R2 verification script**

I'm planning a Python script to count objects and total bytes per language prefix in R2, including manifests and stats, balancing completeness with listing cost; afterward, I'll verify all tasks are done before safely shutting down unused instances.


python3 - <<'PY'
from pathlib import Path
import boto3
import psycopg

# Load env
env = {}
for line in Path('.env').read_text().splitlines():
    line = line.strip()
    if not line or line.startswith('#') or '=' not in line:
        continue
    if line.startswith('export '):
        line = line[7:]
    key, val = line.split('=', 1)
    env[key.strip()] = val.strip().strip('"').strip("'")

bucket = env.get('R2_BUCKET_SHARDS') or env.get('R2_PROCESSED_BUCKET') or env.get('R2_BUCKET')
if not bucket:
    raise SystemExit('No R2 bucket found in .env')

client = boto3.client(
    's3',
    endpoint_url=env.get('R2_ENDPOINT_URL'),
    aws_access_key_id=env.get('R2_ACCESS_KEY_ID'),
    aws_secret_access_key=env.get('R2_SECRET_ACCESS_KEY'),
)

# Pull languages from DB
languages = []
with psycopg.connect(env['DATABASE_URL']) as conn:
    with conn.cursor() as cur:
        cur.execute("select language from language_queue order by language")
        languages = [row[0] for row in cur.fetchall()]

sr_suffixes = ["16", "24", "32", "44"]

def list_objects(prefix: str):
    paginator = client.get_paginator('list_objects_v2')
    count = 0
    size = 0
    for page in paginator.paginate(Bucket=bucket, Prefix=prefix):
        for obj in page.get('Contents', []):
            count += 1
            size += obj.get('Size', 0)
    return count, size

print(f"bucket={bucket}")

for language in languages:
    total_count = 0
    total_size = 0
    for sr in sr_suffixes:
        prefix = f"{language}_{sr}/"
        count, size = list_objects(prefix)
        print(f"{language} {prefix} objects={count} bytes={size}")
        total_count += count
        total_size += size

    manifest_prefix = f"manifests/{language}_"
    stats_prefix = f"stats/{language}_"
    man_count, man_size = list_objects(manifest_prefix)
    stats_count, stats_size = list_objects(stats_prefix)
    print(f"{language} {manifest_prefix} objects={man_count} bytes={man_size}")
    print(f"{language} {stats_prefix} objects={stats_count} bytes={stats_size}")

    total_count += man_count + stats_count
    total_size += man_size + stats_size
    print(f"{language} TOTAL objects={total_count} bytes={total_size}")

# Overall manifest/stats totals
all_man_count, all_man_size = list_objects('manifests/')
all_stats_count, all_stats_size = list_objects('stats/')
print(f"manifests/ TOTAL objects={all_man_count} bytes={all_man_size}")
print(f"stats/ TOTAL objects={all_stats_count} bytes={all_stats_size}")
PY


Output:
Exit code: 0
Wall time: 35.4 seconds
Output:
bucket=xcodec
assamese assamese_16/ objects=78 bytes=85689836399
assamese assamese_24/ objects=76 bytes=114480925629
assamese assamese_32/ objects=78 bytes=121085129397
assamese assamese_44/ objects=76 bytes=42698336956
assamese manifests/assamese_ objects=76 bytes=2214093949
assamese stats/assamese_ objects=38 bytes=15372
assamese TOTAL objects=422 bytes=366168337702
bengali bengali_16/ objects=408 bytes=360792812741
bengali bengali_24/ objects=406 bytes=449239466029
bengali bengali_32/ objects=408 bytes=472307316428
bengali bengali_44/ objects=406 bytes=210868607922
bengali manifests/bengali_ objects=406 bytes=9110336112
bengali stats/bengali_ objects=203 bytes=81746
bengali TOTAL objects=2237 bytes=1502318620978
english english_16/ objects=1198 bytes=1103038722774
english english_24/ objects=1196 bytes=1374119586912
english english_32/ objects=1198 bytes=1526026985548
english english_44/ objects=1194 bytes=660195806881
english manifests/english_ objects=1196 bytes=26172688813
english stats/english_ objects=598 bytes=235244
english TOTAL objects=6580 bytes=4689554026172
gujarati gujarati_16/ objects=822 bytes=853243171048
gujarati gujarati_24/ objects=820 bytes=1031326126762
gujarati gujarati_32/ objects=822 bytes=1009491971280
gujarati gujarati_44/ objects=820 bytes=423080823709
gujarati manifests/gujarati_ objects=820 bytes=20986748779
gujarati stats/gujarati_ objects=410 bytes=165569
gujarati TOTAL objects=4514 bytes=3338129007147
hindi hindi_16/ objects=1182 bytes=1023794631156
hindi hindi_24/ objects=1180 bytes=1293440923944
hindi hindi_32/ objects=1182 bytes=1332622076837
hindi hindi_44/ objects=1180 bytes=515048580539
hindi manifests/hindi_ objects=1180 bytes=26563281155
hindi stats/hindi_ objects=590 bytes=235663
hindi TOTAL objects=6494 bytes=4191469729294
kannada kannada_16/ objects=916 bytes=813101962090
kannada kannada_24/ objects=914 bytes=982655711811
kannada kannada_32/ objects=916 bytes=990820783481
kannada kannada_44/ objects=914 bytes=449323035877
kannada manifests/kannada_ objects=914 bytes=19267117875
kannada stats/kannada_ objects=457 bytes=183778
kannada TOTAL objects=5031 bytes=3255168794912
malayalam malayalam_16/ objects=1308 bytes=1089936091120
malayalam malayalam_24/ objects=1306 bytes=1270126799419
malayalam malayalam_32/ objects=1308 bytes=1206757943520
malayalam malayalam_44/ objects=1306 bytes=526010194193
malayalam manifests/malayalam_ objects=1306 bytes=24498733897
malayalam stats/malayalam_ objects=653 bytes=262206
malayalam TOTAL objects=7187 bytes=4117330024355
marathi marathi_16/ objects=330 bytes=299468671092
marathi marathi_24/ objects=328 bytes=374258328598
marathi marathi_32/ objects=330 bytes=356210482033
marathi marathi_44/ objects=328 bytes=160296506253
marathi manifests/marathi_ objects=328 bytes=7677449198
marathi stats/marathi_ objects=164 bytes=65409
marathi TOTAL objects=1808 bytes=1197911502583
odia odia_16/ objects=388 bytes=293794779507
odia odia_24/ objects=386 bytes=338128127061
odia odia_32/ objects=388 bytes=312870502458
odia odia_44/ objects=386 bytes=147602569460
odia manifests/odia_ objects=386 bytes=6668805992
odia stats/odia_ objects=193 bytes=77634
odia TOTAL objects=2127 bytes=1099064862112
punjabi punjabi_16/ objects=1120 bytes=1077385089275
punjabi punjabi_24/ objects=1118 bytes=1291461843191
punjabi punjabi_32/ objects=1120 bytes=1223927110548
punjabi punjabi_44/ objects=1118 bytes=473331205680
punjabi manifests/punjabi_ objects=1118 bytes=25563867927
punjabi stats/punjabi_ objects=559 bytes=224269
punjabi TOTAL objects=6153 bytes=4091669340890
tamil tamil_16/ objects=1006 bytes=676158604337
tamil tamil_24/ objects=1004 bytes=816661059936
tamil tamil_32/ objects=1006 bytes=705700514297
tamil tamil_44/ objects=1004 bytes=264824010299
tamil manifests/tamil_ objects=1004 bytes=15364123529
tamil stats/tamil_ objects=502 bytes=200103
tamil TOTAL objects=5526 bytes=2478708512501
telugu telugu_16/ objects=1444 bytes=1141095908401
telugu telugu_24/ objects=1440 bytes=1358235826770
telugu telugu_32/ objects=1444 bytes=1296211629528
telugu telugu_44/ objects=1438 bytes=504230640970
telugu manifests/telugu_ objects=1440 bytes=28224856286
telugu stats/telugu_ objects=720 bytes=290364
telugu TOTAL objects=7926 bytes=4327999152319
manifests/ TOTAL objects=10178 bytes=212312417698
stats/ TOTAL objects=5090 bytes=2037651


Summary: auto


**Planning instance shutdown**

All tasks and language queues are complete, so now I'll list all active instances to capture their IDs for destruction, verifying no pending work remains before shutting them down.