o 3NPi@sdZddlZddlZddlZddlZddlmZddlmZm Z m Z m Z ddl Z ddl Z edZGdddeZGdd d eZd/d ed ededeeee ffddZdedefddZddZd edefddZd0dedededede ef ddZ d1d ed!ededeee ee fffd"d#Zd ededed$ed%ed&ed'ed(ed)eded*ede ee ffd+d,Zd ededed$ed'ed(ed)eded*ede ee ffd-d.ZdS)2uAudio download and preprocessing with robust error handling. === v7.0 OPTIMIZATION: Download-Time Resample === Uses ffmpeg directly to create both 16kHz processing audio AND original quality audio in a single pass, eliminating Python resampling overhead. Previous: yt-dlp → temp.wav → torchaudio load → Python resample → save both Now: yt-dlp → ffmpeg pipe → dual output (16kHz + original) in one pass Saves ~10-15s per video by avoiding torchaudio resampling. N)Path)TupleDictAnyOptionalzFastPipelineV6.Downloadc@eZdZdZdS)VideoValidationErrorz*Raised when video fails validation checks.N__name__ __module__ __qualname____doc__rrR/home/ubuntu/.cursor/worktrees/maya3data__SSH__216.81.248.184_/zxg/src/download.pyrrc@r) DownloadErrorz)Raised when download fails after retries.Nr rrrrr rr>@ @ video_url min_duration max_durationreturnc Cstd|ddddd}zt|}|j|dd}|dur.ddifWdWS|dd krG|d sGdd |fWdWS|d d }||kredd|d|d|fWdWS||kr}dd|d|d|fWdWS|ddkrdd|fWdWS|dd d krtd|dd|drdd|fWdWS|dd}td|d|ddd|fWdWS1swYWdStjjy0}zAt |} d | vrdd!ifWYd}~Sd"| vrddifWYd}~Sd#| vrdd$ifWYd}~Sdd%| ifWYd}~Sd}~wt yL}zdd&t |ifWYd}~Sd}~ww)'a Validate YouTube video before processing. Checks: - Video exists and is accessible - Video duration is within acceptable range - Video has audio track - Video is not age-restricted or private Args: video_url: YouTube URL to validate min_duration: Minimum video duration in seconds (default: 30s) max_duration: Maximum video duration in seconds (default: 4 hours) Returns: (is_valid, message, info_dict) u🔍 Validating video: TF)quiet no_warnings extract_flat skip_downloaddownloadNz#Video info extraction returned NoneacodecnoneformatszVideo has no audio trackdurationrzVideo too short: zs (min: s)zVideo too long: zs (max: availabilityprivatezVideo is private age_limitu*⚠️ Video is age-restricted (age_limit=)is_livezLive streams are not supportedtitleUnknownu✅ Video validated: '' (OKzVideo unavailablezVideo unavailable or deletedz Private videozSign inz2Video requires sign-in (age-restricted or private)zDownload error: zValidation failed: ) loggerinfoyt_dlp YoutubeDL extract_infogetwarningutilsrstr Exception) rrrydl_optsydlr-r!r(e error_msgrrrvalidate_video%s`    ("   r:total_durationcCsd}d}d}||kr |S|S)u Calculate dynamic intro skip based on video duration. Strategy (when no chapter info available): - Video ≤ 30 min (1800s): Skip 180s (3 min) - ~10% intro ratio - Video > 30 min: Skip 300s (5 min) - smaller % for longer content This is ONLY used when no intro chapter is detected. Chapter-based detection always takes priority. Args: total_duration: Total video duration in seconds Returns: Intro skip duration in seconds gf@gr@r)r;THRESHOLD_30MIN SKIP_SHORT SKIP_LONGrrrget_dynamic_intro_skipqs r@csgd}gd}d}d}|rt|dkr|d}|dd|dd}|dd} | |} tfd d |D} | d kre| }| rTtd |dd | ddn3td|dd | ddn"| r}t| d }td|dd |ddn td| ddt|dkr|d} | dd| dd}| d|} tfdd |Dr| |d kr||}td| dd |dd|ot|dk} | st|ddr|jdkrt |}|dkrdnd}td|dd|d ||fS)!u Detect intro/outro sections from YouTube chapters. === PRIORITY (v6.8) === 1. Chapter-based detection (HIGHEST priority) - If intro keyword found → skip that chapter - If chapters exist but NO intro keyword → STILL skip first chapter 2. Dynamic duration-based skip (fallback when NO chapters at all) 3. config.intro_skip_seconds (manual override from CLI) Strategy: - Intro: ALWAYS skip first chapter if chapters exist (creator usually puts intro there) - Outro: Last chapter with keywords like "outro", "ending", "credits", "sponsor" - Fallback: Dynamic skip based on video duration (180s for ≤30min, 300s for >30min) Returns: (intro_seconds, outro_seconds): Trim amounts )introopening advertisementsponsoradpromo)outroendingcreditsrDrErFendcardrr( start_timeend_timec3|]}|vVqdSNr.0keyword chapter_titlerr z3detect_intro_outro_from_chapters..i,u( 📖 Chapter intro (keyword match): 'r*.1fr"u( 📖 Chapter intro (first chapter): 'u* 📖 Chapter intro (keyword, capped): 'u" ⚠️ First chapter too long (.0fzs), not skippingc3rOrPrrQrTrrrVrWu 📖 Chapter-based outro: 'auto_intro_skipTr<u≤30minz>30minu, ⏱️ Dynamic intro skip (no chapters): z s (video r&) lenr1loweranyr,r-mingetattrintro_skip_secondsr@)chaptersr;configintro_keywordsoutro_keywords intro_skip outro_skip first_chapter chapter_start chapter_endchapter_durationhas_intro_keyword last_chapter has_chaptersduration_thresholdrrTr detect_intro_outro_from_chapterssH  "" "       rqcCszd|vr|ddddSd|vr |ddddSd|vr0|ddddSt|dd S) z2Extract video ID from various YouTube URL formats.zwatch?v=rZ&rz youtu.be/?z/shorts/N )splithashlibmd5encode hexdigest)rrrrextract_video_idsrzurl output_dirvideo_id max_retriesc Csjtd|t}||d}|r(|jdkr(td||Sdt||ddddd d d }t|D]l}z>t |}|j |dd Wd n1sWwY|r|td t|dd|jddd|WSWq=t y} z t d|dd| ||dkrt d|WYd } ~ q=d } ~ wwtd|dd S)z,Download video for visualization (720p MP4).u*🎥 Downloading video for visualization: z.mp4i@Bu✅ Using cached video: z2bestvideo[height<=720]+bestaudio/best[height<=720]z.%(ext)smp4Tr{)formatouttmplmerge_output_formatrrretriesfragment_retriesrNu✅ Video download: rXz s | Size: g.AMBzVideo download attempt rZ failed: u ❌ Video download failed after z attempts)r,r-timeexistsstatst_sizer4ranger.r/r0r5r2sleeperror) r|r}r~rstart output_filer6attemptr7r8rrr download_video_for_visualizations>  .  rTvalidatecCstd|t}t|}t|j|}|jdddi}|r2t|\}} }|s2td| |s}tddddd} zt | } | j |dd}Wd n1sWwYWnt y|} zt d | d gd }WYd } ~ nd } ~ ww||d } ||d}t|dd}| rTd}|r|sd}td|rTtd| zstt| \}}|jd|dkrt d| |r|nM|}|rz tt|}|j}WnYt| |d||dd|jd|d|jd|t||dg|||rt|nd |d fWSWn0t yS} z#t d| dz| |rC|WnYWYd } ~ nd } ~ ww|dd}|rc|dgng}t|||\}}|jdkr||jkrtd|jd d!|d d"|j}|rt|||| ||j|||||d# }nt|||| ||||||d$ }t|}td%|d d&|d'd(d)t| |fS)*u Download audio from YouTube URL and prepare for processing. === v7.0 OPTIMIZATION: Download-Time Resample === Uses ffmpeg directly for dual-output in single pass: - 16kHz mono WAV for processing pipeline - Original quality mono WAV for high-quality export Previous: yt-dlp → temp.wav → torchaudio load → Python resample → save both Now: yt-dlp → single ffmpeg → dual output (NO Python resampling!) Saves ~10-15s per video. Args: video_url: YouTube URL config: Pipeline configuration validate: Whether to validate video before download (default: True) max_retries: Number of download retries (default: 3) Returns: (audio_path, metadata): Path to processed audio and video metadata Raises: VideoValidationError: If video fails validation DownloadError: If download fails after retries u📥 Downloading: T)parentsexist_okzVideo validation failed: u📋 Fetching video metadata...F)rrrrNzCould not fetch metadata: r))r(rcz _trimmed.wav _original.wavpreserve_original_audiouF⚠️ Cached trimmed file exists but original missing, re-downloadingu✅ Using cached audio: rZ u?⚠️ Cached file seems corrupted or too short, re-downloading https://www.youtube.com/watch?v=r(CachedrKrc r~ youtube_urlr(original_duration intro_skippedprocessed_durationr}rc sample_rateoriginal_sample_rateoriginal_audio_pathoriginal_audio_preservedu#⚠️ Failed to load cached file: z, re-downloadingr!r#⚙️ Manual intro skip override: rXs (was: r") rr~r} trimmed_file original_file target_srrgrhrrr-) rr~r}rrdrgrhrrr-u✅ Download: zs | Duration: rrYs)r,r-rrzrr}mkdirr:rr.r/r0r5r2rar torchaudioloadr4shapeunlinkrr1rqrb!_download_with_ffmpeg_dual_output_download_standard)rrdrrrr~r}r-is_validmessage ydl_info_optsr7r8rrpreserve_original cache_validwaveformsrorig_sr orig_inforrcrgrhresultelapsedrrrdownload_audios                  rrrrrgrhrr-c  CsDtd||d} dt||ddddddd gd d gd d d } d} t| D]}zltd|dd| t| }|j|dd}Wdn1sRwY| s||dD]+}|j dvr|j dkrt j ddt|d d t| dgddd| n| | nqc| rWn2tdty}z"|} td|dd||| dkrtd|WYd}~q+d}~ww| std| d| ztt| }|j}|j|}Wn ty}ztd|d d!}|}WYd}~nd}~wwtd"|d#|d$d%|d&kr|nd&}|d&kr'||n|}||}ddt| d't|d(t|g}|d)d*d+t|d d dt|g|d)d*d d dt|gtd,|d-|d$d.|d$d/z t j |dddd0}Wnt jy}ztd1|jtd2|d}~ww|std3||std4|ztt|}|j|j}Wn|}Yz| WnY||dD]}z| WqYqtd5|jd6|d7|jd6|d8 |d9|| r| d:d;nd;|||t|| r| da === v7.0 OPTIMIZATION: Download-Time Dual Output === Single ffmpeg command creates both outputs: 1. 16kHz mono WAV (for processing) - with intro/outro trim 2. Original quality mono WAV (for export) - with intro/outro trim This eliminates Python resampling overhead (~10-15s per video). Key insight: ffmpeg can output to multiple files from single input stream. u/⚡ Using optimized dual-output download (v7.0)z_raw.wavbestaudio/bestz _raw.%(ext)sTFFmpegExtractAudiowav0keypreferredcodecpreferredquality-ac1r{rrrrpostprocessorspostprocessor_argsrrN Download attempt rZ/rz_raw.*.wavz.mp3z.m4az.opusz.webmrffmpeg-i-ycapture_outputcheckzDownloaded file not foundrrDownload failed after attempts: zCould not get audio info: z, using metadata durationiu 📊 Raw audio: Hz, rXrrz-ssz-tz-mapz0:a-aru# ⚡ FFmpeg dual output: 16kHz + z Hz (trim: zs-r")rtextrzFFmpeg failed: zFFmpeg dual output failed: z$FFmpeg did not create trimmed file: z%FFmpeg did not create original file: u ✅ Dual output complete:  (zHz), zHz)rr(r)rcr)r,r-r4rr.r/r0rglobsuffix subprocessrunrrenamerr5r2rrrr num_framesextendCalledProcessErrorrstderrnamer1)rr~r}rrrrgrhrrr-raw_filer6 last_errorrr7dl_infoext_filer8 audio_info original_sractual_durationrMrN trim_duration ffmpeg_cmdr proc_infortemprrrrs           "   (rc ) Cs ||d} ||d} ||d} t|dd} t|dd}| rDdt| d d d d d d g|dkr7ddgndt|ddgddd}ndt| d d d d d d gdt|jddgddd}d}t|D]}zotd|dd|t|}|j|d dWdn1swY| s| |dD]0}|j dvr|j dkrt j ddt|dt|jddt| dg d d d|n|| nq| rWn1Wqcty}z"|}td|dd|||dkrtd |WYd}~qcd}~ww| std!|d"|z tt| \}}Wnty;}z| r0| td#|d}~ww|}|jd|}|}d}| r||jkrtd$|d%| jt| }ddlm}|j||jd&}||}|j}td'|d(|d)krtd*|d+d,| r| d-gng}t|||\}}|jdkr||jkrtd.|jd+d/|d+d0|j}|}t||}|dkrt||nd}|dkr|jd|n|jd} |dks|dkr|dd|| f}|jddkr |j dd d1}|d  !"}!|!d2kr"td3|!d4d5t#t|||d}"| r| rzktt| \}#}$|#jddkrN|#j dd d1}#t||$}%|dkr_t||$nd}&|dkrm|#jd|&n|#jd}'|#dd|%|'f}#t#t| |#|$t| }"td6| jd7|$d8|#jd|$d+d0Wnty}ztd9|WYd}~nd}~ww| r| | |dD]}(z|(WqYq|d:|| r| d;d<nd<|||jd|t|||| r|n||"|"dud= S)>z=Standard download without original preservation (16kHz only).z _temp.%(ext)sz _temp.wavrrForiginal_audio_sample_raterrTrrrrrrrr{rNrrZrrz_temp.*rrrrrrrrrrzAudio file corrupted: u# 🎵 Preserving original audio: uHz → ) orig_freqnew_frequ! 📊 Created processing copy: Hzu⚠️ Audio very short: rXrrcrrr")dimkeepdimg-C6?u'⚠️ Audio appears to be silent (RMS=z.6fr&u ✅ Saved original quality: rru) ⚠️ Failed to save original audio: rr(r)r)$rar4rrr,r-r.r/r0rrrrrrrr5r2rrrrrrrtorchaudio.transforms transformsResampler1rqrbintmeansqrtitemsave))rr~r}rrdrgrhrrr- temp_filedownloaded_filerroriginal_sr_targetr6rrr7rr8rrrrrT resamplerrcr intro_samples outro_samples end_samplermsoriginal_audio_saved orig_waveformrorig_intro_samplesorig_outro_samplesorig_end_samplerrrrrhs                 "  "0   r)rr)r{)Tr{) r rloggingrvrpathlibrtypingrrrrrr. getLoggerr,r5rrr4floatboolr:r@rqrzrrrdictrrrrrrs   &LS$ ' )    2