o i57 @sdZddlZddlmZmZGdddZGdddeZGdd d eZGd d d eZ dde de de de de f ddZ dS)a Text Chunking Utility for Long Inputs Splits long texts into manageable chunks with smart boundary detection. Supports English, Hindi, Telugu, and mixed-language content. Why Text-Based Chunking? - Preserves semantic meaning (sentences, paragraphs) - Natural boundaries for better prosody - Simpler than token-based (model handles tokenization internally) - Works with SNAC token sliding window (separate layer) N)ListTuplec @speZdZdZddefddZdedeefdd Zdedefd d Z dded edee eeeffddZ dS) TextChunkerz Smart text chunking for long inputs. Splits text at natural boundaries (paragraphs, sentences, commas, spaces) to avoid mid-word/mid-sentence splits. max_chunk_lengthcCs ||_dS)z Initialize text chunker. Args: max_chunk_length: Maximum characters per chunk (default: 1000) N)r)selfrr>/home/ubuntu/veenaModal/veena3modal/processing/text_chunker.py__init__s zTextChunker.__init__textreturncCst||jkr |gSg}|}t||jkrB|d|j}||}|dkr(|j}||d|||d}t||jks|rI|||S)a3 Split text into chunks at natural boundaries. Priority: 1. Paragraph breaks (\n\n) 2. Sentence ends (. ! ?) 3. Commas 4. Spaces Args: text: Text to chunk Returns: List of text chunks Nr)lenr_find_split_pointappendstrip)rr chunks remainingchunk split_pointrrr chunk_text#s  zTextChunker.chunk_textcCsztd|}|r |Sd}tt||}|r|dSttd|}|r.|dS|d}|dkr;|dSdS)z Find the best split point in text. Returns index of split point (0 if none found). Enhanced to handle: - Abbreviations (Dr., Mr., Mrs., etc.) - Decimals (3.14) - Ellipsis (...) \n\nz)[.!?](?!\.\.)(?![0-9])(?![A-Z][a-z]\.)\s+,\s+ rresearchendlistfinditerrfind)rr para_matchsentence_patternsentence_matches comma_matches space_matchrrr rLs    zTextChunker._find_split_point2 overlap_charscCst||jkr|dt|fgSg}d}|t|kr^t||jt|}|||}|t|krB||}|dkrB||}|||}||||f|t|krV||n|}|t|ks|S)a Chunk text with overlap for crossfade stitching. Args: text: Text to chunk overlap_chars: Number of characters to overlap (default: 50) Returns: List of (chunk_text, start_idx, end_idx) tuples r)r rminrrr)rr r(r start_idxend_idxrrrrr chunk_with_overlaprs       zTextChunker.chunk_with_overlapN)r)r') __name__ __module__ __qualname____doc__intr strrrrrr,rrrr rs ))rcs*eZdZdZdedeffdd ZZS)SentenceBoundaryChunkerz Enhanced chunker that prioritizes sentence boundaries. Better handles abbreviations, decimals, and edge cases. r r csd}tt||}g}|D]1}|}|td|d|dtfdddDr-qdvs9d vs9d vr:q||q|rH|d St |S) z Find split point prioritizing sentence boundaries. Enhanced detection to avoid splitting on: - Common abbreviations (Dr., Mr., Mrs., Prof., etc.) - Decimal numbers (3.14) - Ellipsis (...) [.!?](?!\.\.)(?![0-9])\s+rrc3s|]}|vVqdS)Nr).0abbrlookbackrr sz.)zDr.zMr.zMrs.zMs.zProf.zSr.zJr.ze.g.zi.e.zetc.r) rrr startmaxanyrrsuperr)rr r#r$ valid_splitsmatchpos __class__r8r rs    z)SentenceBoundaryChunker._find_split_point)r-r.r/r0r2r1r __classcell__rrrBr r3sr3c@s4eZdZdZdedefddZdedefddZdS) IndicSentenceChunkeru Enhanced chunker with Indic language support (Hindi, Telugu, multilingual). Handles: - Hindi: Devanagari danda (।), double danda (॥) - Telugu: Sentence terminators (same Unicode as Hindi) - English: Standard punctuation (. ! ?) - Mixed language content Why this matters: - Indic languages use different punctuation (। instead of .) - Better prosody when splitting at natural language boundaries - Preserves meaning across languages r r cCstd|}|r |Std|}|r|Std|}|r$|Sttd|}|r4|dS|d}|dkrA|dSdS) u Find split point with Indic language awareness. Priority: 1. Paragraph breaks ( ) 2. Indic sentence markers (। ॥) 3. English sentence markers (. ! ?) 4. Commas 5. Spaces ru [।॥]\s*r4rrrrrr)rr r"indic_sentenceenglish_sentencer%r&rrr rs&    z&IndicSentenceChunker._find_split_pointc Csttd|}ttd|}ttd|}|||}|dkr)dddddS||d}||d}||d}|t||krEd} n|t||krOd } n |t||krYd } nd } | t|d t|d t|d dS) z Detect language composition in text. Returns: dict with language percentages and primary language z[\u0900-\u097F]z[\u0C00-\u0C7F]z[a-zA-Z]runknown)primaryhinditeluguenglishdrJrKrLmixedr)r rfindallr<round) rr devanagari_chars telugu_chars english_chars total_chars hindi_pct telugu_pct english_pctrIrrr detect_language_mixs*    z(IndicSentenceChunker.detect_language_mixN) r-r.r/r0r2r1rdictrXrrrr rEs/rEcs.eZdZdZdedeeffdd ZZS)ParagraphChunkerz Chunker that splits primarily on paragraph boundaries. Useful for document-style text with clear paragraph structure. r r cst||jkr |gStd|}g}d}|D];}|}|sq|r'|d|n|}t||jkr3|}q|r:||t||jkrOt|}||d}q|}q|rY|||r]|S|gS)z#Split text on paragraph boundaries.z\n\n+z ) r rrsplitrrr>rextend)rr paragraphsr current_chunkpara test_chunk para_chunksrBrr rAs,     zParagraphChunker.chunk_text)r-r.r/r0r2rrrDrrrBr rZ:s"rZ]audio1audio2crossfade_samples sample_rater c Csddl}|j||jd}|j||jd}|dkr||St||ks(t||kr,||S|dd|}|dd|}|| d} |d|} | || ||j} ||d| | ||dg} | S)a' Crossfade two audio segments for seamless stitching. Uses linear crossfade with smooth amplitude transition to avoid pops, clicks, or audible artifacts at chunk boundaries. Args: audio1: First audio segment (int16 PCM) audio2: Second audio segment (int16 PCM) crossfade_samples: Number of samples to crossfade (default: 1200 = 50ms @ 24kHz) sample_rate: Sample rate (default: 24000 Hz) Returns: Stitched audio with crossfade Example: >>> audio1 = b'\x00\x01' * 24000 # 1 second >>> audio2 = b'\x00\x01' * 24000 # 1 second >>> result = crossfade_audio(audio1, audio2, crossfade_samples=1200) >>> len(result) < len(audio1) + len(audio2) # Overlap reduces total length True rN)dtypeg?g)numpy frombufferint16r linspaceastype concatenatetobytes) rerfrgrhnparr1arr2fade_outfade_intailhead crossfadedresultrrr crossfade_audiols$   rz)rcrd) r0rtypingrrrr3rErZbytesr1rzrrrr s*  3j5