o i@sDdZddlmZmZmZmZddlZGdddZdefddZdS) u BiCodec Token Utilities - Incremental Parsing Provides O(1) per-token parsing instead of O(n) decode + regex scanning. The key optimization: instead of: tokenizer.decode(all_tokens) # O(n) re.findall(entire_text) # O(n) We do: for new_token_id in new_tokens: # Only new tokens parsed = token_cache.get(token_id) # O(1) lookup This eliminates the O(n²) CPU burn in the streaming hot loop. )DictOptionalTupleListNc @seZdZdZedZedZddZddd Z d e de e e e ffd d Zd e de e e e ffddZdee dee dee de ee ee ffddZdee e ffddZdS)BiCodecTokenParsera Caching parser for BiCodec tokens. Converts vLLM token IDs to (type, value) tuples efficiently. BiCodec tokens in the vocabulary look like: - <|bicodec_semantic_123|> -> ("semantic", 123) - <|bicodec_global_7|> -> ("global", 7) Usage: parser = BiCodecTokenParser(tokenizer) # In hot loop (O(1) per token): for token_id in new_token_ids: result = parser.parse(token_id) if result: token_type, value = result if token_type == "semantic": semantic_buffer.append(value) elif token_type == "global": global_buffer.append(value) z^<\|bicodec_semantic_(\d+)\|>$z^<\|bicodec_global_(\d+)\|>$cCs||_i|_|dS)z Initialize parser with tokenizer. Args: tokenizer: HuggingFace tokenizer with BiCodec special tokens N) tokenizer_cache_prewarm_cache)selfrr 7/home/ubuntu/veenaModal/veena3modal/core/token_utils.py__init__1s zBiCodecTokenParser.__init__returnNcCsZz"|j}|D]\}}|dr||}|r||j|<q WdSty,YdSw)z Pre-populate cache with BiCodec tokens from vocabulary. This avoids lazy cache misses during streaming. z <|bicodec_N)r get_vocabitems startswith_parse_token_stringr Exception)r vocab token_strtoken_idparsedr r r r As     z!BiCodecTokenParser._prewarm_cachercCsH|j|}|rdt|dfS|j|}|r"dt|dfSdS)a Parse a token string into (type, value) tuple. Args: token_str: Token string like "<|bicodec_semantic_123|>" Returns: ("semantic", 123) or ("global", 7) or None if not BiCodec token semanticglobalN)SEMANTIC_PATTERNmatchintgroupGLOBAL_PATTERN)r rrr r r rSs  z&BiCodecTokenParser._parse_token_stringrcCsd||jvr |j|Sz|jj|gdd}||}||j|<|WSty1d|j|<YdSw)a Parse a single token ID into (type, value) tuple. O(1) for cached tokens, O(1) amortized for uncached. Args: token_id: vLLM token ID Returns: ("semantic", value) or ("global", value) or None F)skip_special_tokensN)rrdecoderstripr)r rrrr r r parseis    zBiCodecTokenParser.parse new_token_idssemantic_buffer global_buffercCsN|D] }||}|r"|\}}|dkr||q|dkr"||q||fS)a Parse new tokens and append to existing buffers. This is the main method for incremental streaming: - Only processes NEW tokens (not entire history) - Modifies buffers in-place for efficiency - Returns the updated buffers (same objects) Args: new_token_ids: Only the new token IDs since last call semantic_buffer: Existing semantic token values (modified in-place) global_buffer: Existing global token values (modified in-place) Returns: Tuple of (semantic_buffer, global_buffer) - same objects, modified rr)r#append)r r$r%r&rr token_typevaluer r r parse_incrementals   z$BiCodecTokenParser.parse_incrementalcCs2t|j}tdd|jD}||||dS)z Return cache statistics for debugging. Returns: Dict with cache size and breakdown css|] }|durdVqdS)Nrr ).0vr r r sz5BiCodecTokenParser.get_cache_stats..) total_cachedbicodec_tokensnon_bicodec_tokens)lenrsumvalues)r total bicodec_countr r r get_cache_statss z"BiCodecTokenParser.get_cache_stats)rN)__name__ __module__ __qualname____doc__recompilerrr r strrrrrr#rr*rr6r r r r rs$    !rrcCst|S)z Factory function to create a BiCodec token parser. Args: tokenizer: HuggingFace tokenizer Returns: Configured BiCodecTokenParser instance )r)rr r r create_bicodec_parsers r>) r:typingrrrrr;rr>r r r r s !