o ½Òi· ã@sddZddlZddlZddlmZddlmZe d¡Zde de fdd „Z dd d „Z d e fdd„Z dS)av Utility functions for the extended Indic tokenizer. The extended tokenizer uses BPE merges that compose byte-fallback tokens (<0xNN>) into Indic subwords. Encoding works perfectly. Decoding requires a post-processing step to convert <0xNN> patterns back to Unicode text, because HF's ByteFallback decoder only handles individual byte tokens, not merged multi-byte tokens. éN)ÚPath)ÚPreTrainedTokenizerFastz<0x([0-9A-Fa-f]{2})>ÚtextÚreturncCs g}d}|t|ƒkrKt ||¡}|r:g}|r-| t| d¡dƒ¡| ¡}t ||¡}|s| t|ƒjddd¡n | ||¡|d7}|t|ƒks d  |¡S)uHConvert contiguous <0xNN> byte patterns in text to Unicode. Example: '<0xE0><0xA4><0xA8>' → 'न' (Devanagari NA) This is needed because the extended tokenizer's BPE merges create tokens whose names contain <0xNN> patterns. The standard HF decoder only handles individual <0xNN> tokens, not merged ones. réézutf-8Úreplace)ÚerrorsÚ) ÚlenÚ BYTE_PATTERNÚmatchÚappendÚintÚgroupÚendÚbytesÚdecodeÚjoin)rÚresultÚposÚmÚ byte_vals©rú@/home/ubuntu/training/cohere-transcribe-indic/tokenizer_utils.pyÚdecode_byte_patternss     ý õ rTcCs|j||d}t|ƒS)zŠDecode token IDs to Unicode text with byte-pattern fix. Use this instead of tokenizer.decode() for the extended Indic tokenizer. )Úskip_special_tokens)rr)Ú tokenizerÚidsrÚrawrrrÚ decode_tokens.sr Ú model_pathc Cs¾t|ƒ}|d}|d}| ¡std|›ƒ‚| ¡s"td|›ƒ‚t | ¡¡}tt|ƒ| d¡| d¡| d¡| d¡| d g¡d d }|d }| ¡r]t | ¡¡}| d |j ¡|_ |S)aWLoad the extended fast tokenizer without splitting control tokens. `from_pretrained()` currently honors `split_special_tokens=true` from the saved config, which breaks decoder prompts like `<|startofcontext|>...<|nodiarize|>` into dozens of subword pieces. For training we need those prompt/control tokens to stay atomic. ztokenizer.jsonzspecial_tokens_map.jsonzMissing tokenizer.json in z#Missing special_tokens_map.json in Ú bos_tokenÚ eos_tokenÚ pad_tokenÚ unk_tokenÚadditional_special_tokensF)Útokenizer_filer"r#r$r%r&Úsplit_special_tokensztokenizer_config.jsonÚmodel_max_length) rÚexistsÚFileNotFoundErrorÚjsonÚloadsÚ read_textrÚstrÚgetr))r!Ú model_dirr'Úspecial_tokens_fileÚspecialrÚtokenizer_configÚcfgrrrÚload_extended_tokenizer7s, ù r6)T) Ú__doc__r,ÚreÚpathlibrÚ transformersrÚcompiler r/rr r6rrrrÚs