o qia @sdZddlmZddlmZddlZddlZddlm Z de dee e ffdd Z d e d e d e dee e ffd dZddefddZde de fddZde de ddfddZdS)zx keras_tokenizer module ------------------------------- This module create tokens using a pre-trained sequence model . )Path)TupleN)ndarray vocab_pathreturncCsnt|ddd }|}Wdn1swYtd|}ddt|D}ddt|D}||fS) z Maps characters to integers and vice versa Args: vocab_path (str): path to the vocab file Returns: Two dictionaries containing character to integer mapping and integer to character mapping rzUTF-8)encodingN_cSsi|]\}}||qSr .0idxcharr r Y/home/ubuntu/.local/lib/python3.10/site-packages/urduhack/tokenization/keras_tokenizer.py z_load_vocab..cSsi|]\}}||qSr r r r r rrr)openreadlinelist enumerate)r vocab_filevocabchar2idxidx2charr r r _load_vocabs  rsentencermax_lencCstjt||ftd}tt||f}d}|D](}|dkr'd|d|df<q||vr@|||d|f<|d7}||kr@||fSq||fS)aR Makes the input and output arrays for the data explaining where is a character or a space Args: sentence (str): Sentence to be tokenized char2idx (dict): Dict containing character to integer mapping max_len (int): integer Returns: Input and Output arrays representing features and labels )dtyper )npzeroslenint)rrrinput_output_ char_indexletterr r r_preprocess_sentence!s r(?c Csx|dk}||}||}d}g}t|jdD]} || } | dkr'||| 7}|| |kr4||d}q|||S)a Retrieve the original words from predicted and actual arrays as per char2idx mapping Args: features (array): Input array labels (array): Output array idx2char (dict): Dict mapping integer to character thresh (float): Confidence to tell whether prediction is a character or space Returns: list : Containing ``urdu`` word tokens r)rangeshapeappend) featureslabelsrthreshmasklettersspacesfinaltokensr'r r r r_retrieve_words;s     r6 model_pathcCs$tjj|}t|\}}|||fS)z Loads pre_trained keras model and vocab file Args: model_path (str): Path to the model file vocab_path (str): Path to the vocab file Returns: tuple: contains object )tfkerasmodels load_modelr)r7rmodel_ char2idx_ idx2char_r r r _load_modelWs  r?cCs,t|st|sd}t|dSdS)a Check if the models file exist. Args: model_path (str): path to the tokenizer model file vocab_path (str): Path to the tokenizer vocab file Raises: FileNotFoundError: If model_path does not exist Returns: None zWord tokenizer Model not found!Please run 'urduhack download' in terminal.Doc: https://urduhack.readthedocs.io/en/stable/installation.html#downloading-modelsN)rexistsFileNotFoundError)r7r_errorr r r_is_model_existfs rC)r))__doc__pathlibrtypingrnumpyr tensorflowr8rstrdictrr#r(rr6r?rCr r r rs   "