o oi@sdZddlZddlmZmZmZmZmZddlm Z ddl m Z e e Z zddlZWn ey5edwdedeefd d ZGd d d Zd eedddeefddZdddZGdddZdS)aLanguage model wrapper for kenlm n-gram. This file is based on the implementation of the kenLM wrapper from PyCTCDecode (see: https://github.com/kensho-technologies/pyctcdecode) and is used in CTC decoders. See: speechbrain.decoders.ctc.py Authors * Adel Moumen 2023 N) CollectionOptionalSetTuplecast)CharTrie) get_loggerzwkenlm python bindings are not installed. To install it use: pip install https://github.com/kpu/kenlm/archive/master.zip arpa_pathreturncCst}t|:}d}|D]-}|}|dkrd}n|dkrn|r9t|dkr9|d}t|dkr9||dq Wd n1sDwYt|dkrStd |S) zRead unigrams from arpa file. Taken from: https://github.com/kensho-technologies/pyctcdecode Arguments --------- arpa_path : str Path to arpa file. Returns ------- unigrams : set Set of unigrams. Fz \1-grams:Tz \2-grams:r NzANo unigrams found in arpa file. Something is wrong with the file.)setopenstriplensplitadd ValueError)r unigramsf start_1_gramlinepartsra/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/speechbrain/decoders/language_model.pyload_unigram_set_from_arpas(    rc@s(eZdZdZd ddZed ddZd S) KenlmStateaEWrapper for kenlm state. This is a wrapper for the kenlm state object. It is used to make sure that the state is not modified outside of the language model class. Taken from: https://github.com/kensho-technologies/pyctcdecode Arguments --------- state : kenlm.State Kenlm state object. state kenlm.StatecCs ||_dS)N_state)selfrrrr__init__Ps zKenlmState.__init__r cCs|jS)zGet the raw state object.r r"rrrrSszKenlmState.stateN)rrr r)__name__ __module__ __qualname____doc__r#propertyrrrrrrBs  rr kenlm_model kenlm.Modelcs|t|dkrtdt|t|}tfdd|D}t|dkr%dnt|t|}|dkrrsz(_prepare_unigram_set..r?g?zOnly %s%% of unigrams in vocabulary found in kenlm model-- this might mean that your vocabulary and language model are incompatible. Is this intentional?dr )rloggerwarningrround)rr+ unigram_setretained_fractionrr/r_prepare_unigram_setYs  r8rcCs&zt}W|Stytdw)zGet uninitialized kenlm state. Taken from: https://github.com/kensho-technologies/pyctcdecode Returns ------- kenlm_state : kenlm.State Empty kenlm state. z3To use a language model, you need to install kenlm.)kenlmState ImportErrorr) kenlm_staterrr_get_empty_lm_states   r=c@seZdZdZ     d!ddd eeed ed ed ed eddfddZ e de fddZ de fddZdddefddZdedefddZ d"dededeee ffdd ZdS)# LanguageModelaLanguage model container class to consolidate functionality. This class is a wrapper around the kenlm language model. It provides functionality to score tokens and to get the initial state. Taken from: https://github.com/kensho-technologies/pyctcdecode Arguments --------- kenlm_model : kenlm.Model Kenlm model. unigrams : list List of known word unigrams. alpha : float Weight for language model during shallow fusion. beta : float Weight for length score adjustment of during scoring. unk_score_offset : float Amount of log score offset for unknown tokens. score_boundary : bool Whether to have kenlm respect boundaries when scoring. N??$Tr+r,ralphabetaunk_score_offsetscore_boundaryr c Csb||_|durtdt}d}n t||j}t|}||_||_||_ ||_ ||_ ||_ dS)NzBNo known unigrams provided, decoding results might be a lot worse.) _kenlm_modelr3r4rr8rfromkeys _unigram_set _char_trierBrCrDrE) r"r+rrBrCrDrEr6 char_trierrrr#s    zLanguageModel.__init__cCstt|jjS)z+Get the order of the n-gram language model.)rintrForderr$rrrrLszLanguageModel.ordercCs4t}|jr|j|t|S|j|t|S)zGet initial lm state.)r=rErFBeginSentenceWriteNullContextWriter)r" start_staterrrget_start_states   zLanguageModel.get_start_staterOrcCs(|jrt}|j|d|}|Sd}|S)zCalculate final lm score.zg)rEr=rF BaseScore)r"rO end_statescorerrr_get_raw_end_scoresz LanguageModel._get_raw_end_score partial_tokencCsN|jdurd}n t|j|dk}|j|}t|dkr%|t|d}|S)zGet partial token score.Nr1r)rIrKhas_noderDr)r"rUis_oov unk_scorerrrscore_partial_tokens   z!LanguageModel.score_partial_tokenFword is_last_wordcCst|tstdt|t}|j|j||}t|j dkr&||j vs+||jvr0||j 7}|r9|| |}|j |dt t j|j}|t|fS)z&Score word conditional on start state.z7Wrong input state type found. Expected KenlmState, got rr1) isinstancerAssertionErrortyper=rFrQrrrHrDrTrBmathlog10erC)r" prev_stater[r\rRlm_scorerrrrSs       zLanguageModel.score)Nr?r@rAT)F)r&r'r(r)rrstrfloatboolr#r*rKrLrrPrTrZrrSrrrrr>sF      r>r%)r)r`typingrrrrrpygtrierspeechbrain.utils.loggerrr&r3r9r;rerrr8r=r>rrrrs0     # &