o %ݫi@sdZddlZddlmZmZmZmZmZddlm Z ddl m Z e e Z zddlZWn ey5edwdedeefd d ZGd d d Zd eedddeefddZdddZGdddZdS)aLanguage model wrapper for kenlm n-gram. This file is based on the implementation of the kenLM wrapper from PyCTCDecode (see: https://github.com/kensho-technologies/pyctcdecode) and is used in CTC decoders. See: speechbrain.decoders.ctc.py Authors * Adel Moumen 2023 N) CollectionOptionalSetTuplecast)CharTrie) get_loggerzwkenlm python bindings are not installed. To install it use: pip install https://github.com/kpu/kenlm/archive/master.zip arpa_pathreturncCst}t|dd:}d}|D]-}|}|dkrd}n|dkr!n|r;t|dkr;|d}t|d kr;||d qWd n1sFwYt|dkrUtd |S) zRead unigrams from arpa file. Taken from: https://github.com/kensho-technologies/pyctcdecode Arguments --------- arpa_path : str Path to arpa file. Returns ------- unigrams : set Set of unigrams. zutf-8)encodingFz \1-grams:Tz \2-grams:r NzANo unigrams found in arpa file. Something is wrong with the file.)setopenstriplensplitadd ValueError)r unigramsf start_1_gramlinepartsrW/home/ubuntu/.local/lib/python3.10/site-packages/speechbrain/decoders/language_model.pyload_unigram_set_from_arpas(   rc@s(eZdZdZd ddZed ddZd S) KenlmStateaEWrapper for kenlm state. This is a wrapper for the kenlm state object. It is used to make sure that the state is not modified outside of the language model class. Taken from: https://github.com/kensho-technologies/pyctcdecode Arguments --------- state : kenlm.State Kenlm state object. state kenlm.StatecCs ||_dS)N_state)selfrrrr__init__Ps zKenlmState.__init__r cCs|jS)zGet the raw state object.r!r#rrrrSszKenlmState.stateN)rr r r )__name__ __module__ __qualname____doc__r$propertyrrrrrrBs  rr kenlm_model kenlm.Modelcs|t|dkrtdt|t|}tfdd|D}t|dkr%dnt|t|}|dkrrsz(_prepare_unigram_set..r?g?zOnly %s%% of unigrams in vocabulary found in kenlm model-- this might mean that your vocabulary and language model are incompatible. Is this intentional?dr)rloggerwarningrround)rr, unigram_setretained_fractionrr0r_prepare_unigram_setYs  r9r cCs&zt}W|Stytdw)zGet uninitialized kenlm state. Taken from: https://github.com/kensho-technologies/pyctcdecode Returns ------- kenlm_state : kenlm.State Empty kenlm state. z3To use a language model, you need to install kenlm.)kenlmState ImportErrorr) kenlm_staterrr_get_empty_lm_states   r>c@seZdZdZ     d!ddd eeed ed ed ed eddfddZ e de fddZ de fddZdddefddZdedefddZ d"dededeee ffdd ZdS)# LanguageModelaLanguage model container class to consolidate functionality. This class is a wrapper around the kenlm language model. It provides functionality to score tokens and to get the initial state. Taken from: https://github.com/kensho-technologies/pyctcdecode Arguments --------- kenlm_model : kenlm.Model Kenlm model. unigrams : list List of known word unigrams. alpha : float Weight for language model during shallow fusion. beta : float Weight for length score adjustment of during scoring. unk_score_offset : float Amount of log score offset for unknown tokens. score_boundary : bool Whether to have kenlm respect boundaries when scoring. N??$Tr,r-ralphabetaunk_score_offsetscore_boundaryr c Csb||_|durtdt}d}n t||j}t|}||_||_||_ ||_ ||_ ||_ dS)NzBNo known unigrams provided, decoding results might be a lot worse.) _kenlm_modelr4r5rr9rfromkeys _unigram_set _char_trierCrDrErF) r#r,rrCrDrErFr7 char_trierrrr$s    zLanguageModel.__init__cCstt|jjS)z+Get the order of the n-gram language model.)rintrGorderr%rrrrMszLanguageModel.ordercCs4t}|jr|j|t|S|j|t|S)zGet initial lm state.)r>rFrGBeginSentenceWriteNullContextWriter)r# start_staterrrget_start_states   zLanguageModel.get_start_staterPr cCs(|jrt}|j|d|}|Sd}|S)zCalculate final lm score.zg)rFr>rG BaseScore)r#rP end_statescorerrr_get_raw_end_scoresz LanguageModel._get_raw_end_score partial_tokencCsN|jdurd}n t|j|dk}|j|}t|dkr%|t|d}|S)zGet partial token score.Nr2r)rJrLhas_noderEr)r#rVis_oov unk_scorerrrscore_partial_tokens   z!LanguageModel.score_partial_tokenFword is_last_wordcCst|tstdt|t}|j|j||}t|j dkr&||j vs+||jvr0||j 7}|r9|| |}|j |dt t j|j}|t|fS)z&Score word conditional on start state.z7Wrong input state type found. Expected KenlmState, got rr2) isinstancerAssertionErrortyper>rGrRrrrIrErUrCmathlog10erD)r# prev_stater\r]rSlm_scorerrrrTs       zLanguageModel.score)Nr@rArBT)F)r'r(r)r*rrstrfloatboolr$r+rLrMrrQrUr[rrTrrrrr?sF      r?r&)r*ratypingrrrrrpygtrierspeechbrain.utils.loggerrr'r4r:r<rfrrr9r>r?rrrrs0     # &