o 5ti[@sddlZddlZddlZddlZddlZddlmZddlmZm Z m Z m Z ddl m Z ddl mZer@ddlmZddlmZeeZe dd d ZGd d d ejZd ed ee defddZGdddZGdddZGdddeZdS)N)Iterable) TYPE_CHECKINGAnyOptionalTypeVar)tqdm)utils SqliteDict)InstanceTLM)boundc @s,eZdZd%ddZejdeeee ffddZ ejdeefddZ ejdee fd d Z d&d eee e fde fd dZe d'deede dedBdefddZe d'deedededBdefddZeddZeddZede fddZd(de e Bde dBfdd Zd)d#d$ZdS)*r returnNcCsd|_d|_td|_dS)zDefines the interface that should be implemented by all LM subclasses. LMs are assumed to take text (strings) as input and yield strings as output (inputs/outputs should be tokenization-agnostic.) rN)_rank _world_size CacheHook cache_hookselfrE/home/ubuntu/.local/lib/python3.10/site-packages/lm_eval/api/model.py__init__sz LM.__init__cCdS)aCompute log-likelihood of generating a continuation from a context. Downstream tasks should attempt to use loglikelihood instead of other LM calls whenever possible. :param requests: list[Instance] A list of Instance objects, with property `args` which returns a tuple (context, continuation). `context: str` Context string. Implementations of LM must be able to handle an empty context string. `continuation: str` The continuation over which log likelihood will be calculated. If there is a word boundary, the space should be in the continuation. For example, context="hello" continuation=" world" is correct. :return: list[tuple[float, bool]] A list of pairs (logprob, isgreedy) `logprob: float` The log probability of `continuation`. `isgreedy`: Whether `continuation` would be generated by greedy sampling from `context`. Nrrrequestsrrr loglikelihood%szLM.loglikelihoodcCr)aCompute full log-likelihood of a string, with no truncation, for perplexity computation - We will use the full max context length of the model. - For inputs that exceed the max context length, we divide the tokenized string into chunks of up to the max context length. - IMPORTANT: Each document's loglikelihood/perplexity is computed *separately*, unlike other implementations which may simply concatenate multiple documents together. - IMPORTANT: We maximize the amount of context for each prediction. Specifically, for inputs that we break into multiple chunks, the last input will still a full-sized context. Example: Input tokens: [ 0 1 2 3 4 5 6 7 8 9 ] Prefix: BOS/EOS Max context length: 4 Resulting input/prediction pairs: INPUT: BOS 0 1 2 PRED: 0 1 2 3 INPUT: 3 4 5 6 PRED: 4 5 6 7 INPUT: 5 6 7 8 PRED: 8 9 Observe that: 1. Each token is predicted exactly once 2. For the last pair, we provide the full context, but only score the last two tokens :param requests: list[Instance] A list of Instance objects with property `args` which returns a tuple (context,). string: str String for which we are computing overall loglikelihood :return: list[tuple[float]] A list of tuples (logprob,) logprob: float The log probability of `context` conditioned on the BOS/EOS token. Can also be overridden for custom cases by `prefix_token_id`. Nrrrrrloglikelihood_rolling>s(zLM.loglikelihood_rollingcCr)a"Generate greedily until a stopping sequence :param requests: list[Instance] A list of Instance objects with property `args` which returns a tuple (context, gen_kwargs). context: str Context string gen_kwargs: dict A dictionary of keyword arguments to pass to the generation function e.g. top_k, until, etc. :return: list[str] A list of model generated continuations. continuation: str The generated continuation. Nrrrrrgenerate_untiliszLM.generate_untilT chat_historycCtd)a Defines how to transform few-shot examples provided as chat history into a format that can be used as input to the LM. :param chat_history: list[dict[str, str]] A list of dictionaries with keys 'role' and 'content'. Values are strings representing the role name and the content of the message, respectively. :param add_generation_prompt: bool Whether to append an assistant gen prefix (for e.g. <|assistant|>) to the assistant messages in the chat history. False if prefilling an assistant message. :return: str A string representing the chat history in a format that can be used as input to the LM. zmTo use this model with chat templates, please implement the 'apply_chat_template' method for your model type.NotImplementedError)rr add_generation_promptrrrapply_chat_templatezszLM.apply_chat_templatecls arg_stringadditional_configcCs>|durin|}t|}dd|D}|di||S)as Creates an instance of the LM class using the given argument string and additional config. Parameters: - arg_string: A string containing arguments in the format key1=value1,key2=value2. - additional_config: Optional dictionary containing additional configuration parameters. Returns: - Instance of the LM class. NcSi|] \}}|dur||qSNr.0kvrrr z-LM.create_from_arg_string..r)rsimple_parse_args_stringitems)r&r'r(argsargs2rrrcreate_from_arg_strings zLM.create_from_arg_stringarg_dictcCs0|durindd|D}|di||S)aO Creates an instance of the LM class using the given arg_obj Parameters: - arg_obj: A dict containing arguments in the format key1=value1,key2=value2. - additional_config: Optional dictionary containing additional configuration parameters. Returns: - Instance of the LM class. NcSr)r*rr+rrrr/r0z*LM.create_from_arg_obj..r)r2)r&r6r(rrrcreate_from_arg_objs zLM.create_from_arg_objcC|jSr*)rrrrrrankzLM.rankcCr8r*)rrrrr world_sizer:z LM.world_sizecCr!)aMust be defined for LM subclasses which implement Chat Templating. Should return the name of the tokenizer or chat template used. Used only to properly fingerprint caches when requests are being cached with `--cache_requests`, otherwise not used. zVTo use this model with chat templates, please implement the 'tokenizer_name' property.r"rrrrtokenizer_nameszLM.tokenizer_nameF chat_templatecCr)a)Returns the chat template structure for user/assistant messages if a template is provided. This method is intended to be overridden in a subclass to define a specific chat template format. For models that do not support chat templates, this method returns None by default. r)rr=rrrr=szLM.chat_templaterrcCs ||_dSr*)r)rrrrrset_cache_hooks zLM.set_cache_hook)rN)Tr*F)rrrN)__name__ __module__ __qualname__rabcabstractmethodlisttuplefloatboolrrstrrdictr% classmethodtyper r5r7propertyr9r;r<r=r?rrrrr sX  *      attrr3rcCs(t|gt|}t|dS)Nzutf-8)jsondumpsrFhashlibsha256encode hexdigest)rOr3datrrr hash_argssrWc@s@eZdZdedddfddZdedeed eddfd d ZdS) r cachinglm CachingLMrNcCs|dur d|_dS|j|_dSr*)dbdict)rrXrrrrs zCacheHook.__init__rOreqrescCs&|jdurdSt||}||j|<dSr*)rZrW)rrOr[r\hshrrr add_partials  zCacheHook.add_partial) rArBrCrrrJrrr^rrrrrs"rc@s>eZdZdededdfddZdedefdd Zd d d ZdS)rYlmcache_dbrNcCsZddlm}||_||_tj|rtjtj|dd||dd|_| | dS)zLM wrapper that returns cached results if they exist, and uses the underlying LM if not. :param lm: LM Underlying LM :param cache_db: str Path to cache db rr T)exist_ok) autocommitN) sqlitedictr r_r`ospathdirnamemakedirsrZr?get_cache_hook)rr_r`r rrrrs  zCachingLM.__init__rOcsNtj}dvrtdd|Sdtddtdffdd }|S) N)rrrzPassing through attribute 'z' to underlying LMrr rc szg}g}d}tddjdt|ddD]P}t|j}dkrI|jdd drI|s>td |jdd d }|d||q|j vr_j |}|dusYJ||q|d||qtd t |t |dt ||rt j |ng}d}t ||d dD]#\}} ||dur|d7}||dus| ||<t|j}| j |<qj |S)NFz Loading 'z' responses from cache 'z' where possible...zChecking cached requests)descrr do_samplez"Arguments to lm.generate_until() 'zV' include non-deterministic sampling. Caching will not be performed for such requests.TzCached requests: z, Requests remaining: r)strict) eval_loggerinfor`rrWr3getwarningappendrZlengetattrr_zipcommit) rr\remaining_reqswarnedr[r]obrem_resresptrrrOrrr_fn sH              z"CachingLM.__getattr__.._fn)rrr_rldebugrF)rrOlm_attrr|rr{r __getattr__s  2zCachingLM.__getattr__rcCst|Sr*)rrrrrrh=szCachingLM.get_cache_hook)rr) rArBrCr rJrrrrhrrrrrYs:rYc @sPeZdZdZdZdZeejde fddZ eddZ ej d d e d e dBdee fd d Zejd eeee e fee ee fdeeee ffddZde de deee ee ffddZ d!d edde deeee ffddZej d!de deefddZejd!de dee fddZd!de e Bde dBfddZdS)" TemplateLMz} A class acting as intermediary between the LM base class and boilerplate often included in other LM subclasses. NcausalrcCdSr*rrrrr eot_token_idJszTemplateLM.eot_token_idcCr8r*)rrrrrprefix_token_idOszTemplateLM.prefix_token_idstringadd_special_tokenscKr)a Tokenize a string using the model's tokenizer and return a list of token IDs. NOTE: This method is expected to handle strings which already contain the BOS token (when add_special_tokens=None). Otherwise, will use add_special_tokens if specified. Nr)rrrkwargsrrr tok_encodeTs zTemplateLM.tok_encodercKrr*r)rrrrrr_loglikelihood_tokens_z TemplateLM._loglikelihood_tokenscontext continuationcCs|sJdt|t|}|dkr$|| d|}|d| }|jdkrC|||}||}t|}||d}||fS||}|j|dd}||fS)a Encode a context-continuation pair into separate token ID lists. This method handles the tokenization of context and continuation strings while preserving proper boundary handling. Trailing spaces in the context are moved to the beginning of the continuation to ensure correct tokenization at the word boundary. For Seq2Seq models (encoder-decoder), context and continuation are encoded separately. For other model types (decoder-only), the full sequence is encoded together to ensure proper tokenization, then split at the context boundary. :param context: str The context string. Can be empty (will be handled by the caller). :param continuation: str The continuation string to be scored. :return: tuple[list[int], list[int]] A tuple of (context_enc, continuation_enc) where: - context_enc: Token IDs for the context - continuation_enc: Token IDs for the continuation Note: This method does NOT handle empty context. The caller should handle empty context (see loglikelihood method). zContext cannot be empty!rNrFr)rqrstripbackendr)rrrn_spaces whole_enc context_enccontext_enc_lencontinuation_encrrr _encode_paires     zTemplateLM._encode_pairFr disable_tqdmcCsg}dd|DD]=\}}|dkr4|j|dd}|j|dkr%|jg|fn |dd|ddf\}}n|||\}}|||f||fq |j||d S) a Compute log-likelihood of generating continuations from contexts. This is the concrete implementation for TemplateLM and its subclasses. It tokenizes context-continuation pairs and delegates scoring to _loglikelihood_tokens. **IMPORTANT**: This method is expected to handle empty context strings. When context is empty (""), it uses the model's prefix_token_id (typically BOS or EOS token) as context. If the continuation already starts with the prefix token, it reuses that token as context instead of duplicating it. :param requests: list[Instance] List of Instance objects with property `args` returning (context, continuation) tuples. :param disable_tqdm: bool Whether to disable the progress bar in _loglikelihood_tokens. :return: list[tuple[float, bool]] List of (log_prob, is_greedy) tuples for each request. Implementation details: - Empty context: Uses prefix_token_id (BOS/EOS) as context - Non-empty context: Uses _encode_pair for proper tokenization - Avoids token duplication when continuation starts with prefix_token_id cSsg|]}|jqSr)r3)r,r[rrr sz,TemplateLM.loglikelihood..r>FrrNr)r)rrrrpr)rrrnew_reqsrrrrrrrrs zTemplateLM.loglikelihoodcCrr*rrrrrrrrrz TemplateLM.loglikelihood_rollingcCrr*rrrrrrszTemplateLM.generate_untilr=cCs4|jdurdS|dus|durtddSt|trd}d}z |jjp'|jj}Wn ty3YdSwt|trv|jjdu}|dur_||vrP||}|rOd}nAt d|dt | dd |vrj|d }d}n't d t | dt|t rtd |jjdur|jj}n|jj}d}|rtd |S) a Set and get the appropriate chat template for the model. This method sets the tokenizer's chat_template and returns the template string for reproducibility. The template selection logic is adapted from the Transformers library's `apply_chat_template` method in the Tokenizer class. The original implementation can be found at: https://github.com/huggingface/transformers/blob/fc35907f95459d7a6c5281dfadd680b6f7b620e3/src/transformers/tokenization_utils_base.py#L1687 This method ensures that the right template is chosen based on the following: 0. If the model has no 'tokenizer' attribute: assumes that there is only a single possible chat template, handled on the model provider side internally. Returns the empty string. 1. If the model's tokenizer has multiple templates: a. Use the specified template if it exists in the dictionary. b. Use the default template from the list if no specific template is provided. c. Raise an error if no default template exists and no specific template is provided. 2. If the model's tokenizer has a single template or no template: a. Use the tokenizer's chat template if available. b. Fall back to the default chat template if no tokenizer chat template exists. Args: chat_template (Union[bool, str]): Specifies the chat template to use. - If False or None, no template is applied. - If True, the default or only available template is used. - If a string, the template with the matching name is used. Returns: Optional[str]: The selected chat template, or None if no template is applied. Nr>Fzmodel.chat_template was called with the chat_template set to False or None. Therefore no chat template will be applied. Make sure this is an intended behavior.TzThe specified chat template 'z1' is not available. Available template names are .defaultzThis model has multiple chat templates with no default specified! Please either pass a chat template or the name of the template you wish to use to the `chat_template` argument. Available template names are zChat template name provided, but the tokenizer's chat template is not a dictionary. Using the tokenizer's chat template or the default template instead.aNo chat template is set for this tokenizer, falling back to a default class-level template. This is very error-prone, because models are often trained with templates different from the class default! Default chat templates are a legacy feature and will be removed in Transformers v4.43, at which point any code depending on them will stop working. We recommend setting a valid chat template before then to ensure that this model continues working without issues.) tokenizerrlro isinstancerIr=default_chat_templateAttributeErrorrK ValueErrorsortedkeysrJ)rr=using_default_templatetemplateusing_default_dictselected_templaterrrr=sd          zTemplateLM.chat_templater*r@)rArBrC__doc__rrrNrDrEintrrrJrIrFrrGrHrrrrrr=rrrrrAsb      2 0  r)rDrRrPloggingrdcollections.abcrtypingrrrrrlm_evalrrcr lm_eval.api.instancer getLoggerrArlr ABCr rJrWrrYrrrrrs(       CR