o 5ti0G@s ddlZddlZddlZddlZddlZddlmZddlmZm Z m Z m Z m Z m Z mZmZddlmZddlmZddlmZddlmZddlmZdd lmZeeZGd d d e Zd ed dfddZ eddd e e!e!ffddZ"edGdddeZ#dS)N) lru_cache)AnyDictList NamedTupleOptionalTupleTypecast)tqdm)Instance)LM)register_model) JsonChatStr)simple_parse_args_stringc@seZdZUeed<eed<dS)LogLikelihoodResultlog_likelihood is_greedyN)__name__ __module__ __qualname__float__annotations__boolrrQ/home/ubuntu/.local/lib/python3.10/site-packages/lm_eval/models/ibm_watsonx_ai.pyrs  rcredsreturncsdddddddd}tfd d d Dpd }d vo!d }tfdd dD}|r3|r3|srg}|sB|d|d d|sO|d|d d|sa|d|dd|dddd|d}|d7}t|dS)z Validate credentials for APIClient authentication. Required conditions: - Either ("username" and "password") or "apikey" must be present. - "url" is mandatory. - Either "project_id" or "space_id" must be present. WATSONX_API_KEY WATSONX_TOKEN WATSONX_URLWATSONX_PROJECT_IDWATSONX_SPACE_IDWATSONX_USERNAMEWATSONX_PASSWORD)apikeytokenurl project_idspace_idusernamepasswordc3|]}|VqdSNget.0keyrrr .z&_verify_credentials..)r*r+r%r'c3r,r-r.r0r3rrr44r5)r(r)z0either ('username' and 'password') or 'apikey' ()zurl (zeither 'project_id' (r(z) or 'space_id' (r)zMissing required credentials: z, z. z>Please set the environment variables indicated in parentheses.N)allr/anyappendjoin ValueError)r env_var_maphas_authhas_urlhas_project_or_space_id missing_keys error_msgrr3r_verify_credentialss:   rB)maxsizec szddlm}Wn tytdw|tddtddtddtddtd dtd dtd dd d dvrGdd<tfdddDrWtdtS)a Retrieves Watsonx API credentials from environmental variables. Returns: Dict[str, str]: A dictionary containing the credentials necessary for authentication, including keys such as `apikey` or `token`, `url`, and `project_id`. Raises: AssertionError: If the credentials format is invalid or any of the necessary credentials are missing. r load_dotenvzHCould not import dotenv: Please install lm_eval[ibm_watsonx_ai] package.r#Nr$rrr r!r")r*r+r%r&r'r(r)z cloud.ibm.comr' openshift instance_idc3r,r-r.r0 credentialsrrr4ir5z*get_watsonx_credentials..)r*r+r%zYou're passing `username`, `password`, and `apikey` at the same time, which might cause issues. More info on authentication in different scenarios can be found in the docs: https://ibm.github.io/watsonx-ai-python-sdk/setup_cpd.html) dotenvrE ImportErrorosgetenvr7warningswarnrBrDrrHrget_watsonx_credentialsHs0          rP watsonx_llmc sReZdZdZe d"deddedeeddfddZ d"d ed eee e fddffd d Z e d e ede edefddZddZde eeefde eeefdefddZde ede efddZde ede eeeffddZde eeeffddZedefddZde eeefde eeeffd d!ZZS)# WatsonxLLMz Implementation of LM model interface for evaluating Watsonx model with the lm_eval framework. See https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/model_guide.md for reference. Ncls arg_stringadditional_configrcCs~zddlm}Wn tytdwt|}|||dd}|dd}|dur5|dur5td|ddsKd|d <d|d <d|d <d|d <|j|ddsUd nd|j |dd|j |d d|j |d d|j |d d|j |d d|j|dd|j|dd|j|dd|j|dd|j|dd|j|dd|jdddddi }dd|D}|t|||dS)zi Allow the user to specify model parameters (TextGenerationParameters) in CLI arguments. rGenTextParamsMetaNamesPCould not import ibm_watsonx_ai: Please install lm_eval[ibm_watsonx_ai] package.model_idN deployment_idzI'model_id' or 'deployment_id' is required, please pass it in 'model_args' do_sample temperaturetop_ptop_kseedgreedysamplelength_penaltyrepetition_penaltymin_new_tokensmax_new_tokensstop_sequences time_limittruncate_input_tokensT)generated_tokens input_tokenstoken_logprobs token_rankscSsi|] \}}|dur||qSr-r)r1kvrrr sz5WatsonxLLM.create_from_arg_string..)watsonx_credentialsrYrZgenerate_params)ibm_watsonx_ai.metanamesrWrKrupdatepopr;r/DECODING_METHODLENGTH_PENALTY TEMPERATURETOP_PTOP_K RANDOM_SEEDREPETITION_PENALTYMIN_NEW_TOKENSMAX_NEW_TOKENSSTOP_SEQUENCES TIME_LIMITTRUNCATE_INPUT_TOKENSRETURN_OPTIONSitemsrP)rSrTrU GenParamsargsrYrZrrrrrcreate_from_arg_stringzs^      z!WatsonxLLM.create_from_arg_stringrqrrc szddlm}ddlm}Wn tytdwt||}|dd}|j |||_ |||||d|_ ||_ dS)Nr) APIClient)ModelInferencerXr()rYrZ api_clientr() ibm_watsonx_air ibm_watsonx_ai.foundation_modelsrrKsuper__init__r/setdefault_projectrrmodel _model_id) selfrqrYrZrrrrclientr( __class__rrrs(      zWatsonxLLM.__init__response_tokenscontext_tokenscCsPt|}|d|d|ddkr|d|dkStd|d|d|)a7 Determines whether a stop token has been generated in the `response_tokens` compared to the `context_tokens`. If the tokens do not match as expected, the function raises a RuntimeError, indicating a possible misalignment between the tokens generated by the tokenizer and the model. Args: response_tokens (List[str]): The List of tokens generated as a response by the model. context_tokens (List[str]): The List of tokens representing the input context. Returns: bool: True if the `response_tokens` likely contain a stop token that terminates the sequence, otherwise raises an exception. Raises: RuntimeError: If there is an unexpected mismatch between the `response_tokens` and the `context_tokens`. NzUThere is an unexpected difference between tokenizer and model tokens: context_tokens=z response_tokens=)len RuntimeError)rrcontext_lengthrrr_has_stop_tokens zWatsonxLLM._has_stop_tokencCsN|jjdg|jddddd}tdd|dDr%td |jd d S) a Verifies if the model supports returning log probabilities for input tokens. This function sends a prompt to the model and checks whether the model's response includes log probabilities for the input tokens. If log probabilities are not present, it raises a `RuntimeError`, indicating that the model is not supported. Raises: RuntimeError: If the model does not return log probabilities for input tokens. zThe best ice cream flavor is:Tpromptparams raw_responserresultscss |] }|ddduVqdS)logprobNr.r1r&rrrr4sz;WatsonxLLM._check_model_logprobs_support..rkzModel z< is not supported: does not return logprobs for input tokensN)r generate_textrrr7rr)rtokensrrr_check_model_logprobs_supports   z(WatsonxLLM._check_model_logprobs_supportrkcCsbdd|D}t|}|||r|d8}ttdd||dDtdd||dDdS) aK Calculates the log likelihood of the generated tokens compared to the context tokens. Args: input_tokens (List[Dict[str, float]]): A List of token dictionaries, each containing token information like `text` and `logprob`. context_tokens (List[Dict[str, float]]): A List of token dictionaries representing the input context. Returns: LogLikelihoodResult: An object containing the calculated log likelihood and a boolean flag indicating if the tokens were generated greedily. cSsg|]}|dqS)textrrrrr sz2WatsonxLLM._get_log_likelihood..rcss|] }|ddVqdS)rrNr.rrrrr4  z1WatsonxLLM._get_log_likelihood..Ncss|] }|ddkVqdS)rankrNrrrrrr4r)rr)rrrsumr7)rrkrrrrrr_get_log_likelihoods   zWatsonxLLM._get_log_likelihoodrequestsc Csdd|D}g}t|ddD]Q}|\}}z(t|tr4t|j}|j||j}|dddd}n|j ||j}Wnt yP}zt d |d }~ww| ||jd ||f|q|S) a% Generates text responses for a List of requests, with progress tracking and caching. Args: requests (List[Instance]): A List of instances, each containing a text input to be processed. Returns: List[str]: A List of generated responses. cSg|]}|jqSrrr1requestrrrr*z-WatsonxLLM.generate_until..z#Running generate_until function ...descchoicesrmessagecontentzError while generating text.Ngenerate_until)r isinstancerjsonloadsrrchatrrr Exception eval_loggererrorr9 cache_hook add_partial)rrrrcontext continuationresponseexprrrr"s0      zWatsonxLLM.generate_untilc CsPzddlm}Wn tytdw|t|j}d||j<dd|D}g}t|ddD]j}|\}}z|jj |d d d d }Wnt yY} zt d | d} ~ ww||} z |jj | |d d} Wnt y|} zt d| d} ~ ww|| ddd|} || |jd||f| j| jfq2tttttf|S)a Args: requests: Each request contains Instance.args : Tuple[str, str] containing: 1. an input string to the LM and 2. a target string on which the loglikelihood of the LM producing this target, conditioned on the input, will be returned. Returns: Tuple (loglikelihood, is_greedy) for each request according to the input order: loglikelihood: probability of generating the target string conditioned on the input is_greedy: True if and only if the target string would be generated by greedy sampling from the LM rrVrXrcSrrrrrrrrZrz,WatsonxLLM.loglikelihood..z"Running loglikelihood function ...rT)r return_tokensresultrzError while model tokenize.Nr Error while model generate text.rrk loglikelihood)rsrWrKrcopyrrr~r rtokenizerrrrrr9rrrrr rrrr) rrrrrrrrrtokenized_contextr input_promptrlog_likelihood_responserrrrDsh          zWatsonxLLM.loglikelihoodc Cszddlm}Wn tytdw|t|j}d||j<dd|D}g}t|ddD]A}|\}}z |j j ||d d }Wnt yV} zt d | d } ~ ww||d ddg} || |jd||f| jq2tttttf|S)a Used to evaluate perplexity on a data distribution. Args: requests: Each request contains Instance.args : Tuple[str] containing an input string to the model whose entire loglikelihood, conditioned on purely the EOT token, will be calculated. Returns: Tuple (loglikelihood,) for each request according to the input order: loglikelihood: solely the probability of producing each piece of text given no starting input. rrVrXrcSrrrrrrrrrz4WatsonxLLM.loglikelihood_rolling..z*Running loglikelihood_rolling function ...rTrrNrrkloglikelihood_rolling)rsrWrKrrdeepcopyrrr~r rrrrrrr9rrrr rrrr) rrrrrrrrrrrrrrrrsH        z WatsonxLLM.loglikelihood_rollingcCsdS)Nr)rrrrtokenizer_nameszWatsonxLLM.tokenizer_name chat_historycCstt|Sr-)rrdumps)rrrrrapply_chat_templateszWatsonxLLM.apply_chat_templater-)rrr__doc__ classmethodr strrrrrr staticmethodrrrrrrrr rrrrpropertyrr __classcell__rrrrrRssR D   ""@1rR)$rrloggingrLrN functoolsrtypingrrrrrrr r r lm_eval.api.instancer lm_eval.api.modelr lm_eval.api.registryrlm_eval.models.api_modelsr lm_eval.utilsr getLoggerrrrdictrBrrPrRrrrrs( (       .*