o }oiS @sNddlZddlZddlZddlZddlZddlZddlmm Z ddl m Z ddl mZddlmZddlmZddlmZddlmZmZddlmZdd lmZGd d d eZ  d(dedededefddZ    d)dededededef ddZddZ de!ee"effdd Z#d!edefd"d#Z$d$ed%efd&d'Z%dS)*N)Instance)LM)tqdm) AutoTokenizer)SentencePieceTokenizer)EvaluationConfigEvaluationTarget) NemoQueryLLM)loggingcsreZdZdZfddZddedefddZd d Zd ee fd d Z d ee fddZ dee fddZ Z S) NeMoFWLMEvala] NeMoFWLMEval is a wrapper class subclassing lm_eval.api.model.LM class, that defines how lm_eval interfaces with NeMo model deployed on PyTriton server. Created based on: https://github.com/EleutherAI/lm-evaluation-harness/blob/v0.4.4/docs/model_guide.md This class is deprecated and not used for evaluation with nvidia-lm-eval c sTtjdtdd||_||_||_||_||_||_||_ ||_ | |_ t dS)NzNeMoFWLMEval is deprecated and will be removed in 25.06. Please refer to https://github.com/NVIDIA/NeMo/blob/main/docs/source/evaluation/evaluation_doc.rst and update your code accordingly) stacklevel)warningswarnDeprecationWarning model_nameapi_url tokenizer batch_sizemax_tokens_to_generate temperaturetop_ptop_kadd_bossuper__init__) selfrrrrrrrrr __class__X/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/evaluation/base.pyr,szNeMoFWLMEval.__init__Fsingle_prediction_token return_logitsc Cst|j|dd}d}d}|r|rd}nd}|jt|dtr#|dn|dg|d|d|d|d ||dd }|rM|rE|d d d S|d d dSt|d d dS)z A private method that sends post request to the model on PyTriton server and returns either generated text or logits. model)urlrFTprompt max_tokensrrr)promptsmax_output_lenrrroutput_context_logitsoutput_generation_logitsopenai_format_responsechoicesrcontext_logitsgeneration_logitstext)r r query_llm isinstanceliststr)rpayloadr!r"nqr)r*responserrr _generate_tokens_logitsAs, z$NeMoFWLMEval._generate_tokens_logitscCs$t|trdSt|trdStd)z4 Returns the type of the tokenizer. rrzTokenizer type is not one of SentencePieceTokenizer or HF's AutoTokenizer. Please check how to handle special tokens for this tokenizer)r1rr ValueError)rrrrr tokenizer_typeds  zNeMoFWLMEval.tokenizer_typerequestsc Csi}||j}|dkr|j|d<n |dkr|j|d<d}d}t||djr*d}d |_g}ttdt ||j D] }||||j }g} g} g} g} g} |D]f}|j d}|j d }|jjj |fi|}|jjj |fi|}||jdkr|d d }|d d }t |}t |}|| |jj|d d }| || || || || |qR|j| |j|j|j|jd }|j||dd}t|D]p\}}| |}| |}| |}|s|d d tj|dkddd d f}|d d | d d d f}tjt|d d}tj|tjdd}|jd d}||k}t |d|d !d }t"|#t$|f}||qq:|S)a9 Defines the loglikelihood request. Takes input requests of type list[Instance] where Instance is a dataclass defined in lm_eval.api.instance. Each Instance conists of the input prompt, output prompt, request type(here loglikelihood) and other relevant args like few shot samples. rrradd_special_tokensFz^mmlu_rTNr#r%r&rrr)r")rr )axis)dim)dtyper )%r9rrrematch task_namerrrangelenr argumentsencodereplacedecodeappendrrrrr7 enumeratenpanyF log_softmaxtorchtensorlong unsqueezeargmaxallgathersqueezefloatsumbool) rr:special_tokens_kwargsr9r!mmlu_regex_patternresultsibatchr' continuationscontinuation_encsnum_ctx_tokens_listnum_cont_tokens_listrequestcontext continuation context_enccontinuation_encnum_ctx_tokensnum_cont_tokensr%r4 logits_batchjlogitslogProbs cont_toks greedy_tokens is_greedylogProbs_actualresultrrr loglikelihoodrsv             $   zNeMoFWLMEval.loglikelihoodcCsdS)zX Defines the loglikelihood_rolling request type. Yet to be implemented. Nr)rr:rrr loglikelihood_rollingsz"NeMoFWLMEval.loglikelihood_rollinginputscCsPg}t|D]}|jd}|j||j|j|j|jd}||}||q|S)a? Defines the generate_until request type. Takes input requests of type list[Instance] where Instance is a dataclass defined in lm_eval.api.instance. Each Instance conists of the input prompt, output prompt, request type(here loglikelihood) and other relevant args like few shot samples. rr?) rrHrrrrrr7rL)rrxr_instancer%r4generated_textrrr generate_untils   zNeMoFWLMEval.generate_until)FF)__name__ __module__ __qualname____doc__rr\r7r9r2rrvrwr{ __classcell__rrrr r $s #hr http://0.0.0.0:8080 triton_modelXr base_urlr max_retriesretry_intervalc Cs$ddl}ddl}|d}|d}t|D]m}tdzL||} | jdkr8td| j||Wqtd|j||d d d d } | jdkr]td | j||Wqtd|dWdS|j j y}td|dYnw||qt d|d|ddS)a Wait for FastAPI server and model to be ready. Args: base_url (str): The URL to the FastAPI server (e.g., "http://0.0.0.0:8080"). model_name (str): The name of the deployed model. max_retries (int): Maximum number of retries before giving up. retry_interval (int): Time in seconds to wait between retries. Returns: bool: True if both the server and model are ready within the retries, False otherwise. rNz/v1/completions/z/v1/triton_health&Checking server and model readiness...'Server is not ready. HTTP status code: Server is ready.hellor<)r#r%r&)jsonz&Model is not ready. HTTP status code: Model ' ' is ready.T+Pytriton server not ready yet. Retrying in seconds...Server or model '' not ready after attempts.F) timer:rFr infoget status_codesleeppost exceptionsRequestExceptionerror) rrrrrr:completions_url health_url_r6rrr wait_for_fastapi_servers4           rhttp://0.0.0.0:8000@r$triton_http_portc Csddl}ddl}ddlm}ddlm}m} |dr8d} t | |} | d} | dd d | d |}|d } t |D]}t d zE|| }|jd krct d |j||WqAt d||||dt d|d WdWdS1swYWn6| yt d|dYn%|yt d|dYn|jjyt d|dYnw||qAt d|d|ddS)a Wait for PyTriton server and model to be ready. Args: url (str): The URL of the Triton server (e.g., "grpc://0.0.0.0:8001"). triton_http_port (int): http port of the triton server. model_name (str): The name of the deployed model. max_retries (int): Maximum number of retries before giving up. retry_interval (int): Time in seconds to wait between retries. Returns: bool: True if both the server and model are ready within the retries, False otherwise. rN) ModelClient)#PyTritonClientModelUnavailableErrorPyTritonClientTimeoutErrorzgrpc://z:(\d+)r<zhttp://:z/v2/health/readyrrrr)rinit_timeout_srrTzTimeout: Server or model 'z' not ready yet.z' is unavailable on the server.rrrrrF)rr:pytriton.clientrpytriton.client.exceptionsrr startswithrCsearchgrouprJrFr rrrrrrr)r$rrrrrr:rrrpatternrD grpc_portrrr6rrr wait_for_server_ready2sB            (   rcCst|j|jdS)N.)pkgutil iter_modules__path__r|)ns_pkgrrr _iter_namespacevsrreturncCszddl}Wn tytdwddt|D}i}|D] \}}|ds2td|d|^}}}t|||<q!|S) a Finds all pre-defined evaluation configs across all installed evaluation frameworks. Returns: dict[str, list[str]]: Dictionary of available evaluations, where key is evaluation framework and value is list of available tasks. rNz[Please ensure that core_evals is installed in your env as it is required to run evaluationscSs"i|] \}}}|tjd|dqS)z.input)package) importlib import_module).0findernameispkgrrr sz.list_available_evaluations.. core_evals.z Framework z! is not a submodule of core_evals) core_evals ImportErrorritemsr RuntimeErrorget_available_evaluationsr2keys)rdiscovered_modulesevalsframework_name input_modulertask_name_mappingrrr list_available_evaluations~s"   r eval_taskcspt}fdd|D}t|dkrtddt|dkr4dd|D}tdd |d |dS) z Find framework for executing the evaluation eval_task. This function serches for framework (module) that defines a task with given name and returns the framework name. csg|] \}}|vr|qSrr)rftasksrrr sz"find_framework..rzFramework for task z not found!r<cSs$g|]}|tddddqS)rNr-)rGrJ)rrrrr rs$z#Multiple frameworks found for task z: zL. Please indicate which version should be used by passing .)rrrGr8)rr frameworksframeworks_namesrrr find_frameworks  r target_cfgeval_cfgc Cs|jjdur tdzddlm}Wn tytdwddlm}|j}|j|jddd }t |j |j |j d }|sAt d |j}t|j |j ||j|j|j|j|j|jd }|j} |j|| |j|j|jd } td| d| dS)a Evaluates nemo model deployed on PyTriton server (via trtllm) using lm-evaluation-harness (https://github.com/EleutherAI/lm-evaluation-harness/tree/main). Args: target_cfg (EvaluationTarget): target of the evaluation. Providing nemo_checkpoint_path, model_id, and url in EvaluationTarget.api_endpoint is required to run evaluations. eval_cfg (EvaluationConfig): configuration for evaluations Nz7Please provide nemo_checkpoint_path in your target_cfg.r) evaluatorzfPlease ensure that lm-evaluation-harness is installed in your env as it is required to run evaluations)ioz/contextzmodel.tokenizer)subpath)r$rrzServer not ready for evaluation) rrrrrrrrr)r#rlimit num_fewshotbootstrap_iterszscore: r_) api_endpointnemo_checkpoint_pathr8lm_evalrrnemo.lightningr load_contextrr$nemo_triton_http_portmodel_idrparamsr rmax_new_tokensrrrrtypesimple_evaluate limit_samplesrrr r) rrrrendpointr server_readyrr#rr_rrr _legacy_evaluatesL     r)rrrr )rrrrr )&rrrCrnumpyrNrRtorch.nn.functionalnn functionalrPlm_eval.api.instancerlm_eval.api.modelrr=nemo.collections.common.tokenizers.huggingface.auto_tokenizerr:nemo.collections.common.tokenizers.sentencepiece_tokenizerr#nemo.collections.llm.evaluation.apirrnemo.deploy.nlpr nemo.utilsr r r3intrrrdictr2rrrrrrr sl       Y 8 D