o }oi@sddlmZmZmZmZddlmZmZGdddeZGdddeZ GdddeZ Gd d d eZ Gd d d eZ Gd dde ZdS))AnyDictOptionalUnion) BaseModelFieldc@seZdZUdZedddZeed<edddZeed<ed d dZ eed <ed d ddZ e eed<eddddZ e e ed<d S) ApiEndpointzG Represents evaluation Standard API target.api_endpoint object zUrl of the modelz#http://0.0.0.0:8080/v1/completions/ descriptiondefaulturlzName of the model in API triton_modelmodel_idz,The type of the target (chat or completions) completionstypezPath for nemo 2.0 checkpointNTThis parameter is deprecated and not used for evaluations with NVIDIA Evals Factory.r r deprecatednemo_checkpoint_pathzTHTTP port that was used for the PyTriton server in the deploy method. Default: 8000.i@nemo_triton_http_port)__name__ __module__ __qualname____doc__rr str__annotations__rrrrrintrrW/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/evaluation/api.pyrs rc@s$eZdZUdZeddZeed<dS)EvaluationTargetz: Represents evaluation Standard API target object z&Api endpoint to be used for evaluationr api_endpointN)rrrrrr!rrrrrrr(s rcseZdZUdZedddZeed<edddZeed<edd dZ e e e efed <ed d d d Z e e ed<edd dZe e ed<edd dZe e ed<edddZe e ed<edd dZe eed<edddZe e ed<ededZe eeefed<edddd Ze e ed <ed!ddd Ze e ed"<ed#d$dd Ze eed%<ed&d'dd Ze ed(<fd)d*ZZS)+ ConfigParamszA Represents evaluation Standard API config.params object z5Limits to the top tokens within a certain probabilityrr top_pzPTemp of 0 indicates greedy decoding, where the token with highest prob is chosen temperaturez=Limit evaluation to `limit` samples. Default: use all samplesN limit_samplesz6Number of examples in few-shot context. Default: None.zThis parameter is deprecated and not used for evaluations with NVIDIA Evals Factory. In order to specify few-shot please use extra.num_fewshot.r num_fewshotzmax tokens to generatemax_new_tokenszNumber of REST request retries max_retriesz4Number of parallel requests to be sent to the server parallelismzName of the tasktaskzREST response timeouti,request_timeoutzJFramework specific parameters to be used for evaluation (e.g. num_fewshot))r default_factoryextraz batch size to use for evaluationr batch_sizezms r>c@seZdZUdZedeeefddfddZe ddZ ee d <e d d d Z e ee d <e ddd Zee d<e ddd Zee d<e dd d Ze ee d<e ddd Zed Be d<e ddd Zed Be d<d S) AdapterConfigzcAdapter is a mechanism for hooking into the chain of requests/responses btw benchmark and endpoint. run_configreturnzAdapterConfig | NonecCsl|dididdrdS|didid}|s!dS|dididd|d <t|S) zFactory. Shall return `None` if the adapter_config is not passed, or validate the schema. Args: run_config: is the main dict of a configuration run, see `api_dataclasses`. targetr!streamFNadapter_configr endpoint_type)getrDmodel_validate)rErIrrrget_validated_config}s  z"AdapterConfig.get_validated_configz*The URL where the model endpoint is servedr api_urlzWLocal port to use for the adapter server. If `None` (default) will choose any free portNr local_portzMWhether to use the clean-reasoning-tokens adapter. See `end_reasoning_token`.F use_reasoningz0Token that singifies the end of reasoning outputzend_reasoning_tokenz=A custom system prompt to replace original one (if not None).custom_system_promptzUMaximum number of responses to log. Set to 0 to disable. If None, all will be logged.max_logged_responseszTMaximum number of requests to log. Set to 0 to disable. If None, all will be logged.max_logged_requests)rrrr staticmethodr;rrrNrrOrrPrrrQr<rRrSrUrVrrrrrDzs> rDc@seZdZdZdS)MisconfigurationErrorzG Exception raised when evaluation is not correctly configured. N)rrrrrrrrrXsrXN)typingrrrrpydanticrrrrr"r>rD ExceptionrXrrrrs= 9