o 5tie@sddlZddlZddlZddlZddlZddlmZddlmZm Z ddl m Z ddl m Z ddl mZmZddlmZddlmZdd lmZmZmZmZmZmZmZmZmZmZmZerddd l m!Z!e"e#Z$e d d Gd ddZ%GdddZ&dS)N) defaultdict)asdict dataclass)datetime)Path) TYPE_CHECKINGAny) load_dataset)MetadataConfigs) get_file_datetimeget_file_task_nameget_results_filenamesget_sample_results_filenameshandle_non_serializable hash_string info_oncerandom_name_id sanitize_listsanitize_model_namesanitize_task_nameHfApiF)initc @seZdZUdZdZedBed<dZedBed<dZedBed<dZ edBed<dZ edBed<dZ e dBed<dZ edBed <dZedBed <dZedBed <dd dZedeeeefBdBd edBfddZdedeeeefBdedBd edBde d df ddZdddZdS)GeneralConfigTrackeraTracker for the evaluation parameters. Attributes: model_source (str | None): Source of the model (e.g. hf, vllm, etc.) model_name (str | None): Name of the model. model_name_sanitized (str | None): Sanitized model name for directory creation. system_instruction (str | None): System instruction/prompt provided to the model. system_instruction_sha (str | None): SHA hash of the system instruction for tracking and reproducibility. fewshot_as_multiturn (bool | None): Whether few-shot examples are formatted as multi-turn conversations. chat_template (str | None): Chat template used for formatting prompts. chat_template_sha (str | None): SHA hash of the chat template for tracking and reproducibility. start_time (float): Start time of the experiment. Logged at class init. end_time (float): End time of the experiment. Logged when calling `GeneralConfigTracker.log_end_time`. total_evaluation_time_seconds (str | None): Inferred total evaluation time in seconds (from the start and end times). N model_source model_namemodel_name_sanitizedsystem_instructionsystem_instruction_shafewshot_as_multiturn chat_templatechat_template_shatotal_evaluation_time_secondsreturncCst|_dS)zStarts the evaluation timer.N)time perf_counter start_timeselfr)V/home/ubuntu/.local/lib/python3.10/site-packages/lm_eval/loggers/evaluation_tracker.py__init__FszGeneralConfigTracker.__init__ model_argscCsdtdtdtfdd}gd}t|tr'|D]}||vr$t||SqdSt|trB|D]}|d|vrA|||dSq.dS) z1Extracts the model name from the model arguments.r,keyr#cSs||d}|ddS)z=Extracts the model name from the model arguments using a key.,r)split)r,r-args_after_keyr)r)r*extract_model_nameNsz@GeneralConfigTracker._get_model_name..extract_model_name)peftdelta pretrainedmodelpathengine=N)str isinstancedict)r,r2prefixesr-prefixr)r)r*_get_model_nameJs  z$GeneralConfigTracker._get_model_namecCs^||_t|p t|_t|j|_||_|rt|nd|_ ||_ |r't|nd|_ ||_ dS)z!Logs model parameters and job ID.N) rrr?rrrrrrrr r!r)r(rr,rr rr)r)r*log_experiment_args_s   z(GeneralConfigTracker.log_experiment_argscCs t|_t|j|j|_dS)zMLogs the end time of the evaluation and calculates the total evaluation time.N)r$r%end_timer:r&r"r'r)r)r* log_end_timeus z!GeneralConfigTracker.log_end_timer#N)__name__ __module__ __qualname____doc__rr:__annotations__rrrrrboolr r!r"r+ staticmethodr<rr?r@rBr)r)r)r*r%s8  ( rc@seZdZdZ            d dedBdededed ed ed ed ed ededededdfddZed!d edBddfddZ d!de de dBddfddZ dede ddfddZ d"ddZ dS)#EvaluationTrackerzKeeps track and saves relevant information of the evaluation process. Compiles the data from trackers and writes it to files, which can be published to the Hugging Face hub if requested. NF output_pathhub_results_org hub_repo_namedetails_repo_nameresults_repo_namepush_results_to_hubpush_samples_to_hub public_repotokenleaderboard_urlpoint_of_contactgatedr#c Cst|_||_||_||_||_| |_| |_|| |_ | |_ |j s*|s&|r*t d|j rE|dkrE|s5|rE|j d}t d|d|dkrZ|dkrO|nd}|dkrW|n|}n |}|}t d|d||_|d|d |_|d||_|d|d |_d S) aCreates all the necessary loggers for evaluation tracking. Args: output_path (str | None): Path to save the results. If not provided, the results won't be saved. hub_results_org (str): The Hugging Face organization to push the results to. If not provided, the results will be pushed to the owner of the Hugging Face token. hub_repo_name (str): The name of the Hugging Face repository to push the results to. If not provided, the results will be pushed to `lm-eval-results`. Deprecated in favor of details_repo_name and results_repo_name. details_repo_name (str): The name of the Hugging Face repository to push the details to. If not provided, defaults to `lm-eval-results`. results_repo_name (str): The name of the Hugging Face repository to push the results to. If not provided, defaults to details_repo_name. push_results_to_hub (bool): Whether to push the results to the Hugging Face hub. push_samples_to_hub (bool): Whether to push the samples to the Hugging Face hub. public_repo (bool): Whether to push the results to a public or private repository. token (str): Token to use when pushing to the Hugging Face hub. This token should have write access to `hub_results_org`. leaderboard_url (str): URL to the leaderboard on the Hugging Face hub on the dataset card. point_of_contact (str): Contact information on the Hugging Face hub dataset card. gated (bool): Whether to gate the repository. zHugging Face token is not defined, but 'push_results_to_hub' or 'push_samples_to_hub' is set to True. Please provide a valid Hugging Face token by setting the HF_TOKEN environment variable.rLnamez>hub_results_org was not specified. Results will be pushed to 'z'.zlm-eval-resultszhub_repo_name was specified. Both details and results will be pushed to the same repository. Using hub_repo_name is no longer recommended, details_repo_name and results_repo_name should be used instead./z-privateN)rgeneral_config_trackerrMrRrSrTrVrW_apiapi gated_repo ValueErrorwhoami eval_loggerwarning details_repodetails_repo_private results_reporesults_repo_private) r(rMrNrOrPrQrRrSrTrUrVrWrXr)r)r*r+sL-  zEvaluationTracker.__init__z HfApi | NonecCs|sdSddlm}||dS)z8Initializes the Hugging Face API if a token is provided.Nrr)rU)huggingface_hubr)rUrr)r)r*r\s  zEvaluationTracker._apiresultssamplesc Cs|j|jrztdi}|r+|D]\}}dd|D}td|||<q|d|i|t |jt j |dt dd}t |jrJ|jnt }td d |_|jd kru|jjd d d ||jd|jd } n|t|jj}|jd d d |d|jd } | j|dd|jr|jr|jr|jn|j } |jj!| d|j d d|jj"| t| t#j$t|jj%| j&dd|jj%dtd| WdSWdSWdSt'y} zt(dtt)| WYd} ~ dSd} ~ wwtddS)a Saves the aggregated results and samples to the output path. Pushes them to the Hugging Face hub if requested. Args: results (dict): The aggregated results to save. samples (dict | None): The samples results to save. zSaving results aggregatedcSs$g|]}|d|d|dqS)doc_hash prompt_hash target_hashr)).0sr)r)r* sz=EvaluationTracker.save_results_aggregated..rL task_hashesF)indentdefault ensure_ascii:-.jsonTparentsexist_ok_results_utf-8encodingdatasetrepo_id repo_typeprivaterzzAdding aggregated results for )rpath_or_fileobj path_in_reporcommit_messagezVSuccessfully pushed aggregated results to the Hugging Face Hub. You can find them at: z!Could not save results aggregatedNztd|t|jr|jnt}|jdkr#|j}n|t|jj }|j ddd|d|d|j d}t td|d t |d d d a}|D]V}i}t|d D] \}} i|d|<t| D]\} } | |d|d| <qjq[t|d|d<t|d|d<||d <t|d|d<tj|tddd} || qQWdn1swY|jr>|jrAddlm} m}m}|jr|jn|j}|jj|d|j ddz|jr| }|jd|d|ddid}||Wnt y}zt!d t"t#|WYd}~nd}~ww|jj$|t||jj dd!|d"|jj%d#t"d$|d%|WdSWdSWdSt yc}zt!d&t"t#|WYd}~dSd}~wwt"d'dS)(zSaves the samples results to the output path. Pushes them to the Hugging Face hub if requested. Args: task_name (str): The task name to save the samples for. samples (dict): The samples results to save. zSaving per-sample results for: rwTrxsamples_r{z.jsonlzSaving per-task samples to z/*.jsonlar}r~ arguments gen_args_arg_respsfiltered_respstargetF)rsrt Nr)build_hf_headers get_sessionhf_raise_for_statusrrz$https://huggingface.co/api/datasets/z /settingsrXauto)urlheadersrzCould not gate the repositoryzAdding samples results for z to )r folder_pathrrrz-Successfully pushed sample results for task: z0 to the Hugging Face Hub. You can find them at: zCould not save sample resultsz8Output path not provided, skipping saving sample results)&rMradebugrrrrr:r[rrrropen enumeraterrrrwriter]rShuggingface_hub.utilsrrrrTrcrdrr^putrrbrr upload_folderr)r(rrir7file_results_samplesfsampleriargjtmp sample_dumprrrrrrrr)r)r*save_results_samples@s      * z&EvaluationTracker.save_results_samplesc"sddlm}m}m}td|jr|jn|j}|j j |dd}t |}t |}t dd|D]8}t|}|j} |j} t| } t| } t| } | d| }| d }t|| |<t||<q0t}|D]Z}t|}|j}|j} t|}td d |}td t|j}| d }td d |}||kr||d gi}|d |t|gd|||<||d dt|gdqn|D]e}t|}|j} |j} t| } t| }t| } td d |}td t| j}| d| }td d |}||kr0||d gi}|d |t|gd|||<||d dt|gdqttfddd}fdd|Dd}|||dd}td|dd}|dd}d|i}||tj|dd}d}|j j!dkr|d|j j"d|j j"d 7}n ||j j"d!7}|d"t#|d#d$t#|d%7}|j j!dkr|d&|d'|d(7}|d)d*|$d+d,d-|d.7}||d/|j j"d0|j j"|j%|j&d1} |'| |j(| | j)d2}!|!j*|dd3d4S)5zvCreates a metadata card for the evaluation results dataset. Pushes the card to the Hugging Face hub. r) DatasetCardDatasetCardData hf_hub_urlzRecreating metadata cardr)rrcSs tjSN)rminrr)r)r)r*s z:EvaluationTracker.recreate_metadata_card..__ __resultsz[^\w.]r{z** data_files)r0r7latestcs|Srr))k)latest_task_results_datetimer)r*rs)r-cs g|] }dd|vr|qS)rurv)r)rmr)latest_datetimer)r*rosz.)rfilenamerrtrain)rr0rhall)rrzADataset automatically created during the evaluation run of model hf[z](https://huggingface.co/z) rzThe dataset is composed of r.zk configuration(s), each one corresponding to one of the evaluated task. The dataset has been created from aW run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the latest results. An additional configuration "results" store all the aggregated results of the run. To load the details from a run, you can for instance do the following: zC```python from datasets import load_dataset data = load_dataset( "z ", name="z", split="latest" ) ``` z:## Latest results These are the [latest results from run z](z /resolve/z/blob/z) (note that there might be results for other tasks in the repos if successive evals didn't cover the same tasks. You find each in the results and the "latest" split for each eval): ```python z ```zhttps://huggingface.co/zEvaluation run of )dataset_summaryrepo_url pretty_namerVrW)r)rN)+rgrrrrarrTrcrdr]list_repo_filesr rrrrYrr r rmaxr resubgetappendr:valuesr rrrr[rrlenrrVrWto_dataset_card_data from_templater push_to_hub)"r(rrrr files_in_repo results_files sample_files file_pathrrrresults_datetimetask_name_sanitized samples_key results_key card_metadataresults_filename eval_dateeval_date_sanitized config_name sanitized_last_eval_date_resultscurrent_resultscurrent_details_for_tasklatest_model_namelast_results_filelast_results_file_pathlatest_results_file results_dictnew_dictionaryresults_stringr card_datacardr))rrr*recreate_metadata_cards                    z(EvaluationTracker.recreate_metadata_card) NrLrLrLrLFFFrLrLrLFrrC) rDrErFrGr:rIr+rJr\r<rrrr)r)r)r*rK{sv      [  Z hrK)'rloggingrrr$ collectionsr dataclassesrrrpathlibrtypingrrdatasetsr datasets.utils.metadatar lm_eval.utilsr r r rrrrrrrrrgr getLoggerrDrarrKr)r)r)r*s&     4  U