o 5tiJ @s0ddlZddlZddlmZGdddeZdS)N) SubCommandcsFeZdZdZdejffdd Zd ddZd ejddfd d Z Z S) Listz$Command for listing available tasks. subparserscsNtj|i||jddddtdtjd|_||jj |j ddS)Nlsz/List available tasks, groups, subtasks, or tagszLList available tasks, groups, subtasks, or tags from the evaluation harness.z>lm-eval list [tasks|groups|subtasks|tags] [--include_path DIR]uI examples: # List all available tasks (includes groups, subtasks, and tags) $ lm-eval ls tasks # List only task groups (like 'mmlu', 'glue', 'superglue') $ lm-eval ls groups # List only individual subtasks (like 'mmlu_abstract_algebra') $ lm-eval ls subtasks # Include external task definitions $ lm-eval ls tasks --include_path /path/to/external/tasks # List tasks from multiple external paths $ lm-eval ls tasks --include_path "/path/to/tasks1:/path/to/tasks2" organization: • Groups: Collections of tasks with aggregated metric across subtasks (e.g., 'mmlu') • Subtasks: Individual evaluation tasks (e.g., 'mmlu_anatomy', 'hellaswag') • Tags: Similar to groups but no aggregate metric (e.g., 'reasoning', 'knowledge', 'language') • External Tasks: Custom tasks defined in external directories evaluation usage: After listing tasks, use them with the run command! For more information tasks configs are defined in https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks )help descriptionusageepilogformatter_class)func) super__init__ add_parsertextwrapdedentargparseRawDescriptionHelpFormatter_parser _add_args set_defaults_execute)selfrargskwargs __class__C/home/ubuntu/.local/lib/python3.10/site-packages/lm_eval/_cli/ls.pyr s#z List.__init__returnNcCs2|jjdgdddd|jjdtdddd dS) Nwhat)tasksgroupssubtaskstags?z4What to list: tasks (all), groups, subtasks, or tags)choicesnargsrz--include_pathDIRz7Additional path to include if there are external tasks.)typedefaultmetavarr)r add_argumentstr)rrrrr3s zList._add_argsrcCsddlm}||jd}|jdkrt|d S|jdkr)t|jdddd S|jdkr9t|jddd d S|jd krIt|jddd d S|jd urU|jd Sd S) zExecute the list command.r) TaskManager) include_pathr r!F) list_subtasks list_tagsr") list_groupsr0r#)r1r/N) lm_eval.tasksr-r.rprintlist_all_tasksr print_help)rrr- task_managerrrrrBs       z List._execute)rN) __name__ __module__ __qualname____doc__r_SubParsersActionr r Namespacer __classcell__rrrrrs  )r)rrlm_eval._cli.subcommandrrrrrrs