o 5ti; @sddlmZddlZddlmZmZddlmZddlm Z m Z ddl m Z e r4ddl mZddlmZeeZeGd d d ZeGd d d eZdS) ) annotationsN)asdict dataclass) getsource) TYPE_CHECKINGAny)default_gen_kwargs)Callable) OutputTypec @seZdZUdZdZded<dZded<dZded <dZd ed <dZ d ed <dZ ded<dZ ded<dZ ded<dZ ded<dZded<dZded<ddZeddddddddddddZdS) FewshotConfigzConfiguration for few-shot example formatting. These fields override the parent TaskConfig fields when formatting few-shot examples (as opposed to the test example). note: num_fewshot is also runtime dependent, so is not included here. defaultstrsamplerN str | Nonesplit Callable[..., list[dict]] | None process_docszlist[int] | Nonefewshot_indicesz,list[dict] | Callable[[], list[dict]] | Nonesamplesstr | Callable[..., str] | None doc_to_text-str | Callable[..., str] | dict | list | None doc_to_choice doc_to_target gen_prefixfewshot_delimitertarget_delimitercCs*|jdur|jdurtddSdSdS)NzABoth split and samples are configured; split will take precedence)rr eval_loggerwarningselfr!G/home/ubuntu/.local/lib/python3.10/site-packages/lm_eval/config/task.py __post_init__+s zFewshotConfig.__post_init__) fewshot_splitrrrrrrrcfgdictr$returnc Ks8|||||||| d|| } | dd|di| S)Nrrrrrrrrrr r!) setdefault) clsr%r$rrrrrrr overloadscfg_dictr!r!r" from_dict1s   zFewshotConfig.from_dict)r%r&r$rrrrrrrrrrrrrrrr'r )__name__ __module__ __qualname____doc__r__annotations__rrrrrrrrrrr# classmethodr-r!r!r!r"r s0            r c@seZdZUdZded<dZded<dZded<dZded<dZded <dZ ded <dZ d ed <dZ ded <dZ ded<dZ ded<dZded<dZded<dZded<dZded<dZded<dZded<dZded<dZded<dZded<dZded<dZded <d!Zded"<d#Zded$<dZd%ed&<dZd'ed(<dZd)ed*<d+Zd,ed-<dZd ed.<d/Z d0ed1<dZ!ded2<dZ"ded3<dZ#ded4<dZ$ded5<dZ%d ed6<dGd9d:Z&d;d<Z'd=d>Z(dHdIdAdBZ) dHdJdEdFZ*dS)K TaskConfigNrtask task_aliaszstr | list | NonetagzCallable | Nonecustom_dataset dataset_path dataset_namez dict | Nonedataset_kwargstraining_splitvalidation_split test_splitr$rzCallable | str | Nonerr doc_to_image doc_to_audioFbool unsafe_codez#Callable | str | dict | list | Nonerprocess_results use_promptr description rz rz%dict[str, Any] | FewshotConfig | Nonefewshot_configz int | None num_fewshotz list | None metric_listgenerate_untilr output_typegeneration_kwargsintrepeats filter_listshould_decontaminatedoc_to_decontamination_queryrmetadatar'Nonec Cs|jdur>|jdkrtd|jdd|jvr#t|jd|jd<d|jvr=t|jdt|j|jg|jd<n|jdkrUt|j|_t|jd|jt |j t s`|j dur|t j |j pfi|j|j|j|j|j|j|j|jd |_ dS|j |_ dS) NrK[zJ] passed `generation_kwargs`, but not using `output_type: generate_until`! temperatureuntilzS: No `until` specified in `generation_kwargs`! Defaulting to the fewshot_delimiter=zA: No `generation_kwargs` specified in task config, defaulting to r()rMrLrrr5floatreprrr isinstancerHr&r r-r$rrrrrrrr!r!r"r#sJ          zTaskConfig.__post_init__cCs t||SN)getattr)r itemr!r!r" __getitem__s zTaskConfig.__getitem__cCs t|||Sr\)setattr)r r^valuer!r!r" __setitem__s zTaskConfig.__setitem__ keep_callabler&cCst|}t|D]?\}}|dur||q |dkr<|D]}|D]\}}t|r5|j||d||<q$q|||<q t|rI|j||d||<q |S)aeDumps the current config as a dictionary object, as a printable format. null fields will not be printed. Used for dumping results alongside full task configuration :return: dict A printable dictionary version of the TaskConfig object. # TODO: should any default value in the TaskConfig not be printed? NrJ)rc)rlistitemspopcallableserialize_function)r rcr,kv metric_dict metric_key metric_valuer!r!r"to_dicts$    zTaskConfig.to_dictraCallable | strc Cs2|r|Szt|WSttfyt|YSw)zSerializes a given function or string. If 'keep_callable' is True, the original callable is returned. Otherwise, attempts to return the source code of the callable using 'getsource'. )r TypeErrorOSErrorr )r rarcr!r!r"rhs  zTaskConfig.serialize_function)r'rU)F)rcrAr'r&)raror'ro)+r.r/r0r5r2r6r7r8r9r:r;r<r=r>r$rrrr?r@rBrrCrDrFrrrHrIrJrLrMrPrQrRrSrrTr#r_rbrnrhr!r!r!r"r4QsV                                 ( r4) __future__rlogging dataclassesrrinspectrtypingrrlm_eval.defaultsrcollections.abcr lm_eval.api.instancer getLoggerr.rr r&r4r!r!r!r"s      <