o 5tiB@s~ddlZddlmZmZddlmZddlmZmZm Z m Z m Z eGddde Z eGddde ZGd d d ejZdS) N)asdict dataclass) getsource)AnyCallableListOptionalUnionc@s^eZdZUdZeeed<dZeeed<dZeeed<dZ ee ee fed<d d Z dS) AggMetricConfigNmetricmean aggregationFweight_by_sizenone filter_listcCsD|jdkrt|jstd|jdt|jtr |jg|_dSdS)Nr zTCurrently, 'mean' is the only pre-defined aggregation across groups' subtasks. Got 'z'.)r callable ValueError isinstancerstrselfrE/home/ubuntu/.local/lib/python3.10/site-packages/lm_eval/api/group.py __post_init__s  zAggMetricConfig.__post_init__) __name__ __module__ __qualname__r rr__annotations__r rrr listrrrrrr s  r c@seZdZUdZeeed<dZeeed<dZee ee fed<dZ ee e e e efed<dZeeed<ddZd d Zd d ZddedefddZ dde eefde eeffddZdS) GroupConfigNgroup group_aliastaskaggregate_metric_listmetadatacCs t||SN)getattr)ritemrrr __getitem__%s zGroupConfig.__getitem__cCs t|||Sr%)setattr)rr'valuerrr __setitem__(s zGroupConfig.__setitem__cCs:|jdurt|jtr|jg|_dd|jD|_dSdS)NcSs(g|]}t|trtdi|n|qS)r)rdictr ).0r'rrr 0sz-GroupConfig.__post_init__..)r#rr,rrrrr+s    zGroupConfig.__post_init__F keep_callablereturncCs<t|}t|D]\}}t|r|j||d||<q |S)addumps the current config as a dictionary object, as a printable format. null fields will not be printed. Used for dumping results alongside full task configuration :return: dict A printable dictionary version of the TaskConfig object. # TODO: should any default value in the TaskConfig not be printed? )r/)rritemsrserialize_function)rr/cfg_dictkvrrrto_dict5s  zGroupConfig.to_dictr*c Cs2|r|Szt|WSttfyt|YSw)zSerializes a given function or string. If 'keep_callable' is True, the original callable is returned. Otherwise, attempts to return the source code of the callable using 'getsource'. )r TypeErrorOSErrorr)rr*r/rrrr2Fs  zGroupConfig.serialize_function)F)rrrr rrrr!r"r rr#rr r,r$r(r+rboolr6rr2rrrrrs(    rc@speZdZ ddeeddfddZeddZedd Zed d Z ed d Z ede fddZ ddZ dS)ConfigurableGroupNconfigr0cCstdi||_dS)Nr)r_config)rr;rrr__init__XszConfigurableGroup.__init__cC|jjSr%r<r rrrrr ^zConfigurableGroup.groupcCr>r%)r<r!rrrrr!br@zConfigurableGroup.group_aliascCr>r%)r<versionrrrrrAfr@zConfigurableGroup.versioncCs |jSr%)r<r6rrrrr;js zConfigurableGroup.configcCr>r%r?rrrr group_namenr@zConfigurableGroup.group_namecCsd|jd|jdS)NzConfigurableGroup(group=z ,group_alias=))r r!rrrr__repr__rszConfigurableGroup.__repr__r%)rrrrr,r=propertyr r!rAr;rrBrDrrrrr:Ws$      r:)abc dataclassesrrinspectrtypingrrrrr r,r rABCr:rrrrs =