o 5ti W@s*UdZddlmZddlZddlmZddlZddlZddl Z ddl m Z ddl m Z ddlmZddlmZmZmZmZmZmZeeZer\ddl m Z mZdd lmZdd lmZgd Zed Z ed Z!e"ej#BZ$e%Z&de'd<e ddd`ddZ( dadbdd Z)dcd$d%Z*Gd&d'd'ee Z+e+d(Z,d)e'd*<e+d+Z-d,e'd-<e+d.Z.d/e'd0<e+d1Z/d2e'd3<e+d4Z0d2e'd5<e+d6Z1d7e'd8<d9d:Z2e/Z3e0Z4e.Z5e1Z6d;dgd?gd@Z7dAdBZ8dddDdEZ9e,Z:dedFdGZ;dfdKdLZdgdhdSdTZ?dedUdVZ@didXdYZAdidZd[ZBdjd^d_ZCdS)kaRegistry system for lm_eval components. This module provides a centralized registration system for models, tasks, metrics, filters, and other components in the lm_eval framework. ## Usage Examples ### Registering a Model ```python from lm_eval.api.registry import register_model from lm_eval.api.model import LM @register_model("my-model") class MyModel(LM): def __init__(self, **kwargs): ... ``` ### Registering with Lazy Loading ```python # Register without importing the actual implementation model_registry.register("lazy-model", target="my_package.models: LazyModel") ``` ### Looking up Components ```python from lm_eval.api.registry import get_model # Get a model class model_cls = get_model("gpt-j") model = model_cls(**config) ``` ) annotationsN)Callable) lru_cache)MappingProxyType) TYPE_CHECKINGAnyGenericTypeVarcastoverload)rIterable)FilterLM)Registrymodel_registryfilter_registryaggregation_registrymetric_registrymetric_agg_registryhigher_is_better_registry freeze_allregister_model get_modelregister_metric get_metricregister_aggregationget_aggregationget_metric_aggregationis_higher_betterregister_filter get_filterMODEL_REGISTRYFILTER_REGISTRYMETRIC_REGISTRYMETRIC_AGGREGATION_REGISTRYAGGREGATION_REGISTRYHIGHER_IS_BETTER_REGISTRYDEFAULT_METRIC_REGISTRYTDr_MISSING)maxsizeph PlaceholderreturncCsFt|tr|d\}}}|std|dtt||S|S)aMaterialize a lazy placeholder into the actual object. This is at module level to avoid memory leaks from lru_cache on instance methods. Args: ph: Either a string path "module:object" or an EntryPoint instance Returns: The loaded object Raises: ValueError: If the string format is invalid ImportError: If the module cannot be imported AttributeError: If the object doesn't exist in the module :zInvalid lazy path 'z', expected 'module:object') isinstancestr partition ValueErrorgetattr importlib import_moduleload)r.mod_attrr=H/home/ubuntu/.local/lib/python3.10/site-packages/lm_eval/api/registry.py_materialise_placeholderds r?queryr3options Iterable[str]max_suggestionsint list[str]cCsd|}g}|D]'}|}||vs!t|dkr/||ddr/||t||kr/|Sq|S)z_suggest_similar}s  rOnamealiaskeyscCs~t||}t|}d|d|d}|r|dd|d7}|dd|dd 7}t|d kr=|d t|d 7}|S) z2Build a helpful KeyError message with suggestions.zUnknown z 'z'.z Did you mean: , ?z Available: Nz... (z total))rOsortedjoinrH)rPrQrRrL availablemsgr=r=r>_build_key_error_msgs  rZc@seZdZdZddd9d d Zdd d:ddZd;ddZedd dZd__init__s zRegistry.__init__targetaliasesrgT | Placeholder | NoneCallable[[T], T]csdfdd dfd d }|d ur?tdkrtdjd|Wd ddS1s6wYddS|S)aRegister an object under one or more aliases. Can be used as a decorator or called directly for direct registration. Args: *aliases: Names to register the object under. If empty, uses object's __name__ target: For direct calls only - a value, placeholder string "module:object", or EntryPoint to register under the alias Returns: Decorator function (or no-op if direct registration) Examples: >>> # As decorator >>> @model_registry.register("name1", "name2") >>> class MyModel(LM): ... pass >>> >>> # Direct registration with a lazy placeholder >>> model_registry.register("lazy-name", target="mymodule:MyModel") Raises: ValueError: If alias is already registered with a different target TypeError: If an object doesn't inherit from base_cls (when specified) rQr3rgT | Placeholderr0r]c sj|}|dur;||kr;t|tr*t|tr*||jd|jkr*|j|<dStjd|d|d|dj durYt|trYt |j sYt |dj dj|j|<dS)Nr1z alias 'z' already registered (existing=z, new=)z must inherit from z to be a ) r`getr2r3type __module____name__r5r^r_ issubclass TypeError)rQrgcurrentrdr=r>_stores2    z!Registry.register.._storeobjr)csVp t|dt|f}j|D]}||qWd|S1s$wY|S)Nrp)r6r3rc)rvnamesrPrurhrdr=r> decorators  z$Registry.register..decoratorNz/Exactly one alias required when using 'target='rcSs|SNr=)xr=r=r>sz#Registry.register..)rQr3rgrkr0r])rvr)r0r))rHr5rc)rdrgrhryr=rxr>registers  zRegistry.registerr.r/r)cCstdt|S)zMaterialize a placeholder using the module-level cached function. Args: ph: Placeholder to materialize Returns: The materialized object, cast to type T r))r r?)rdr.r=r=r> _materialises zRegistry._materialiserQcCdSr{r=rdrQr=r=r>rmz Registry.getdefaultr*T | DcCrr{r=)rdrQrr=r=r>rmrD | Anyc Csz|j|}Wn%ty,}z|tur|WYd}~Stt|j||j|d}~wwt|ttj frj|j (|j|}t|ttj frW| |}t|jt sV||j|<n|}Wdn1scwY|}|j durt|trt||j st|d|j d|dd|S)awRetrieve an object by alias, materializing if needed. Thread-safe lazy loading: if the alias points to a placeholder, it will be loaded and cached before returning. Args: alias: The registered name to look up default: Default value to return if alias not found (can be None) Returns: The registered object, or default if not found Raises: KeyError: If an alias is not found and no default provided TypeError: If a materialized object doesn't match base_cls ImportError/AttributeError: If lazy loading fails Nz does not inherit from z (alias 'z'))r`KeyErrorr+rZr^rRr2r3md EntryPointrcrrr_rnrqrr)rdrQrrgexcfreshconcreter=r=r>rmsF      cCs ||S)z)Allow dict-style access: registry[alias].)rmrr=r=r> __getitem__I zRegistry.__getitem__boolcCs ||jvS)zCheck if alias is registered.)r`rr=r=r> __contains__MrzRegistry.__contains__cC t|jS)z Iterate over registered aliases.)iterr`rtr=r=r>__iter__QrzRegistry.__iter__cCr)z(Return the number of registered aliases.)rHr`rtr=r=r>__len__UrzRegistry.__len__cCs6tdd|jD}d|jdt|d|dS)z6Return a detailed string representation for debugging.css$|] }t|ttjfsdVqdS)rzN)r2r3rr).0vr=r=r> [s  z$Registry.__repr__..z Registry(z , entries=z, materialized=rl)sumr`valuesr^rH)rd materializedr=r=r>__repr__YszRegistry.__repr__cC |jS)zReturn all registered aliases.)r`rRrtr=r=r>rR`rz Registry.keyscCr)zvReturn all registered objects. Note: Objects may be placeholders that haven't been materialized yet. )r`rrtr=r=r>rd zRegistry.valuescCr)zuReturn (alias, object) pairs. Note: Objects may be placeholders that haven't been materialized yet. )r`itemsrtr=r=r>rkrzRegistry.items str | NonecCs^|j|}t|ttjfrdSzt|}t|d}|d|WSt y.YdSw)zGet the source location of a registered object. Args: alias: The registered name Returns: "path/to/file.py:line_number" or None if not available Nrzr1) r`rmr2r3rrinspectgetfilegetsourcelines Exception)rdrQrvpathliner=r=r>origints   zRegistry.origincCs<|jtt|j|_WddS1swYdS)zMake the registry read-only to prevent further modifications. After freezing, attempts to register new objects will fail. This is useful for ensuring registry contents don't change after initialization. N)rcrdictr`rtr=r=r>freezes"zRegistry.freezecCs.t|jtr t|j|_|jtdS)zErase registry (for isolated tests). Clears both the registry contents and the materialization cache. Only use this in test code to ensure a clean state between tests. N)r2r`rrclearr? cache_clearrtr=r=r>_clears    zRegistry._clear)rPr3r[r\r0r])rhr3rgrir0rj)r.r/r0r))rQr3r0r))rQr3rr*r0r)rQr3rrr0r)rQr3r0r)r0r3)rQr3r0r)rpro __qualname____doc__rer~rr rmr+rrrrrrRrrrrrr=r=r=r>rs.  N   3    rmodelzRegistry[type[LM]]rfilterzRegistry[type[Filter]]r aggregationzRegistry[Callable[..., float]]rmetriczRegistry[Callable]rmetric_aggregationrhigher_is_betterzRegistry[bool]rcCs"ttttttfD]}|qdS)zFreeze all registries to prevent further modifications. This is useful for ensuring registry contents are immutable after initialization, preventing accidental modifications during runtime. N)rrrrrrr)rr=r=r>rs r perplexityacc)word_perplexitybyte_perplexity bits_per_byteacc_norm exact_match) loglikelihoodloglikelihood_rollingmultiple_choicegenerate_untilcsddlmfdd}|S)aDecorator to register a model class. Args: *names: One or more names to register the model under Returns: Decorator function Example: >>> @register_model("my-model", "my-model-alias") >>> class MyModel(LM): ... pass rrcs.t|sJd|jdtj||S)NzModel 'z' must extend LM class)rqrprr~clsrrwr=r>decoratesz register_model..decorate)lm_eval.api.modelr)rwrr=rr>rs r model_namec CsZttdkr ddl}zt|WSty,}ztd|ddt|d}~ww)zGet a model class by name. Args: model_name: The registered name of the model Returns: The model class Raises: ValueError: If model name is not found rNzAttempted to load model 'z<', but no model for this name found! Supported model names: rS)rHrlm_eval.modelsrmrr5rWrR)rlm_evaler=r=r>rs   rcfdd}|S)zDecorator to register a filter class. Args: name: Name to register the filter under Returns: Decorator function cs,tvr tddt||S)NzRegistering filter `z` that is already in Registry)r eval_loggerinfor~rrPr=r>rsz!register_filter..decorater=rPrr=rr>r s r filter_namestr | Callablerc CsNt|r|Sz ttd|WSty&}z td|d|d}~ww)zGet a filter by name. Args: filter_name: The registered name of the filter, or a callable Returns: The filter class/function Raises: KeyError: If a filter name is not found and is not callable r3zfilter `z` is not registered!N)callablerrmr rrwarning)rrr=r=r>r!!s r!c r)aPDecorator to register a metric function. Args: **args: Keyword arguments including - metric: Name to register the metric under (required) - higher_is_better: Whether higher scores are better - aggregation: Name of aggregation function to use Returns: Decorator function csddvsJd}t||dvrtj|dddvr0td}tj||d|S)Nrrrfr)rr~rrrmr)fnrPagg_fnargsr=r>rLs z!register_metric..decorater=)rrr=rr>r?s rFhf_evaluate_metricrCallable | NonecCs|ttdkr ddl}|s|tvrt|Std|dz ddl}||}|jWSt y=t |dYdSw)zGet a metric function by name. Args: name: The metric name hf_evaluate_metric: If True, skip the local registry and use HF evaluate Returns: The metric compute function, or None if not found rNz"Could not find registered metric 'z1' in lm-eval, searching in HF Evaluate library...zW not found in the evaluate library! Please check https://huggingface.co/evaluate-metric) rHrlm_eval.api.metricsrmrrevaluater9computererror)rPrr hf_evaluate metric_objectr=r=r>ras$     rcr)zDecorator to register an aggregation function. Args: name: Name to register the aggregation under Returns: Decorator function cst||Sr{)rr~)rrr=r>rsz®ister_aggregation..decorater=rr=rr>rs rCallable[..., float] | NonecCFttdkr ddl}zt|WSty"t|dYdSw)zGet an aggregation function by name. Args: name: The aggregation name Returns: The aggregation function, or None if not found rNz% not a registered aggregation metric!)rHrrrmrrrrPrr=r=r>r   rcCr)zGet the aggregation function for a metric. Args: name: The metric name Returns: The aggregation function for that metric, or None if not found rNz. metric is not assigned a default aggregation!)rHrrrmrrrrr=r=r>rrr metric_name bool | NonecCsHttdkr ddl}zt|WSty#td|dYdSw)zCheck if higher values are better for a metric. Args: metric_name: The metric name Returns: True if higher is better, False otherwise, None if not found rNz+higher_is_better not specified for metric 'z'!)rHrrrmrrr)rrr=r=r>rs    r)r.r/r0r)r@)rAr3rBrCrDrEr0rF)rPr3rQr3rRrCr0r3)rr3)rPr3)rrr0r)F)rPr3rrr0r)rPr3r0r)rr3r0r)Dr __future__rr7importlib.metadatametadatarrloggingracollections.abcr functoolsrtypesrtypingrrrr r r getLoggerrprr lm_eval.api.filterr rr__all__r)r*r3rr/objectr+__annotations__r?rOrZrrrrrrrrr$r%r&r'r(rrr"r r!r#rrrrrrr=r=r=r>sv "        "       " "