o 5ti@sUddlmZddlZddlmZddlmZer-ddlmZm Z ddlm Z m Z e dZ e eZGdd d ZGd d d eZGd d d eZGdddeZeedZded<dddZdS)) annotationsN)Random) TYPE_CHECKING)IterableSequence)AnyTypeVar_Tc@s^eZdZ d!dddd"d d Z  d#d$ddZd%ddZd&ddZddZed!d'dd Z dS)(ContextSamplerN)rndfewshot_indicesdfSequence[dict[str, Any]] | Noner int | Noner list[int] | NonereturnNonecKs$t||_|pg|_||_d|_dSNF)rr r r _loaded)selfr r r kwargsrH/home/ubuntu/.local/lib/python3.10/site-packages/lm_eval/api/samplers.py__init__s   zContextSampler.__init__ninteval_docdict[str, Any] | NoneSequence[dict[str, Any]]cKs|dksJd|dkrgS|r||_|jsJd|s%|j||n|||j||d|}t||ksHJdt|d|d|S)a" Sample n documents from the pool. Args: n: Number of documents to sample eval_doc: Optional document to exclude from sampling df: Optional list of documents to sample from Returns: List of sampled documents rz.Error: number of samples requested must be >=0z+Error: no documents available for sampling.z+Error: number of fewshot samples returned (z!) not equal to number requested (z).)r r sample fewshot_docs rm_eval_doclen)rrrr rresrrrr s zContextSampler.samplecCst||_|SN)rr )rr rrrset_rndEs zContextSampler.set_rndcCs||_d|_|Sr)r r)rr rrr replace_dfIszContextSampler.replace_dfcsDjrjSjrjrjsfddjD_d_tjS)z'Return cached fewshot docs if availablecsg|]}j|qSr)r ).0irrr Ssz/ContextSampler.fewshot_docs..T)rr r listr*rr*rr!Ns  zContextSampler.fewshot_docsdocr _iter Iterable[_T] Sequence[_T]cs4|dur fdd|DSfdd|Dd|S)Ncg|]}|kr|qSrrr(xr-rrr+Zz.ContextSampler.rm_eval_doc..cr1rrr2r4rrr+\r5r)r-r.rrr4rr"Ws zContextSampler.rm_eval_docr%)r rr rr rrrNN)rrrrr rrr)r r)r r)r-r r.r/rr0) __name__ __module__ __qualname__rr r&r'r! staticmethodr"rrrrr s &  r c@eZdZddddZdS) FirstNSamplerNrrcKs2|t|jksJdt|jd|jd|S)z Draw the first `n` samples in order from the specified split. Used for tasks with "canonical" ordered fewshot examples, such as MMLU and CMMLU. z7Error: number of fewshot samples requested exceeds the z that are available.N)r#r rrrr rrrrr aszFirstNSampler.sampler6rrr7r8r9r rrrrr<`r<c@r;)BalancedSamplerNrrcKt)z TODO: this should return approximately class-balanced samples from our fewshot examples. TODO: what order should they be in? maybe random? NotImplementedErrorr=rrrr mszBalancedSampler.sampler6r>r?rrrrrAlr@rAc@r;) ManualSamplerNrrcKrB) rCr=rrrr wszManualSampler.sampler6r>r?rrrrrEvr@rE)defaultfirst_nzdict[str, type[ContextSampler]]SAMPLER_REGISTRYnamestrc CsDzt|WSty!}ztd|ddt|d}~ww)Nz!Attempted to use contextsampler 'zH', but no sampling strategy for this name found! Supported model names: z, )rIKeyErrorjoinkeys)rJerrr get_samplers rP)rJrK) __future__rloggingrandomrtypingrcollections.abcrrrrr getLoggerr7 eval_loggerr r<rArErI__annotations__rPrrrrs"   O