o i{0 @sXUddlmZddlmZddlZddlmZddlmZddlmZddl Z ddl Z ddl Z ddl Z ddl mZddl mZdd l mZdd l mZdd l mZdd l mZdd l mZddl mZddl mZddl mZddl mZzddl mZWneyddlmZYnwddlZzddlmZddlmZWn eydZdZYnwddl m!Z!ddl"m#Z#ddl"m$Z$ddl"m%Z%ddl&m'Z'ddl(m)Z)ddl(m*Z*ddl(m+Z+ddl(m,Z,ddl-m.Z.ddl-m/Z/dd l0m1Z1erdd!l2m3Z3dd"l4m5Z5dd#l4m6Z6dd$l7m8Z8e'e9Z:ee;ede?d%e@e;d%ffZAee;ee?eAe@e;eAffZBe@e;eAfZCe@e;eBfZDeeDeeCgeAfZEeeDeeCgeeAfZFGd&d'd'ZGd(e;d)dfd*d+ZHed,d-Gd.d/d/ZIed,d-Gd0d1d1ZJGd2d3d3eZKGd4d5d5eZLGd6d7d7eZMGd8d9d9eZNedureduree?ee?efZOeePd:<eeeDeAeAgeeAd'ffeKeOfZQeePd;<eeeDeAeAgeeeAd'ffeMeOfZReePd<<n,eeeDeAeAgeeAd'ffeKfZQeePd;<eeeDeAeAgeeeAd'ffeMfZReePd<<eeeeDeeAeeAe@e;eeAfgeAfeLfZSeeeeDeeAeeAe@e;eeAfgeeAfeNfZTd=ed)e>fd>d?ZUd=ed)e>fd@dAZVd=ed)e>fdBdCZWd=ed)e>fdDdEZXeduredurdqd=edGe>d)efdHdIZYd=ed)efdJdKZZndqd=edGe>d)efdLdIZYd=ed)efdMdKZZGdNdOdOeZ[GdPdQdQedFdRZ\GdSdTdTe\Z]GdUdVdVedFdRZ^GdWdXdXe^Z_GdYdZdZedFdRZ`Gd[d\d\e]e`ZaGd]d^d^eZbGd_d`d`eZcGdadbdbZdGdcddddeZeGdedfdfZfGdgdhdheZgGdidjdjZhGdkdldlZiGdmdndnZjd)e;fdodpZkdS)r)ABC)abstractmethodNdeepcopy) dataclass)field) TYPE_CHECKING)Any) Awaitable)Callable)Iterator)Optional)Sequence) TypedDict)Union)cast)overload) TypeAlias)BaseConversationalMetric) BaseMetric)config) ERROR_MSG) ERROR_STACK) ERROR_TYPE) get_logger)DD_SITES_NEEDING_APP_SUBDOMAIN)EXPERIMENT_CONFIG)EXPERIMENT_EXPECTED_OUTPUT)EXPERIMENT_RECORD_METADATA)convert_tags_dict_to_list) safe_json) __version__)LLMObs)LLMObsExperimentEvalMetricEvent)LLMObsExperimentsClient)ExportedLLMObsSpanJSONTypec@s\eZdZdZ    d dedeedeedeeeefdeeeefddf d d ZdS) EvaluatorResultaContainer for evaluator results with additional metadata. This class allows evaluators to return not just a value, but also reasoning, assessment, metadata, and tags alongside the evaluation result. Example:: def my_evaluator(input_data, output_data, expected_output): score = calculate_score(output_data, expected_output) return EvaluatorResult( value=score, reasoning="The output matches the expected format", assessment="pass" if score > 0.8 else "fail", metadata={"confidence": 0.95}, tags={"category": "accuracy"} ) Nvalue reasoning assessmentmetadatatagsreturncCs"||_||_||_||_||_dS)aInitialize an EvaluatorResult. :param value: The primary evaluation result (numeric, boolean, string, etc.) :param reasoning: Optional explanation of why this evaluation result was produced :param assessment: Optional categorical assessment (e.g., "pass", "fail", "good", "bad") :param metadata: Optional dictionary of additional metadata about the evaluation :param tags: Optional dictionary of tags to categorize or label the evaluation N)r(r)r*r+r,)selfr(r)r*r+r,r/N/home/ubuntu/.local/lib/python3.10/site-packages/ddtrace/llmobs/_experiment.py__init__Zs  zEvaluatorResult.__init__)NNNN) __name__ __module__ __qualname____doc__r&r strdictr1r/r/r/r0r'Gs&r'namer-cCs>t|ts td|stdtd|std|ddS)zValidate that evaluator name is valid. :param name: The evaluator name to validate :raises TypeError: If the name is not a string :raises ValueError: If the name is empty or contains invalid characters zEvaluator name must be a stringzEvaluator name cannot be emptyz^[a-zA-Z0-9_-]+$zEvaluator name 'zW' is invalid. Name must contain only alphanumeric characters, underscores, and hyphens.N) isinstancer6 TypeError ValueErrorrematch)r8r/r/r0_validate_evaluator_nameqs   r>T)frozenc@steZdZUdZeeefed<eed<dZe e ed<e edZ eeefed<dZ e eed<dZe eed <dS) EvaluatorContextawContext object containing all data needed for evaluation. This frozen dataclass wraps all metadata needed to run an evaluation, providing better state management and extensibility compared to individual parameters. :param input_data: The input data that was provided to the task (read-only). Dictionary with string keys mapping to JSON-serializable values. :param output_data: The output data produced by the task (read-only). Any JSON-serializable type. :param expected_output: The expected output for comparison, if available (read-only). Optional JSON-serializable type. :param metadata: Additional metadata including dataset record metadata and experiment configuration (read-only). Dictionary with string keys mapping to JSON-serializable values. :param span_id: The span ID associated with the task execution, if available (read-only). Optional string. :param trace_id: The trace ID associated with the task execution, if available (read-only). Optional string. input_data output_dataNexpected_outputdefault_factoryr+span_idtrace_id)r2r3r4r5r7r6r __annotations__rCr r&rr+rFrGr/r/r/r0r@s r@c@sheZdZUdZeeed<eeed<eeed<ee eefed<e edZ eee e fed<dS) SummaryEvaluatorContextaiContext object containing all data needed for summary evaluation. :param inputs: list of all input data from the dataset records (read-only). :param outputs: list of all outputs produced by the task (read-only). :param expected_outputs: list of all expected outputs (read-only). :param evaluation_results: Dictionary mapping evaluator names to their results (read-only). :param metadata: list of metadata for each dataset record, each combined with experiment configuration (read-only). Each element contains the record's metadata merged with {"experiment_config": ...}. inputsoutputsexpected_outputsevaluation_resultsrDr+N) r2r3r4r5listDatasetRecordInputTyperHr&r7r6rr+r r/r/r/r0rIs   "rIc @szeZdZdZddeefddZedede e e ffdd Z  dd ed eed ee eefde eeffd dZdS) BaseEvaluatoraThis class provides a unified interface for evaluators. Subclasses must implement the `evaluate` method. **Evaluator Return Values** LLM Observability supports storing and representing the following evaluator return value types: - **Numeric**: int/float values - **Boolean**: pass/fail boolean values - **Null**: None values - **JSON serializable**: string/dict/list values, which will be serialized into strings - **EvaluatorResult**: Any of the above values plus optional associated reasoning, assessment, metadata, and tags Example (simple return):: class SemanticSimilarityEvaluator(BaseEvaluator): def __init__(self, threshold=0.8): super().__init__(name="semantic_similarity") self.threshold = threshold self.model = load_embedding_model() def evaluate(self, context: EvaluatorContext): score = self.model.compare(context.output_data, context.expected_output) return score Example (with EvaluatorResult):: class SemanticSimilarityEvaluator(BaseEvaluator): def __init__(self, threshold=0.8): super().__init__(name="semantic_similarity") self.threshold = threshold self.model = load_embedding_model() def evaluate(self, context: EvaluatorContext): score = self.model.compare(context.output_data, context.expected_output) return EvaluatorResult( value=score, reasoning=f"Similarity score: {score:.2f}", assessment="pass" if score >= self.threshold else "fail", metadata={"threshold": self.threshold}, tags={"type": "semantic"} ) Note: The ``evaluate`` method may be called concurrently from multiple threads. Avoid modifying instance attributes inside ``evaluate()``; use local variables instead. Nr8cC,|dur |}n|jj}t|||_dS)zInitialize the evaluator. :param name: Optional custom name for the evaluator. If not provided, the class name will be used. Name must contain only alphanumeric characters and underscores. Nstrip __class__r2r>r8r.r8r/r/r0r1   zBaseEvaluator.__init__contextr-cCtd)agPerform evaluation. This method must be implemented by all subclasses. :param context: The evaluation context containing input, output, and metadata :return: Evaluation results - can be a JSONType value (dict, primitive, list, None) or an EvaluatorResult object containing the value plus additional metadata -Subclasses must implement the evaluate methodNotImplementedErrorr.rWr/r/r0evaluate zBaseEvaluator.evaluateml_app eval_namevariable_mappingcCrX)Nz+This evaluator does not support publishing.rZ)r.r_r`rar/r/r0_build_publish_payloadsz$BaseEvaluator._build_publish_payloadN)NN)r2r3r4r5r r6r1rr@rr&r'r]r7r rbr/r/r/r0rPs . rPc@:eZdZdZd deefddZedede fdd Z dS) BaseSummaryEvaluatora"Base class for summary evaluators that operate on aggregated experiment results. Summary evaluators receive all inputs, outputs, expected outputs, and per-row evaluation results at once, allowing them to compute aggregate metrics. Subclasses must implement the `evaluate` method. Example:: class AverageScoreEvaluator(BaseSummaryEvaluator): def __init__(self, target_evaluator: str): super().__init__(name="average_score") self.target_evaluator = target_evaluator def evaluate(self, context: SummaryEvaluatorContext): scores = context.evaluation_results.get(self.target_evaluator, []) if not scores: return None return sum(scores) / len(scores) Nr8cCrQ)aInitialize the summary evaluator. :param name: Optional custom name for the evaluator. If not provided, the class name will be used. Name must contain only alphanumeric characters and underscores. NrRrUr/r/r0r1rVzBaseSummaryEvaluator.__init__rWr-cCrX)aPerform summary evaluation on aggregated experiment results. This method must be implemented by all subclasses. :param context: The summary evaluation context containing all inputs, outputs, expected outputs, and per-row evaluation results :return: Evaluation result as a JSON-serializable value (dict, primitive, list, None) rYrZr\r/r/r0r]+r^zBaseSummaryEvaluator.evaluaterc r2r3r4r5r r6r1rrIr&r]r/r/r/r0res rec@sBeZdZdZd deefddZedede e e ffdd Z dS) BaseAsyncEvaluatorz*Base class for async row-level evaluators.Nr8cCrQ)zInitialize the async evaluator. :param name: Optional custom name for the evaluator. If not provided, the class name will be used. Name must contain only alphanumeric characters and underscores. NrRrUr/r/r0r1;rVzBaseAsyncEvaluator.__init__rWr-c td)zPerform async evaluation.rYrZr\r/r/r0r]JzBaseAsyncEvaluator.evaluaterc) r2r3r4r5r r6r1rr@rr&r'r]r/r/r/r0rg8s  rgc@rd) BaseAsyncSummaryEvaluatorzVBase class for async summary evaluators that operate on aggregated experiment results.Nr8cCrQ)aInitialize the async summary evaluator. :param name: Optional custom name for the evaluator. If not provided, the class name will be used. Name must contain only alphanumeric characters and underscores. NrRrUr/r/r0r1SrVz"BaseAsyncSummaryEvaluator.__init__rWr-crh)zBPerform async summary evaluation on aggregated experiment results.rYrZr\r/r/r0r]briz"BaseAsyncSummaryEvaluator.evaluatercrfr/r/r/r0rjPs rj_DeepEvalListType EvaluatorTypeAsyncEvaluatorType evaluatorcC t|tS)zCheck if an evaluator is a class-based evaluator (inherits from BaseEvaluator). :param evaluator: The evaluator to check :return: True if it's a class-based evaluator, False otherwise )r9rPrnr/r/r0_is_class_evaluator rqcCs(tdustdur dSt|tpt|tS)zCheck if an evaluator is a deep eval evaluator (inherits from BaseMetric or BaseConversationalMetric). :param evaluator: The evaluator to check :return: True if it's a class-based deepeval evaluator, False otherwise NF)rrr9rpr/r/r0_is_deep_eval_evaluatorsrscCro)zCheck if an evaluator is a class-based summary evaluator (inherits from BaseSummaryEvaluator). :param evaluator: The evaluator to check :return: True if it's a class-based summary evaluator, False otherwise )r9rerpr/r/r0_is_class_summary_evaluatorrrrtcCs"t|t ot|t ot| S)zCheck if an evaluator is a function-based evaluator. :param evaluator: The evaluator to check :return: True if it's a function evaluator, False otherwise )r9rPrersrpr/r/r0_is_function_evaluators  ruFis_asyncc sNddlm d dtttfdtdttdtffdd }td d |_ |S) zWrapper to run deep eval evaluators and convert their result to an EvaluatorResult. :param evaluator: The deep eval evaluator to run :return: A callable function that can be used as an evaluator r LLMTestCaseNrArBrCr-c sXt|t|t|d}|j}j}jrdnd}j}t||||d}|S)aOWrapper to run deep eval evaluators and convert their result to an EvaluatorResult. :param input_data: The input data :param output_data: The output data :param expected_output: The expected output :return: An EvaluatorResult containing the score, reasoning, and assessment input actual_outputrCpassfailr(r)r*r+)r6measurescorereasonsuccessscore_breakdownr' rArBrCdeepEvalTestCaserr)r*r+ eval_resultrxrnr/r0wrapped_evaluators"  z7_deep_eval_evaluator_wrapper..wrapped_evaluatorr8deep_eval_evaluatorrc deepeval.test_caserxr7r6r r r&r'getattrr2)rnrvrr/rr0_deep_eval_evaluator_wrappers  rc sNddlm d dtttfdtdttdtffdd }td d |_ |S) zTSync factory that returns an async callable for use with await in async experiments.rrwNrArBrCr-c s`t|t|t|d}|IdHj}j}jr!dnd}j}t||||d}|S)Nryr|r}r~)r6 a_measurerrrrr'rrr/r0rs$z=_deep_eval_async_evaluator_wrapper..wrapped_evaluatorr8rrcr)rnrr/rr0"_deep_eval_async_evaluator_wrappers  rcC|S)zDummy wrapper; should never be called but used to satisfy type checking. :param evaluator: The deep eval evaluator to run :return: A callable function that can be used as an evaluator r/)rnrvr/r/r0rscCr)zHDummy wrapper; should never be called but used to satisfy type checking.r/rpr/r/r0r#sc@seZdZUeed<eed<dS)Projectr8_idNr2r3r4r6rHr/r/r/r0r(s  rc@eZdZUeeed<dS)_DatasetRecordRawOptionalr,N)r2r3r4rNr6rHr/r/r/r0r- r)totalc@s.eZdZUeed<eed<eeefed<dS)DatasetRecordRawrArCr+N) r2r3r4rOrHr&r7r6r r/r/r/r0r1s rc@s:eZdZUeed<eed<eeefed<e eed<dS)_UpdatableDatasetRecordOptionalrArCr+r,N) r2r3r4rOrHr&r7r6r rNr/r/r/r0r7s rc@eZdZUeed<dS)UpdatableDatasetRecord record_idNrr/r/r/r0r> rc@r)_DatasetRecordOptional canonical_idN)r2r3r4r r6rHr/r/r/r0rBrrc@r) DatasetRecordrNrr/r/r/r0rFrrc@sZeZdZUeed<eed<eed<eed<eed<eeefed<eeeefed<dS) TaskResultidxrFrG timestampoutputr+errorN) r2r3r4intrHr6r&r7r r/r/r/r0rJs rc@s.eZdZUeed<eeeeeffed<dS)EvaluationResultr evaluationsN)r2r3r4rrHr7r6r&r/r/r/r0rTs rc@seZdZdefddZdS)_ExperimentRunInforun_interationcCst|_|d|_dS)N)uuiduuid4r_run_iteration)r.rr/r/r0r1Zs z_ExperimentRunInfo.__init__N)r2r3r4rr1r/r/r/r0rYsrc@seZdZUeed<eeed<eed<eed<eed<eeefed<e ed<e ed<eeeee ffed <eee fed <eeeefed <d S) ExperimentRowResultrrrFrGrrzrrCrr+rN) r2r3r4rrHr r6r7NonNoneJSONTyper&r/r/r/r0r`s  rc@s6eZdZdedeeeeeffdeefddZ dS) ExperimentRunrunsummary_evaluationsrowscCs(|j|_|j|_|p i|_|pg|_dSrc)rrun_idr run_iterationrr)r.rrrr/r/r0r1os zExperimentRun.__init__N) r2r3r4rr7r6r&rNrr1r/r/r/r0rnsrc@s>eZdZUeeeeeffed<eeed<ee ed<dS)ExperimentResultrrrunsN) r2r3r4r7r6r&rHrNrrr/r/r/r0r{s  rc@s(eZdZUeed<eed<eeeed<eed<eeed<eed<eed<ded <e ee fed <e ee fed <eeed <d Z d;dede dedeedededed ddeeeddfddZ   dDatasetr8 description filter_tagsr_records_version_latest_versionr$ _dne_client_new_records_by_record_id!_updated_record_ids_to_new_fields_deleted_record_idsiPNproject dataset_idrecordslatest_versionversionr-c CsP||_||_||_| p g|_||_||_||_||_||_i|_ i|_ g|_ dSrc) r8rrrrrrrrrrr) r.r8rrrrrrrrr/r/r0r1s   zDataset.__init__T deduplicatecreate_new_version bulk_uploadcCs||||dS)aPushes any local changes in this dataset since the last push. :param deduplicate: Wether to deduplicate the records or not. Does not deduplicate against existing data if bulk_upload is False. :param create_new_version: Whether to create a new version of the dataset when changes are detected, or update the existing version. :param bulk_upload: - True: Uploads all records in a single request. This method does not support deduplication against existing data and is best suited for initial uploads. - False: Splits the data into batches and uploads them individually. This method supports deduplication against existing records but does not provide transactional guarantees when the same dataset is modified concurrently by multiple clients. - None: The SDK chooses between the above two approaches using data size. N)_push)r.rrrr/r/r0pushsz Dataset.pushc CsN|jstd|jstdd}|}|s|dur1||jkr1td||jj|j|j|dnntd|t |j }|jj |j|j dt |j ||j||d\}}} t |j} t| || D]\} } } | |j| d <| rz| |j| d <|j| =qet|d kpt|jd k}|d kr||_td ||j|j|_g|_i|_ |S)Nz}Dataset ID is required to push data to Experiments. Use LLMObs.create_dataset() or LLMObs.pull_dataset() to create a dataset.zLLMObs client is required to push data to Experiments. Use LLMObs.create_dataset() or LLMObs.pull_dataset() to create a dataset.Fz&dataset delta is %d, using bulk upload)rz'dataset delta is %d, using batch updater)r project_idinsert_recordsupdate_recordsdelete_record_idsrrrrrz new_version %d latest_version %d)rr;r_estimate_delta_sizeBATCH_UPDATE_THRESHOLDloggerdebugdataset_bulk_uploadrrNrvaluesdataset_batch_updaterrrkeysziplenrr)r.rrr data_changed delta_sizeupdated_records new_versionnew_record_idsnew_canonical_ids pending_keyskeyrrr/r/r0rsT    z Dataset._pushindexrecordcsxtfdddDrtd|j|d}i|j|d|id|i|j|<i|j|d|i|j|<dS)Nc3s|]}|vVqdSrcr/).0krr/r0 sz!Dataset.update..)rArCr+zhinvalid update, record should contain at least one of input_data, expected_output, or metadata to updater)allr;rrget)r.rrrr/rr0update s& zDataset.updatecCs6tj}i||dd}||j|<|j|dS)N)rr)rrhexrrappend)r.rrrr/r/r0rs  zDataset.appendcCs|D]}||qdSrc)r)r.rrr/r/r0extend%s zDataset.extendcCsz|j|d}d}|j|=|dus|dkrtd|dS||jvr&|j|=||jvr1|j|=d}|r;|j|dSdS)NrTz/encountered unexpected record_id on deletion %sF)rrwarningrrrr)r.rrshould_append_to_be_deletedr/r/r0delete)s   zDataset.deletecCtd|jS)Nz/llm/datasets/ _get_base_urlrr.r/r/r0url=z Dataset.urlcC|jSrc)rrr/r/r0rBzDataset.latest_versioncCrrc)rrr/r/r0rFrzDataset.versioncCs,tt|jtt|j}td||S)zQrough estimate (in bytes) of the size of the next batch update call if it happenszestimated delta size %d)rr rrrr)r.sizer/r/r0rJs zDataset._estimate_delta_sizecCdSrcr/r.rr/r/r0 __getitem__PzDataset.__getitem__cCrrcr/rr/r/r0rSrcCs |j|Src)rrrr/r/r0rV cC t|jSrc)rrrr/r/r0__len__Y zDataset.__len__cCrrc)iterrrr/r/r0__iter__\rzDataset.__iter__c szddl}Wnty}ztd|d}~wwt}g}|jD]}i|di}t|trF|D]\}}|d|f<|d|fq3n |d<|d|di} t| trq| D]\} } | d| f<|d| fq^n | d<|d|di} t| tr| D]\} }|d| f<|d| fqnt dt | | q g}|D]fd d |D}| |q|j ||j|d S) Nrz[pandas is required to convert dataset to DataFrame. Please install via `pip install pandas`rA)rArrC)rCrr+zunexpected metadata format %scsg|]}|dqSrc)r)rcol flat_recordr/r0 sz(Dataset.as_dataframe..)datacolumns)pandas ImportErrorsetrrr9r7itemsaddrrtyper DataFrame MultiIndex from_tuples)r.pde column_tuples data_rowsrrAinput_data_colinput_data_valrCexpected_output_colexpected_output_valr+ metadata_col metadata_val records_listrowr/r r0 as_dataframe_sV               zDataset.as_dataframerc)TTN)r-N)#r2r3r4r6rHr rNrrr7rrrrr1boolrrrrrrpropertyrrrrrrslicerrr rr#r/r/r/r0rs         E &rc@steZdZUdZeeefed<eee e fed<eee e fed<e ddZ       dod ed eeefd ed eee e fdededeeeefdeededdeeee e fdeedeeddfddZedefddZdedeedeedeeedef d d!Z "    dpd#ed$ed%ed&ed'ed(ed)ed*eed+eed,eeeefdeeeefdd-fd.d/Z d0eded-fd1d2Z!d3eedefd4d5Z"d6e#deee$ffd7d8Z%deed9eede&ee'eeeeeeee$feeeefffd:d;Z(dqdd?Z)drdAedBeddfdCdDZ* E  F dsdGedHed3eedIedJee+ege,fde-f dKdLZ.e/dMede-fdNdOZ0dPe-defdQdRZ1e/dPe-defdSdTZ2dPe-ddfdUdVZ3dFdWdXfdYe&ee4fdedZe5j6dIedJe+ege,fdeef d[d\Z7d ddFd]dXfdGededHed3eedIedJe+ege,fdeefd^d_Z8d dEdFd`dXfdeedHedGedIedJe+ege,fdeef dadbZ9 Edtdeed9eedHedGedeef dcddZ: e FdudGedHedfeede-fdgdhZ;       dvd#ed$ediedjdkeedleed*eed+eed,eeeefdeeeefddfdmdnZ.z[Test subset of {} records] {})r8rrrrrrr) rr7rformatr8rrrrrrr)r.resubset_records subset_namer/r/r0_get_subset_dataset]s zExperiment._get_subset_datasetexccCsHt\}}}|durt|jnd}dt|||}t|||dS)NzUnknown Exceptionr)messagerstack)sysexc_inforr2join tracebackformat_exceptionr6)r.rkexc_type exc_valueexc_tb exc_type_name exc_stackr/r/r0_build_evaluator_errornsz!Experiment._build_evaluator_error eval_resultscCsg}g}g}g}i}t|D]Q\}} || d|j|} || d|| d| di} |i| d|ji||d} | D]\} }| |vrTg|| <|| |dqHq|||||fS)NrrArCr+experiment_configrr()rCrr7rr:r)r.rBryrJrKrL metadata_listeval_results_by_namerrErrecord_metadataeval_result_at_idx_by_namer8rMr/r/r0_prepare_summary_evaluator_dataxs&    z*Experiment._prepare_summary_evaluator_datastatusrcCsX|jr|jsdSz|jjjtt|j||dWdSty+tjd|ddYdSw)N)rrz(Failed to update experiment status to %sT)ro) r1rrexperiment_updaterr6 Exceptionrr)r.rrr/r/r0_update_statuss " zExperiment._update_statusTllmobs_not_enabled_error ensure_uniquec Cs|jr|jjs t||jj|j}|dd|_|j|jd<|jj |j |j j |j|j j |jt|j|j|j| \}}||_ t||jd<||_dS)NrrrrZ)r1enabledr;rproject_create_or_getr>rr?r9experiment_creater8r7rrr:rr8r;r6r@)r.rrrrZexperiment_run_namer/r/r0_setup_experiments,   zExperiment._setup_experiment rjobs raise_errors max_retries retry_delayc sdtdtfdd}|dur|}nt|stdt|j|dkr'td|d kr/td |d g|_ |j d |_ d |_ |d zzht|jD]`}t|}t|j|jd<t|j|jd<|j||||||dIdH} |j| ||||dIdH} |j| | ||dIdH} ||| | | } || } |jjtt|j| t|j|j | qNWnt!yd|_ |dwW|"|j }|#|n |"|j }|#|w|j r|jd|$|d|S|d|S)a/Run the experiment by executing the task on all dataset records and evaluating the results. :param jobs: Maximum number of concurrent task and evaluator executions (default: 10) :param raise_errors: Whether to raise exceptions on task or evaluator errors (default: False) :param sample_size: Optional number of dataset records to sample for testing (default: None, uses full dataset) :param max_retries: Maximum number of retries for failed tasks and evaluators (default: 0) :param retry_delay: Callable that takes the attempt number (0-based) and returns the delay in seconds before the next retry. Default: ``0.1 * (attempt + 1)`` :return: ExperimentResult containing evaluation results and metadata attemptr-cS d|dSNg?rr/rr/r/r0_default_retry_delayrz,Experiment.run.._default_retry_delayNz&retry_delay must be a callable, got {}rzjobs must be at least 1rzmax_retries must be >= 0zLLMObs is not enabled. Ensure LLM Observability is enabled via `LLMObs.enable(...)` and create the experiment via `LLMObs.async_experiment(...)` before running the experiment.Frunningrrrr)rrrr)rT interruptedfailed)r completed)%rr[callabler:rgrr2r;r _run_resultsr=clear _interrupted _has_errorsrranger;rr6rr9r _run_task_run_evaluators_run_summary_evaluatorsrKrdr1rexperiment_eval_postrrr BaseException _build_result_log_experiment_summary_build_error_summary)r.rrrerrrrrrBrrH run_resultexperiment_evalsresultr/r/r0rsl              zExperiment.run run_resultscCs.|r|djni|r|dj|dSg|dS)Nrrrr)rr)rr/r/r0rs zExperiment._build_resultrc Cs|dg}g}|jr|d|jt||jt|D]\}}|j}|jdkr2d|d|jnd}t dd|D}i} |D]3} | dpIi D]'\} } | | d d d } | d d7<t | t rs| d rs| d d7<qLqAd|j}|r|d|7}|d|t|t| |r|d|t|| D](\}} | d r|d|| d t|| d q|d|| d t|qq|jr|dt|jd|jd|S)Nrz1Experiment '{}' was interrupted after {}/{} runs.rz Run {}/{}rcss2|]}t|dtr|ddrdVqdS)rrlrN)r9rr7)rr"r/r/r0rs  z8Experiment._format_experiment_summary..rr)rerrorsrrrzExperiment '{}'z - {}z{}: {} rows, {} evaluator(s).z Task errors: {}/{}z" {}: {}/{} evaluated, {} error(s)z {}: {}/{} evaluatedzRetries ({}): {}z  )rrrrgr8rr;rCrsumr setdefaultr9r7r=rp)r.rrpartsrun_idxrr run_labeltask_error_count eval_statsr"r8r statsheaderr`r/r/r0_format_experiment_summary sJ     z%Experiment._format_experiment_summaryc Csg}|dgD]Y}|jD]S}|d}t|tr-|dr-|d|dd|d|dp3iD])\}}t|ts@q6|d}t|tr_|dr_|d||dd|dq6q q|sfd Sd t|S) Nrrrlz{}: {}rErrorrz {} ({}): {}z unknown errorz; ) rrr9r7rrgrrpr)rrrr"rNr8r eval_errr/r/r0r6s$      zExperiment._build_error_summarycCs8||}|s dS|jrtjntj}||ddiddS)Nproductllmobs)extra)rrrrinfo)r.rmsglog_fnr/r/r0rHs z"Experiment._log_experiment_summarycCrrr/rr/r/r0U zExperiment. idx_record semaphorecsF|jr|jjs dS|4IdH|\}}|jj|jj|jt|j|j|jj |j |j |j dP}|jj |d} |j r?| |_| rN| dd} | dd} nd\} } |d} |d d} |d }i|jt|jjt| t|jd }|d g}|D]}d |vr|d d\}}|||<q}|r||d<d}d}td|D]p}z"t|jr|| |jIdH}n t|j| |jIdH}d}WnLty}z?t}||kr|jd||d|d||zt||IdHW|IdHn|IdHwWYd}~qd}~ww|dkrt||d<|r'd|_ |j!||jj"|| ||d|#t$|dd|vrF|#t%|d|jrQ|#t&|j|| | |j'|||j |jj d|(t)|(t*|(t+ddWdWdIdHS1swYWdIdHdS1IdHswYdS)z%Process single record asynchronously.N)r8rZrrr4r0rr5)spanrFrrG)rrrArr)rdataset_record_idrZr,:rdataset_record_canonical_idz%task row {}: attempt {}/{} failed: {}rretriesT)rArBr,rCr+)dataset_record_indexr5r4)rlrmr)rrFrGrrr+r),r1r _experimentr(r2rr6rr7r8r>r? export_spanr<rArr9splitrasyncioiscoroutinefunctionr: to_threadrrnror=rrgreleasesleepacquirer set_exc_infoannotate _set_ctx_itemrrrstart_nsget_tagrrr)r.rrrrrrrr span_contextrFrGrArrr, record_tagstagrr(rB last_exc_inforrr/r/r0_process_recordOs             0zExperiment._process_recordcCrrr/rr/r/r0rrcsjrjjs gS|}t|fddt|D}tj|ddiIdH} g} | D]A} t| tr>|r=| q2| sAq2| } | | | dpNi} t| t rs| d}| d}| d}|rs|rst d | d |||q2j| S) Nc s g|] }j|dqS)r)r)rrrrrr.rr/r0r sz(Experiment._run_task..return_exceptionsTrrlrmrzError on record {}: {} {} {}r)r1rrjr SemaphorerCgatherr9rrrr7 RuntimeErrorrgflush)r.rrrrerrsubset_datasetcorosresultsrBrrEerr_dicterr_msg err_stackerr_typer/rr0rs>          zExperiment._run_taskcCrrr/rr/r/r0rrc st|dtdtdtttttffffdd fddt|D}tj|IdH}g}t|D] \} } | | | dq8|S) NrrEr-cs4IdHgj|}|d}|d}|d}|di}i}jD]<}d} d} d} tdD]#} d} d} zt|trf|j} i|dji} t|||| |d|d d }| |IdH}n\t |rx|j } ||||IdH}nJt |r|j} i|dji} t|||| |d|d d }t |j |IdH}nt|r|j } t ||||IdH}n td |t|} d}i}t|tr|jr|j|d <|jr|j|d <|jr|j|d<|jr|j|d<|j} n|} WngtyX}zZi}|} | kr=jd| || dd|zt | IdHW IdHn IdHwWYd}~q4d_!rNt"d| d||WYd}~q4d}~ww| | d||| <q%|WdIdHS1IdHsvwYdS)NrArrCr+rrrzrFrG)rArBrCr+rFrGzHEvaluator %s is neither a BaseEvaluator instance nor a callable functionr)r*r,z/evaluator '{}' row {}: attempt {}/{} failed: {}Tz Evaluator z failed on row r(r)#r7rr)rr9rgr8r:r@r]rrr2rqrrurrr6r'r)r*r+r,r(rrxr=rrgrrrrr)rrErrArBrCr+ row_resultsrneval_result_valuerevaluator_namercombined_metadatarWrextra_return_valuesr)rrrr.rr/r0 _evaluate_rows                  2z1Experiment._run_evaluators.._evaluate_rowcsg|] \}}||qSr/r/)rrrE)rr/r0r Ysz.Experiment._run_evaluators..rr) rrrrr7r6r&rCrr) r.rBrrrrrrrrrr/)rrrrr.rr0rs 2qzExperiment._run_evaluatorsc s||\t|dtdtttttffffdd fddjD}tj |d iIdH}g}i}t |D]%\} } t | t rSqIt tttttff| \} } | || <|| |dqI|S) Nsummary_evaluatorr-c sZ4IdHd}d}d}z[t|tr)|j}td}||IdH}n=t|r<|j}|IdH}n*t|rW|j}td}t |j|IdH}n|j}t |IdH}|}Wn%t y}zd_  |}rt d|d|WYd}~nd}~ww|||dfWdIdHS1IdHswYdS)Nr)rJrKrLrMr+TzSummary evaluator z failedr)r9rjr8rIr]rrr2rtrrrrxr)rrrrrWrr)r|rLrJr{rKrr.rr/r0_evaluate_summary_singlessh     0zDExperiment._run_summary_evaluators.._evaluate_summary_singlecsg|]}|qSr/r/)rr)rr/r0r rfz6Experiment._run_summary_evaluators..rr)rrrr tupler6r7r&r*rrCr9rrr) r.rBryrrrrr evals_dictrrrrJr/) rr|rLrJr{rKrr.rr0rbs2  8 z"Experiment._run_summary_evaluatorsrrc st|pd}t|j|jd<t|j|jd<||||dIdH}|j|||dIdH}||||g}||}|j j t t|j|t |jig|gdS)Nrrr)rrr)rr6rr9rrrrKrdr1rrrr) r.rrrrrBrrrr/r/r0_run_task_single_iterations  z%Experiment._run_task_single_iterationrr%rUis_summary_evalc Cs|jstd|dur$t|tr t|dtr t|dts$td|dur3|s3|jdur3td|dur<|s<|j}|durFt|dntt d} | ||d|r[|dd nd |rd|dd nd | |rjd nd |||| } |j j tt|j| gt|jdS) aSubmit an evaluation metric for a distributed experiment. :param eval_name: Name of the evaluation metric :param eval_value: Value of the evaluation metric :param span: Optional span context dict with span_id and trace_id. If None and not a summary eval, uses the last span from _run_task_single_iteration. :param timestamp_ms: Optional timestamp in milliseconds :param is_summary_eval: Whether this is a summary-level evaluation :param reasoning: Optional reasoning string :param assessment: Optional assessment string :param metadata: Optional metadata dict :param tags: Optional tags dict z4this method is only used for distributed experimentsNrFrGz`span` must be a dictionary containing both span_id and trace_id keys. LLMObs.export_span() can be used to generate this dictionary from a given span.zCunexpected state, must supply span or must run the experiment firstrVgeArr`rL)r<r;r9r7rr6r:rArtimer^r1rrrrrr9) r.r`rMrrUrr)r*r+r,rOr]r/r/r0_submit_eval_metricsB $ zExperiment._submit_eval_metric)rNNNNNF)rLNNNNrc)T)rFNrN)Fr)rFr)NNNNNNN)=r2r3r4r5rTaskType AsyncTaskTyperHrrlrmSummaryEvaluatorTypeAsyncSummaryEvaluatorType classmethodr,r6rr r7 ConfigTyperr$r1r%rrrNrrrrKr&r^rdrjrr rxrrOrrrr r[rr staticmethodrrrrrrrrrrrrrr/r/r/r0r's           /  -     - 4   K*    h  .    Z    r'c@seZdZdZ      ddedeeefdede ee e fded ed e e eefd e ed e d de e eeefde eddfddZ     d dedede edede eegefdef ddZedefddZdS)!SyncExperimentzThin synchronous wrapper around the async-native ``Experiment``. Provides a blocking ``run()`` method for callers that do not have an event loop. rNr8r-r.r/r0rr,rr1r"r2rr-c Cs$t||||||||| | | d |_dS)N) r8r-r.r/r0rr,rr1r2r)r'r) r.r8r-r.r/r0rr,rr1r2rr/r/r0r1s zSyncExperiment.__init__rFrrrrerrc Cs|jj|||||d}ztWntyt|YSwddl}|jjdd}|tj|} | WdS1sAwYdS)aRun the experiment synchronously. :param jobs: Maximum number of concurrent task and evaluator executions (default: 1) :param raise_errors: Whether to raise exceptions on task or evaluator errors (default: False) :param sample_size: Optional number of dataset records to sample for testing (default: None, uses full dataset) :param max_retries: Maximum number of retries for failed tasks and evaluators (default: 0) :param retry_delay: Callable that takes the attempt number (0-based) and returns the delay in seconds before the next retry. Default: ``0.1 * (attempt + 1)`` :return: ExperimentResult containing evaluation results and metadata )rrrerrrNr) max_workers) rrrget_running_looprconcurrent.futuresfuturesThreadPoolExecutorsubmitr) r.rrrerrcoro concurrentpoolfuturer/r/r0r3s"  $zSyncExperiment.runcCs|jjSrc)rrrr/r/r0rXszSyncExperiment.url)rNNNNN)rFNrN)r2r3r4r5r6rrrrrrlrmr r7rrrrr1r$r r[rrr%rr/r/r/r0rsj        %rcCs"d}tjtvr d}d|tjS)Nrzapp.zhttps://)r_dd_siter) subdomainr/r/r0r]s r)F)labcrrrcopyr dataclassesrrr<rnrrqtypingrr r r r r rrrrrrrtyping_extensionsrdeepeval.metricsrrddtracerddtrace.constantsrrrddtrace.internal.loggerrddtrace.llmobs._constantsrrrrddtrace.llmobs._utilsrr ddtrace.versionr!ddtrace.llmobsr"ddtrace.llmobs._writerr#r$ddtrace.llmobs.typesr%r2rr6rr[r$rNr7r&rrrOrrr'r>r@rIrPrergrjrkrHrlrmrrrqrsrtrurrrrrrrrrrrrrrrrr'rrr/r/r/r0sR                                  "   *S2              )   L