o i@sdZddlmZddlmZddlZddlZddlmZddlm Z ddlm Z ddlm Z dd lm Z dd lm Z dd lmZdd lmZdd lmZddlmZddlmZddlmZddlmZddlmZddlmZddlmZddlmZddlmZerddlmZeeZ ddZ!ddZ"ddZ#ddZ$d d!Z%d"d#Z&d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2Z'd3Z(Gd4d5d5e d6d7Z)eGd8d9d9Z*Gd:d;d;Z+Gdd?d?Z-dS)@zDPrompt optimization framework for iteratively improving LLM prompts.deepcopy) dataclassN) TYPE_CHECKING)Any)Callable)Optional)Sequence) TypedDict)Union) get_logger) BaseEvaluator) ConfigType)Dataset) DatasetRecord)DatasetRecordInputType) EvaluatorType)ExperimentResult)ExperimentRowResult)JSONType)SummaryEvaluatorType)LLMObscCs<t|stdt|}|j}d|vsd|vrtddS)zzValidate the task function signature. :raises TypeError: If task is not callable or missing required parameters. z!task must be a callable function. input_dataconfigz=Task function must have 'input_data' and 'config' parameters.Ncallable TypeErrorinspect signature parameters)tasksigparamsr#W/home/ubuntu/.local/lib/python3.10/site-packages/ddtrace/llmobs/_prompt_optimization.py validate_task#s r%cCsDt|stdt|}|j}d|vsd|vsd|vr tddS)zValidate the optimization_task function signature. :raises TypeError: If optimization_task is not callable or missing required parameters. z.optimization_task must be a callable function. system_prompt user_promptrzoptimization_task function must have 'system_prompt' and 'user_prompt' parameters. It should call an LLM with these prompts and return an optimized prompt.Nr)optimization_taskr!r"r#r#r$validate_optimization_task0s r)cCst|ts tddS)zuValidate that dataset is an LLMObs Dataset object. :raises TypeError: If dataset is not a Dataset instance. z)Dataset must be an LLMObs Dataset object.N) isinstancerr)datasetr#r#r$validate_dataset@s r,cCs"|dur t|tstddSdS)ztValidate the test_dataset parameter type. :raises TypeError: If test_dataset is provided but not a string. Nz-test_dataset must be a dataset name (string).)r*strr) test_datasetr#r#r$validate_test_datasetIsr/cCst|tsdStdd|Dstddt|kr dks*ntdt|t|dkr8|dur8td t|d krF|durFtd t|d vrPtd dS)zValidate dataset_split parameter when it is a tuple of ratios. :raises ValueError: If tuple ratios are invalid, don't sum to 1.0, or are incompatible with the test_dataset parameter. Ncss6|]}t|ttfod|kodknVqdS)rN)r*intfloat).0vr#r#r$ Zs4z)validate_dataset_split..z@dataset_split ratios must be floats between 0 and 1 (exclusive).gGz?g)\(?z1dataset_split ratios must sum to 1.0, got {:.4f}.zCannot use a 3-tuple dataset_split with test_dataset. Use a 2-tuple (train, valid) when providing a separate test dataset, or a 3-tuple (train, valid, test) without test_dataset.z{A 2-tuple dataset_split requires test_dataset. Use a 3-tuple (train, valid, test) to split without a separate test dataset.)r7r6z.dataset_split tuple must have 2 or 3 elements.)r*tupleall ValueErrorsumformatlen) dataset_splitr.r#r#r$validate_dataset_splitRs"  r?csn|std|D],}t|trqt|stdt|}|jd}tfdd|Ds4td|qdS)zqValidate the list of evaluators. :raises TypeError: If evaluators is empty or contains invalid entries. zUEvaluators must be a non-empty list of BaseEvaluator instances or callable functions.zBEvaluator must be a BaseEvaluator instance or a callable function.)r output_dataexpected_outputc3s|]}|vVqdSNr#)r3paramr"r#r$r5|sz&validate_evaluators..z+Evaluator function must have parameters {}.N) rr*r rrrrr9r<) evaluators evaluatorr!evaluator_required_paramsr#rDr$validate_evaluatorsms  rHzADon't be afraid to be creative when creating the new instruction!z'Keep the instruction clear and concise.z?Make sure your instruction is very informative and descriptive.zNThe instruction should include specific details such as numbers or conditions.zBe more specific in your instructions. Instead of 'handle errors', specify exactly what types of errors and how to handle them.zEAdd concrete examples to illustrate the expected format and behavior.zZUse clear structure with numbered steps or bullet points to organize complex instructions.zIAdd explicit constraints and validation rules to prevent common mistakes.z`Provide more context about the domain and use case to help the model understand the task better.zBAddress edge cases and corner scenarios that might cause failures.zZBe explicit about output formatting requirements, including JSON structure and data types.zBUse simpler, clearer language to reduce ambiguity in instructions.zCInclude validation steps or self-checking mechanisms in the prompt.zHClearly state what aspects are most important when there are trade-offs.)creativesimple descriptionspecific specificityexamples structure constraintscontext edge_cases formattingclarity validation priorities*c@sZeZdZUdZeed<eed<eed<eed<eed<e ee ee ffed<eed<d S) IterationDataz)Data for a single optimization iteration. iterationpromptresultsscoreexperiment_urlsummary_evaluationstrain_experiment_urlN) __name__ __module__ __qualname____doc__r1__annotations__r-rr2dictrr#r#r#r$rXs  rXF)totalc@s*eZdZUdZeed<eed<eed<dS)TestPhaseResultz]Results from the final test phase of prompt optimization (when dataset splitting is enabled).r[r\r]N)r`rarbrcrrdr2r-r#r#r#r$rgs  rgc@seZdZdZdededededede ee ee fgefdd fd d Z defd d Z defddZdeedefddZdedefddZdefddZd S)OptimizationIterationzRepresents a single iteration in the prompt optimization process. Each iteration analyzes the current prompt's performance and suggests improvements. rYcurrent_promptcurrent_resultsr(rlabelization_functionreturnNcCs(||_||_||_||_||_||_dS)a@Initialize an optimization iteration. :param iteration: The iteration number (0-indexed). :param current_prompt: The current prompt being evaluated. :param current_results: Results from the previous experiment run. :param optimization_task: Function to generate prompt improvements. :param config: Configuration for the optimization task. :param labelization_function: Function to generate labels from individual results. Takes an individual result dict and returns a string label. N)rYrirj_optimization_task_config_labelization_function)selfrYrirjr(rrkr#r#r$__init__s  zOptimizationIteration.__init__c Cs|}|}z |j|||jd}Wn,ty?}z td|jtdt|j tdt |d}WYd}~nd}~ww|sLt d|j|j S|S)aRun the optimization task to generate an improved prompt. Follows the LLM-as-a-judge pattern: 1. Loads the optimization prompt template from _prompt_optimization.md 2. Builds user prompt with examples from evaluation results `` 3. Calls optimization_task (LLM) with system and user prompts 4. Returns improved prompt :return: The improved prompt string. )r&r'rz-Iteration %s: Failed to run optimization_taskzException type: %szException message: %sNzSIteration %s: optimization_task returned empty 'new_prompt', keeping current prompt) _load_system_prompt_build_user_promptrmrn ExceptionlogerrorrYtyper`r-warningri)rpr&r'improved_prompter#r#r$runs2   zOptimizationIteration.runc Csddl}ddl}|j|jt}|j|d}t|ddd }|}Wdn1s/wY|j d}d}|rLd d d d t |d d d g}| d|} g} d|j vrk|j d} | d| | d|tt} t| } | d| d| r| d | 7} | S)aLoad and prepare the optimization system prompt. Loads the template from _prompt_optimization.md and replaces placeholders. Adds evaluation model information and random tip at the end. :return: System prompt string with output format injected. rNz_prompt_optimization.mdrzutf-8)encodingevaluation_output_formatrr z$## Prompt Output Format RequirementszSThe optimized prompt must guide the LLM to produce JSON output with this structure:zH**If this output format is not clearly specified in the initial prompt**z)**add it as your first improvement step**z{{STRUCTURE_PLACEHOLDER}} model_namezK IMPORTANT: The improved prompt will be applied to this evaluation model: znConsider the capabilities, limitations, and characteristics of this specific model when optimizing the prompt.z **TIP: z**)osrandompathdirnameabspath__file__joinopenreadrngetr-replaceappendchoicelistTIPSkeys)rprr current_dir template_pathftemplate output_formatstructure_placeholderr&additional_parts eval_modeltip_keytip_textr#r#r$rssH     z)OptimizationIteration._load_system_promptindividual_resultsc Cs|r|jdur dSi}|D]}|t|}|r'||vr g||<|||q |s,dSt|dkr[s z5OptimizationIteration._add_examples..)keyz### r) rorerr=rvryitemsrrsorted_format_exampler) rprexamples_by_labelresultlabelrNlabel_examplesformatted_partsexampler#r#r$ _add_examples2s8    z#OptimizationIteration._add_examplesrc Csg}|d}|d||d}|r|d||d}|r*|d||d}|rO|D]\}}t|trNd|vrN|d |d |dq4d |S) zFormat an example for display in the user prompt. :param example: Example result dict. :return: Formatted string. inputzInput: rAzExpected Output: outputzActual Output: evaluations reasoningz Reasoning (z): r)rrr*rer) rprpartsrexpectedrr eval_name eval_datar#r#r$rbs  z%OptimizationIteration._format_examplec Csd|jdg}|j}|d}|rD|d|D]#\}}|di}t|tr>|D]\}}|d|d|q.q|d|d g} | rT||| d |} | S) aBuild user prompt with current prompt and evaluation examples. Includes: - Current prompt being optimized - Performance metrics - Examples from results (TP, TN, FP, FN if available) :return: User prompt string. zInitial Prompt: rr^zPerformance Metrics:valuez- z: rrrowsz ) rirjrrrr*rerr) rp prompt_partsr[ summary_evals_summary_metric_datar metric_name metric_datar final_promptr#r#r$rts"        z(OptimizationIteration._build_user_prompt)r`rarbrcr1r-rrrrrerrqr|rsrrrrrtr#r#r#r$rhs, ,:0rhc @seZdZdZ d!dededeededee ddf d d Z e defd d Z e dee fd dZe deefddZe defddZe dee fddZe deefddZe deefddZdeefddZdee fddZdeefddZdefdd ZdS)"OptimizationResultaResults from running a prompt optimization. Contains all iteration results and metadata about the optimization process. Example usage:: result = optimization.run() # Access best iteration print(f"Best prompt: {result.best_prompt}") print(f"Best score: {result.best_score}") print(f"Best iteration: {result.best_iteration}") # Access full history for iteration in result.get_history(): print(f"Iteration {iteration['iteration']}: {iteration['score']}") Nnameinitial_prompt iterationsbest_iteration test_phaserlcCs"||_||_||_||_||_dS)aInitialize optimization results. :param name: Name of the optimization run. :param initial_prompt: The starting prompt. :param iterations: list of results from each iteration (IterationData). :param best_iteration: Index of the iteration with best performance. :param test_phase: Results from the final test phase (when dataset splitting is enabled). N)rrrr _test_phase)rprrrrrr#r#r$rqs  zOptimizationResult.__init__cCs,|jr |jt|jkr|jS|j|jdS)z3Get the best performing prompt from all iterations.rZ)rrr=rrpr#r#r$ best_promptszOptimizationResult.best_promptcC*|jr |jt|jkr dS|j|jdS)z/Get the evaluation score of the best iteration.Nr\rrr=rr#r#r$ best_scorezOptimizationResult.best_scorecCr)z.Get the experiment URL for the best iteration.Nr]rrr#r#r$best_experiment_urlrz&OptimizationResult.best_experiment_urlcCs t|jS)zz8OptimizationResult.get_score_history..rrr#r#r$get_score_historyz$OptimizationResult.get_score_historycCr)ziGet list of prompts across all iterations. :return: list of prompts in iteration order. cSr)rZr#rr#r#r$r rz9OptimizationResult.get_prompt_history..rrr#r#r$get_prompt_historyrz%OptimizationResult.get_prompt_historyc CsDd|jd|jd|j|jdurd|jdndg}|jdur9|d|jd|jr9|d |j|jD]}|d |jkrU|d }|rS|d |nq<|jdurl|j d i}|rl|d ||d|jD](}|d }|d}|d}||jkrdnd}|d|d|dd||qtd |S)znGet a human-readable summary of the optimization results. :return: Formatted summary string. zOptimization: zTotal iterations: zBest iteration: Nz Best score: z.4fzBest score: N/Az Test score: zTest experiment: rYr^z% Best iteration summary evaluations: z Test set summary evaluations: z Score progression:r\r]z <- BESTrrz Iteration z (score: z): r) rrrrrrrrrrr) rplinesrYrtest_summary_evalsiter_numr\urlmarkerr#r#r$summarys8        $ zOptimizationResult.summaryrB)r`rarbrcr-rrXr1rrgrqpropertyrr2rrrrrrrrrrrr#r#r#r$rsD  rc#@seZdZdZ      d.dedeeeege fdeeeegefde d e e d ed ed e e d eeeeeeffgefdeeeeefgefdeddeeeefdedeeeeeeeffgefdeeeedffdee ddf"ddZdedeede fddZdefddZ d/d!edefd"d#Zd!edefd$d%Zd!edefd&d'Z  (d0d)ed*ed!edee d+edeeeff d,d-ZdS)1PromptOptimizationa,Iteratively optimize LLM prompts using experiments and evaluations. PromptOptimization runs a baseline experiment with an initial prompt, then iteratively improves the prompt based on evaluation results. Each iteration analyzes performance and generates improved prompt suggestions. NFrr r(r+rE project_namersummary_evaluators compute_scorerk_llmobs_instancertagsmax_iterationsstopping_conditionr>.r.rlcCs||_||_||_||_||_||_||_| |_| |_| pi|_ ||j d<| |_ | |_ |s1t dt |tr:d|vr>t d|d|_|d|_||_||_t|pV|du|_d|_t |tre||_dS|durnd|_dS|rud|_dSd|_dS) aInitialize a prompt optimization. :param name: Name of the optimization run. :param task: Task function to execute. Must accept ``input_data`` and ``config`` parameters. :param optimization_task: Function to generate prompt improvements. Must accept ``system_prompt`` (str), ``user_prompt`` (str), and ``config`` (dict). Must return the new prompt. :param dataset: Dataset to run experiments on. :param evaluators: list of evaluators to measure task performance. Can be either class-based evaluators (inheriting from BaseEvaluator) or function-based evaluators that accept (input_data, output_data, expected_output) parameters. :param project_name: Project name for organizing optimization runs. :param config: Configuration dictionary. Must contain: - ``prompt`` (mandatory): Initial prompt template - ``model_name`` (optional): Model to use for task execution - ``evaluation_output_format`` (optional): the output format wanted - ``runs`` (optional): The number of times to run the experiment, or, run the task for every dataset record the defined number of times. :param summary_evaluators: list of summary evaluators (REQUIRED). Can be either class-based evaluators (inheriting from BaseSummaryEvaluator) or function-based evaluators that accept (inputs: list, outputs: list, expected_outputs: list, evaluations: dict) and return aggregated metrics. :param compute_score: Function to compute iteration score (REQUIRED). Takes summary_evaluations dict from the experiment result and returns float score. Used to compare and rank different prompt iterations. :param labelization_function: Function to generate labels from individual results (Optional but highly valuable) Takes an individual result dict (with "evaluations" key) and returns a string label Used to categorize examples shown to the optimization LLM. Example: lambda r: "Very good" if r["evaluations"]["score"] >= 0.8 else "Bad" :param _llmobs_instance: Internal LLMObs instance. :param tags: Optional tags to associate with the optimization. :param max_iterations: Maximum number of optimization iterations to run. :param stopping_condition: Optional function to determine when to stop optimization. Takes summary_evaluations dict from the experiment result and returns True if should stop. :param dataset_split: Controls dataset splitting. Accepts: - ``False`` (default): No splitting, use full dataset for everything. - ``True``: Split with default ratios (60/20/20 without test_dataset, 80/20 with). - ``(train, valid, test)`` tuple: Custom 3-way split ratios. Must sum to 1.0. Cannot be combined with ``test_dataset``. - ``(train, valid)`` tuple: Custom 2-way split ratios. Must sum to 1.0. Requires ``test_dataset`` for the test set. :param test_dataset: Optional separate test dataset. When provided, the main dataset is split into train/valid (80/20) and this dataset is used for the final test. Implicitly enables dataset splitting. :raises ValueError: If required config parameters or compute_score are missing. rzconfig parameter is requiredrZz"config must contain a 'prompt' keyrN)g?皙?)g333333?rr)r_taskrm_dataset _evaluators_summary_evaluators_stopping_conditionro_compute_score_tagsr_max_iterationsr:r*re_initial_promptr _model_namern _test_datasetbool_dataset_split_enabled _split_ratiosr8)rprr r(r+rErrrrrkrrrrr>r.r#r#r$rqAs<B         zPromptOptimization.__init__ split_namerecordsc CsJtd|d|jj|jj|jjdd|D|jj|jj|jj|jjdS)anCreate a sub-dataset from a list of records. Follows the same pattern as Experiment._run_task for creating subset datasets. :param split_name: Name suffix for the sub-dataset (e.g. "train", "valid", "test"). :param records: list of DatasetRecord dicts to include. :return: A new Dataset instance with the specified records. [z] cSsg|]}t|qSr#r)r3r}r#r#r$rrz8PromptOptimization._make_sub_dataset..)rproject dataset_idrrKlatest_versionversion _dne_client) rrrr_idrK_latest_version_versionr)rprrr#r#r$_make_sub_datasets  z$PromptOptimization._make_sub_datasetcCsd|jdur tdt|j}tt}|||jdur:|jd}t |t |}|d|}||d}|j}n5t |}|jd|jd}} t ||} t || |} |d| }|| | }|| d} | d| }| d|} | d|}d| fd|fd|ffD]\}}t |dkrtd|d t |jd qt d t | t |t || ||fS) aSplit the dataset into train, valid, and test subsets. When ``test_dataset`` was provided, the main dataset is split into train/valid (80/20) and the external test dataset is used as-is. Otherwise, the main dataset is split three ways (60/20/20). Records are shuffled with a fixed seed for reproducibility. :return: tuple of (train_dataset, valid_dataset, test_dataset). :raises ValueError: If any split would be empty. Nz*_split_dataset called without split ratiosrr0testtrainvalidzDataset split 'z' is empty. Dataset has z) records, which is too few for splitting.z2Dataset split: %d train, %d valid, %d test records)rr:rrrRandom_DATASET_SPLIT_SEEDshufflerr1r=rrvinfo)rprrng train_ratio split_idx train_records valid_recordstest_dsn valid_ratio train_end valid_end test_recordstrain_dsvalid_dsrdsr#r#r$_create_split_datasetss>                 z)PromptOptimization._create_split_datasetsr0jobscCs>|jr|jjs tdtd|j|jr||S||S)a^Run the prompt optimization process. Executes a baseline experiment with the initial prompt, then iteratively improves the prompt based on evaluation results. When dataset splitting is enabled, train examples are used for the optimization LLM, valid scores rank iterations, and a final test experiment provides an unbiased score. :param jobs: Number of parallel jobs for experiment execution. :return: OptimizationResult containing all iteration results. :raises ValueError: If LLMObs is not enabled or dataset is too small for splitting. zLLMObs is not enabled. Ensure LLM Observability is enabled via `LLMObs.enable(...)` and create the optimization via `LLMObs.prompt_optimization(...)` before running.z Starting prompt optimization: %s) renabledr:rvr rr_run_with_split_run_without_split)rprr#r#r$r|s  zPromptOptimization.runc Csvg}d}d}d}d}d}t|j}||||\} } | di} || } ||| | | | d} || | p6d}|}| }td|td|j dD]e}t ||||j |j |j d}|}||||\}} |di} || }||||| | d} || td |td | |dur|dus||kr|}|}|}|}|jr|| rtd |nqIt|jt|j||d S) zRun optimization without dataset splitting (original behavior). :param jobs: Number of parallel jobs for experiment execution. :return: OptimizationResult containing all iteration results. rNr^)rYrZr[r\r]r^zBaseline score: %.3fr0rYrirjr(rrkz Iteration %s%s)Stopping condition met after iteration %s)rrrr)r-r_run_experimentrrrrvr rangerrhrmrnror|rrr)rprall_iterationsrrr best_resultsrYrirjr]rbaseline_scoreiteration_dataioptimization_iteration new_prompt new_results new_scorer#r#r$rsx            z%PromptOptimization._run_without_splitc Cs|\}}}g}d}d}d}d} d} t|j} |j| | ||dd\} } |j| | ||dd\}}|di}||}| | ||||| d}|||pMd}| }| } td |t d |j d D]v}t ||| |j |j |jd }|}|j||||dd\} } |j||||dd\}}|di}||}||||||| d}||td ||td ||dur|dus||kr|}|}|}| } |jr||rtd|nq`td||j||||dd\}}|di}||}td|t|jt|j||t|||ddS)aRun optimization with train/valid/test dataset splitting. Train examples are shown to the optimization LLM, valid scores rank iterations, and a final test experiment on the best prompt provides an unbiased score. :param jobs: Number of parallel jobs for experiment execution. :return: OptimizationResult containing all iteration results and test results. rNr)r+suffixrr^)rYrZr[r\r]r^r_r zBaseline score (valid): %.3fr0r!z Iteration %s (valid score: %.3f)r"r#z=Running final test experiment with best prompt (iteration %s)rzTest score: %.3f)r[r\r])rrrrr)rr-rr$rrrrvr r%rrhrmrnror|rrrrg)rprrrrr&rrrbest_train_resultsrYri train_results train_url valid_results valid_urlrr(r)r*r+r,r.rtest_urlrrr#r#r$rls                    z"PromptOptimization._run_with_splitrrrYrZr/c Cs|p|j}|dkr dnd|}|r|d|}|j|d}|j|B} |jd} d} | dur7t| tr7| } |jj|jd||j d||j |j |j | | d } | j d |d } | | jfS) aRun an experiment for a given iteration and prompt. :param iteration: The iteration number. :param prompt: The prompt to test. :param jobs: Number of parallel jobs. :param dataset: Optional dataset override. If not provided, uses ``self._dataset``. :param suffix: Optional suffix appended to the experiment name (e.g. "_train", "_valid"). :return: tuple of (experiment results dictionary, experiment URL). rbaseline iteration_r)rrZrunsNr)rrr+r rErrr8T) raise_errorsr)rrrnrr*r1r experimentrrrrrr|r)rprYrZrr+r/riteration_nameconfig_updatesexperiment_config runs_valueruns_intr:experiment_resultsr#r#r$r$s2      z"PromptOptimization._run_experiment)NNrNFN)r0)Nrr) r`rarbrcr-rrrrrrr rrrerr2r1rr r8rqrrrrrr|rrrr$r#r#r#r$r9s       i6 ]w r).rccopyr dataclassesrrrtypingrrrrr r r ddtrace.internal.loggerr ddtrace.llmobs._evaluatorsr ddtrace.llmobs._experimentrrrrrrrrrddtrace.llmobsrr`rvr%r)r,r/r?rHrr rXrgrhrrr#r#r#r$sp                         z