o i&@sdZddlZddlmZddlmZddlmZddlmZddlmZddlm Z dd lm Z Gd d d eZ Gd d d eZ dS)z&String matching evaluators for LLMObs.N)Any)Callable)Optional)Pattern) BaseEvaluator)EvaluatorContext)EvaluatorResultcseZdZdZdZ      ddeded ed eee geefd eee geefd eef fd d Z de de fddZ ZS)StringCheckEvaluatoraPEvaluator that performs string comparison operations. Compares the output_data with expected_output using various string operations. Supports operations similar to OpenAI's String Check Grader. Operations: - 'eq': Equals (exact match) - default - 'ne': Not equals (inequality) - 'contains': Contains substring (case-sensitive) - 'icontains': Contains substring (case-insensitive) Example:: # Exact match (default) evaluator = StringCheckEvaluator(operation="eq", case_sensitive=True) result = evaluator.evaluate(context) # Returns: True if output == expected_output # Not equals evaluator = StringCheckEvaluator(operation="ne") # Returns: True if output != expected_output # Contains (case-sensitive) evaluator = StringCheckEvaluator(operation="contains") # Returns: True if expected_output is in output # Contains (case-insensitive) evaluator = StringCheckEvaluator(operation="icontains") # Returns: True if expected_output is in output (ignoring case) # Extract field from dict output evaluator = StringCheckEvaluator( operation="eq", output_extractor=lambda x: x.get("message", "") if isinstance(x, dict) else str(x), ) # Extracts "message" field from dict before comparison :param operation: String comparison operation: 'eq', 'ne', 'contains', 'icontains' (default: 'eq') :param case_sensitive: Whether to perform case-sensitive comparison (default: True, ignored for 'icontains') :param strip_whitespace: Whether to strip leading/trailing whitespace before comparison (default: False) :param output_extractor: Optional function to extract/transform output_data before comparison. Must return a str or None. (default: None) :param expected_output_extractor: Optional function to extract/transform expected_output before comparison. Must return a str or None. (default: None) :param name: Optional custom name for the evaluator )eqnecontains icontainsr TFN operationcase_sensitivestrip_whitespaceoutput_extractorexpected_output_extractornamecstj|d||jvrtd|jd||dur#t|s#td|dur/t|s/td||_||_||_||_ ||_ dS)aInitialize the StringCheckEvaluator evaluator. :param operation: String comparison operation: 'eq', 'ne', 'contains', 'icontains' :param case_sensitive: Whether to perform case-sensitive comparison :param strip_whitespace: Whether to strip whitespace before comparison :param output_extractor: Optional function to extract/transform output_data before comparison. Must return a str or None. :param expected_output_extractor: Optional function to extract/transform expected_output before comparison. Must return a str or None. :param name: Optional custom name for the evaluator :raises ValueError: If operation is invalid rzoperation must be one of z, got: N,output_extractor must be a callable functionz5expected_output_extractor must be a callable function) super__init__VALID_OPERATIONS ValueErrorcallable TypeErrorrrrrr)selfrrrrrr __class__^/home/ubuntu/.local/lib/python3.10/site-packages/ddtrace/llmobs/_evaluators/string_matching.pyr@s  zStringCheckEvaluator.__init__contextreturncCs<|j}|j}|jdur||}|jdur||}|dur3|dur3|jdk}t||r/ddSddS|dus;|durW|jdkrFtdddS|jdkrQtdddStdddSt|}t|}|jrj|}|}|jd ksr|j sz| }| }|jdkr||k}n|jdkr||k}n||v}t||rddSddS) zPerform string comparison evaluation. :param context: The evaluation context :return: EvaluatorResult with boolean value and pass/fail assessment Nr passfailvalue assessmentFr Tr ) output_dataexpected_outputrrrrstrrstriprlower)rr!outputexpectedresult output_str expected_strrrr evaluatefs:              zStringCheckEvaluator.evaluate)r TFNNN)__name__ __module__ __qualname____doc__rr*boolrrrrrrr2 __classcell__rrrr r s./&r c sheZdZdZ    ddedededeeegeefd eef fd d Z d e d e fddZ Z S)RegexMatchEvaluatoraEvaluator that performs regex pattern matching. Checks if the output_data matches a given regex pattern. Useful for validating structured outputs like emails, phone numbers, URLs, or any custom format requirements. Example:: # Validate email format evaluator = RegexMatchEvaluator(pattern=r"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$") result = evaluator.evaluate(context) # Returns: True if pattern matches, False otherwise # Extract field from dict output before regex matching evaluator = RegexMatchEvaluator( pattern=r"\d{3}-\d{4}", output_extractor=lambda x: x.get("phone", "") if isinstance(x, dict) else str(x) ) # Extracts "phone" field from dict before pattern matching :param pattern: The regex pattern to match against :param match_mode: How to match - 'search' (partial), 'match' (from start), or 'fullmatch' (entire string) :param flags: Optional regex flags (e.g., re.IGNORECASE) :param output_extractor: Optional function to extract/transform output_data before matching. Must return a str or None. (default: None) :param name: Optional custom name for the evaluator searchrNpattern match_modeflagsrrc stj|d|dvrtd|z t|||_Wntjy/}ztd|d}~ww|durr#) r(rrr*r<r;r:r>r?)rr!r-r0r>r/rrr r2s      zRegexMatchEvaluator.evaluate)r:rNN)r3r4r5r6r*intrrrrrrr2r8rrrr r9s$$r9) r6r@typingrrrrddtrace.llmobs._experimentrrrr r9rrrr s