o i@srdZddlmZddlmZddlmZddlmZddlmZdee dee d e fd d Z Gd d d eZ dS)z*Semantic similarity evaluators for LLMObs.)Callable)Optional) BaseEvaluator)EvaluatorContext)EvaluatorResultvec1vec2returnc Cst|t|krtdt|dt|z(ddl}|j|}|j|}|dks/|dkr2WdSt|||||WStyGYnwd\}}}t||D]\}} ||| 7}|||7}|| | 7}qR|d}|d}|dksy|dkr{dS|||S)a'Calculate cosine similarity between two vectors. Uses numpy if available for optimized performance, otherwise falls back to a single-pass pure Python implementation. :param vec1: First vector :param vec2: Second vector :return: Cosine similarity score between -1 and 1 zVectors must have same length: z != rN)r r r g?) len ValueErrornumpylinalgnormfloatdot ImportErrorzip) rrnp magnitude1 magnitude2rmag1mag2abrW/home/ubuntu/.local/lib/python3.10/site-packages/ddtrace/llmobs/_evaluators/semantic.py_cosine_similarity s,        rcsXeZdZdZ  d deegeefdedeeffdd Z d e d e fd d Z Z S)SemanticSimilarityEvaluatora Evaluator that measures semantic similarity using embeddings. Compares the semantic meaning of output_data and expected_output using embedding vectors. Useful for evaluating open-ended responses where exact matches are too strict but semantic equivalence matters. The evaluator requires an embedding function that converts text to vectors. This can be OpenAI embeddings, sentence transformers, or any custom model. Example with OpenAI:: from openai import OpenAI client = OpenAI() def get_embedding(text): response = client.embeddings.create( input=text, model="text-embedding-3-small" ) return response.data[0].embedding evaluator = SemanticSimilarityEvaluator( embedding_fn=get_embedding, threshold=0.8 ) result = evaluator.evaluate(context) # Returns: 0.92 (similarity score between 0.0 and 1.0) Example with sentence-transformers:: from sentence_transformers import SentenceTransformer model = SentenceTransformer('all-MiniLM-L6-v2') def get_embedding(text): return model.encode(text).tolist() evaluator = SemanticSimilarityEvaluator(embedding_fn=get_embedding) :param embedding_fn: Function that takes text string and returns embedding vector (list of floats) :param threshold: Minimum similarity score (0-1) required to pass (default: 0.7) :param name: Optional custom name for the evaluator ffffff?N embedding_fn thresholdnamecsRtj|dt|stdd|krdks!ntd|||_||_dS)aIInitialize the SemanticSimilarityEvaluator evaluator. :param embedding_fn: Function that converts text to embedding vector :param threshold: Minimum similarity score (0-1) to pass :param name: Optional custom name for the evaluator :raises ValueError: If threshold is not between 0 and 1 )r"z(embedding_fn must be a callable functionrz(threshold must be between 0 and 1, got: N)super__init__callable TypeErrorr r r!)selfr r!r" __class__rrr%^s  z$SemanticSimilarityEvaluator.__init__contextr c Cs|j}|j}|dur|durtdddS|dus|dur"tdddS|t|}|t|}t||}|dd}||jkrBdnd}t||dS) zPerform semantic similarity evaluation. :param context: The evaluation context :return: EvaluatorResult with similarity score and pass/fail assessment based on threshold Ng?pass)value assessmentr failr#) output_dataexpected_outputrr strrr!) r(r+outputexpectedoutput_embeddingexpected_embedding similaritynormalized_similarityr.rrrevaluatevs     z$SemanticSimilarityEvaluator.evaluate)rN)__name__ __module__ __qualname____doc__rr3listrrr%rrr: __classcell__rrr)rr2s.rN) r>typingrrddtrace.llmobs._experimentrrrr?rrrrrrrs     '