o i@sddlZddlZddlZddlZddlZddlmZddlmZddlmZddlm Z ddlm Z ddl Z ddl m Z ddl mZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZdd lmZddlmZddl m!Z!ddl"m#Z#ddl"m$Z$ddl%m&Z'ddl(m)Z)ddl(m*Z*ddl(m+Z+ddl(m,Z,ddl(m-Z-ddl(m.Z.ddl(m/Z/ddl(m0Z0ddl(m1Z1ddl(m2Z2dd l3m4Z4dd!l3m5Z5dd"l3m6Z6dd#l3m7Z7dd$l3m8Z8dd%l3m9Z9dd&l3m:Z:dd'l3m;Z;dd(lm?Z?dd*l@mAZAdd+l@mBZBdd,lCmDZDeeEZFGd-d.d.ed/d0ZGGd1d2d2ed/d0ZHGd3d4d4eHZIGd5d6d6ed/d0ZJGd7d8d8ed/d0ZKdHd9eeLd:eLfd;d<ZMGd=d>d>eZNGd?d@d@eNZOGdAdBdBeNZPGdCdDdDeNZQdEeId:eIfdFdGZRdS)IN)Any)Optional) TypedDict)Union)cast)quote)urlparse)config)agent)EVP_EVENT_SIZE_LIMIT)EVP_PAYLOAD_SIZE_LIMITEVP_PROXY_AGENT_BASE_PATH)EVP_SUBDOMAIN_HEADER_NAME) get_logger)PeriodicService)RLock)Response) RetryError)fibonacci_backoff_with_jitter) _telemetry)AGENTLESS_EVAL_BASE_URL)AGENTLESS_EXP_BASE_URL)AGENTLESS_SPAN_BASE_URL)DROPPED_IO_COLLECTION_ERROR)DROPPED_VALUE_TEXT) EVAL_ENDPOINT)EVAL_SUBDOMAIN_NAME)EXP_SUBDOMAIN_NAME) SPAN_ENDPOINT)SPAN_SUBDOMAIN_NAME) ConfigType)Dataset) DatasetRecord)DatasetRecordRaw) Experiment)JSONType)Project)UpdatableDatasetRecord)get_connection) safe_json)_Meta) _SpanLink) __version__c@s~eZdZUdZeed<eed<eed<eed<eed<eeefed<eeefed<ee ed <e ed <e ed <e ed <d S)LLMObsSpanDataz4Structure of LLMObs span data attached to APM spans.name parent_idtrace_idml_app session_idtagsmetrics span_linksr is_evaluation_spanmetaN) __name__ __module__ __qualname____doc__str__annotations__dictrlistr,r!boolr+rBrBJ/home/ubuntu/.local/lib/python3.10/site-packages/ddtrace/llmobs/_writer.pyr.9s   r.F)totalc@sFeZdZUeed<eed<eed<eeed<eeed<eed<dS)_LLMObsSpanEventOptionalr3servicestatus_messagecollection_errorsr6r N)r9r:r;r=r>r@r,r!rBrBrBrCrEIs    rEc@szeZdZUeed<eed<eed<eeed<eed<eed<eed<eed<eed <eee fed <eeefed <d S) LLMObsSpanEventspan_idr1r0r4r/start_nsdurationstatusr8r5_ddN) r9r:r;r=r>r@intr+r?rrBrBrBrCrIRs  rIc@seZdZUeeeeeffed<eed<eed<eed<eed<eed<eed<eed<eed <e eed <eed <eed <d S)LLMObsEvaluationMetricEventjoin_on metric_typelabelcategorical_valuenumerical_value score_value boolean_valuer2 timestamp_msr4 assessment reasoningN) r9r:r;r?r=r>floatrArOr@rBrBrBrCrP`s   rPc@seZdZUeed<eed<eed<eed<eed<eed<eed<eed<eed <eee fed <e eeefed <e eed <eed <eed<eed<eee fed<dS)LLMObsExperimentEvalMetricEvent metric_sourcerJr1rXrRrSrTrVrW json_valueerrorr4 experiment_idrZrYmetadataN) r9r:r;r=r>rOr[rAr?r&rr@rBrBrBrCr\os"  r\user_defined_agentless_enabledreturncCs\|dur|Szt}Wn tyd}Ynw|durdS|dg}tdd|D S)zUDetermine whether to use agentless mode based on agent availability and capabilities.NT endpointscss|]}t|vVqdSNr ).0endpointrBrBrC sz'should_use_agentless..)r info Exceptiongetany)rb agent_infordrBrBrCshould_use_agentlesss   rncseZdZdZdZdZdZdZdZdZ dddde dddfde de de d e d e d e d e d e ddffdd ZfddZd*fdd ZddZdeeefdeddfddZddZd+ddZd ed!efd"d#Zede fd$d%Zd&d'Zd(d)ZZS),BaseLLMObsWriterzBBase writer class for submitting data to Datadog LLMObs endpoints.ir/_idintervaltimeout is_agentless_site_api_key_app_key _override_url_default_projectrcNc sHtt|j|dt|_g|_d|_||_|ptj |_ |p tj |_ ||_ |p-tjdd|_||_||_|jpE|rC|jd|j ntj|_|rL|jnt|j|_t|j} |jrm| jdkrm| jdvrm|jd|_d d i|_|r|j |jd <|j r|j |jd <n|j |jt!<t"|j#d |j$d|j#dddd|j%|_&dS)N)rtrDD_LLMOBS_OVERRIDE_ORIGINrq.unix)/rqr Content-Typeapplication/json DD-API-KEYDD-APPLICATION-KEY-?S?cSs t|tSre) isinstancerresultrBrBrCs z+BaseLLMObsWriter.__init__..attempts initial_waituntil)'superro__init__r_lock_buffer _buffer_size_timeoutr _dd_api_keyrx_dd_siterwryosenvironrkrzr{ _agentlessAGENTLESS_BASE_URL agent_configtrace_agent_url_intakeENDPOINTr _endpointrschemepathlstrip_headersEVP_SUBDOMAIN_HEADER_VALUErrRETRY_ATTEMPTSrt _send_payload_send_payload_with_retry) selfrtrurvrwrxryrzr{override_url_parsed __class__rBrCrs@         zBaseLLMObsWriter.__init__cs2tt|td|jj|jt |j dS)Nzstarted %r to %r) rrostartloggerdebugrr9_urlatexitregister on_shutdown)rargskwargsrrBrCrszBaseLLMObsWriter.startcs6tt|j|dtd|jj|jt |j dS)Nruzstopped %r to %r) rrostoprrrr9rr unregisterr)rrurrBrCrszBaseLLMObsWriter.stopcCs |dSre)periodicrrBrBrCrs zBaseLLMObsWriter.on_shutdownevent event_sizecCs|jJt|j|jkr(td|jj|jtj d|j dd WddS|j |t kr8t d||j||j |7_ WddS1sPwYdS)zOInternal shared logic of enqueuing events to be submitted to LLM Observability.z2%r event buffer full (limit is %d), dropping event buffer_full event_typer_NzRmanually flushing buffer because queueing next event will exceed EVP payload limit)rlenr BUFFER_LIMITrwarningrr9 telemetryrecord_dropped_payload EVENT_TYPErr rrappendrrrrBrBrC_enqueues   "zBaseLLMObsWriter._enqueuec CsZzt|}td||j|WSty,tjd||jddtj||jddYdSw)Nz&encoded %d LLMObs %s events to be sentz$failed to encode %d LLMObs %s eventsT)exc_infoencoding_errorr)r*rrr TypeErrorr_rr)rpayload num_eventsenc_llm_eventsrBrBrC_encodes zBaseLLMObsWriter._encodec Cs|j|js WddS|j}g|_d|_Wdn1s#wY|jr8|jds8tddS||}| |t |}|sIdSz | |t |WdSt yzt jt ||jddtjdt ||j|jddd id YdSw) NrrzA Datadog API key is required for sending data to LLM Observability in agentless mode. LLM Observability data will not be sent. Ensure an API key is set either via DD_API_KEY or via `LLMObs.enable(api_key=...)` before running your application.connection_errorr(failed to send %d LLMObs %s events to %sTsend_to_telemetryFrextra)rrrrrrkrr_datarrrrjrrrr_r)reventsdatarrBrBrCrs<   zBaseLLMObsWriter.periodicrrc Cst|j}z]zE|d|j||j|}|jdkr7tjd||j |j |j| ddidt j ||j ddn td ||j |j t|WW|Stybtjd ||j |jd ddid w|w) NPOSTi,zJfailed to send %d LLMObs %s events to %s, got response code %d, status: %srF)r http_errorrzsent %d LLMObs %s events to %srTr)r)rrequestrr getresponserMrr_rrreadrrrrfrom_http_responsecloserj)rrrconnresprBrBrCrs<      zBaseLLMObsWriter._send_payloadcCs|j|jSre)rrrrBrBrCr6szBaseLLMObsWriter._urlcCst)zRReturn payload containing events to be encoded and submitted to LLM Observability.)NotImplementedErrorrrrBrBrCr:szBaseLLMObsWriter._datacCs"|j|j|j|j|j|j|jdS)N)rtrurvrwrxrz)r _intervalrrrwrxrzrrBrBrCrecreate>szBaseLLMObsWriter.recreatere)rcN) r9r:r;r<rrrrrrr'r[rAr=rrrrrrIrPrOrrrbytesrpropertyrrr __classcell__rBrBrrCrosV    1   roc@sPeZdZdZdZeZeZe Z de ddfddZ de e deeeffd d ZdS) LLMObsEvalMetricWriterzDWrites custom evaluation metric events to the LLMObs Evals Endpoint.evaluation_metricrrcNcCstt|}|||dSre)rr*rrrBrBrCenqueueQs zLLMObsEvalMetricWriter.enqueuercCsddd|idiS)Nrrr5type attributesrBrrBrBrCrUszLLMObsEvalMetricWriter._data)r9r:r;r<rrrrrrrrPrr@r?r=rrrBrBrBrCrIs"rc@seZdZdZeZeZdZdZ dZ dZ dhZ de fde d e d ed efd d Zeejde dejddddde fde d e d ed efddZdee effddZdQde d e de d ed ef ddZde d dfddZd e d!ee d"e d efd#d$Zed%ee e!fd&e"d efd'd(Z# ) )dRde d*e d+e$e!d,e$e d-e$e d.e"d/e"d e%e&e$e e$ee ffd0d1Z'   dSd e d!ee d2ee&d3ee$e d ef d4d5Z(dTde d6e$e)d.e"fd7d8Z*dUd9ee d e+fd:d;Z,dUdfd?d@Z- A    B )dVd9e de d*e dCe&dDeee efd3ee$e d"ee dEee&dFe"d e%e e ffdGdHZ.  dWdIe dJee dKee d dfdLdMZ/dIe dNe$e0d3e$e d dfdOdPZ1dS)XLLMObsExperimentsClient experimentrqg$@gN@csvNmethodrbodyrcc CsTz |||||WSty)}zt|jdtr$|jdWYd}~Sd}~ww)Nr)_request_with_retryrrrr)rrrrruerBrBrCrcszLLMObsExperimentsClient.requestrrrcCst|to |jdkS)Ni)rrrMrrBrBrCrpz LLMObsExperimentsClient.rc Csd|j|jd}|js|j|t<|rt|dnd}t|j |d}z$|j |j |}t d|| |||||} t| W|S|w)Nrrrrutf-8urlruz requesting %s)rxryrrrjsondumpsencoder)rrrrrrrrr) rrrrruheaders encoded_bodyrrrrBrBrCrls   z+LLMObsExperimentsClient._request_with_retry evaluationcCsJd}|jd|ddd|idid}|jdkr#td |d d |jdS) Nz1/api/unstable/llm-obs/v1/config/evaluators/customPUTrevaluator_configrrrzFailed to publish evaluator eval_name: )rrM ValueError)rrrrrBrBrCpublish_custom_evaluators z0LLMObsExperimentsClient.publish_custom_evaluatorr content_typec Csv||j|jd}t|j|jd}z%|j|j|}td|||||||| }t |W| S| w)Nrrzrequesting %s, %s) rxryr)rBULK_UPLOAD_TIMEOUTrrrrrrrr) rrrr rrrrrrBrBrCmultipart_requests z)LLMObsExperimentsClient.multipart_request dataset_idcCsLd}|jd|ddd|gddid}|jd kr$td td |dS) Nz(/api/unstable/llm-obs/v1/datasets/deleterrdatasetssoft)r dataset_idsrrrzFailed to delete dataset r)rrMridget_json)rr rrrBrBrCdataset_deletes z&LLMObsExperimentsClient.dataset_delete dataset_name project_name descriptionc Cs||}|d}td||d|d}dd||ddi}|d ||}|jd kr=td |d |jd ||} | dd} | dusO| dkrYtd| dud| ddd} t||| g|| | |dd S)Nrsz)getting records with project ID %s for %sz/api/unstable/llm-obs/v1/z /datasetsrrr/rrrrzFailed to create dataset r rrqz/unexpected dataset state, invalid ID (is None: )rcurrent_version r/projectr recordsrlatest_versionversion _dne_client filter_tags) project_create_or_getrkrrrrMrrr") rrrrr project_idrrr response_datar  curr_versionrBrBrCdataset_creates8      z&LLMObsExperimentsClient.dataset_createrecord is_updatecCsd}d|vr|ddurdn|d}d}d|vr$|ddur in|d}ttttf|d||d}|rAtt|d|d<|S|d}|rOtt||d<|S) Nexpected_outputrqra input_datainputr)ra record_idrr4)rr?r=r&rkr()r'r(r)rarjr4rBrBrC_get_record_jsons" z(LLMObsExperimentsClient._get_record_jsonTr#insert_recordsupdate_recordsdelete_record_ids deduplicatecreate_new_versionc sfdd|D}fdd|D} d|d|d} dd||| tt|||d d i} d | | } | jd krKtd |d| jd| jd| j| } | d}|r]|dddnd}|rhdd|Dng}|rsdd|Dng}|||fS)Ncg|]}|dqS)Fr/rfrrrBrC z@LLMObsExperimentsClient.dataset_batch_update..cr5Tr6r7rrBrCr9r:/api/v2/llm-obs/v1/ /datasets/z /batch_updaterr)r0r1delete_recordsr3r4)rrrrrzFailed to update dataset rz, rrrcSsg|]}|dqS)rrBr7rBrBrCr9rcSsg|] }|ddqS)r canonical_id)rkr7rBrBrCr9s)rr&rrMrreasonrr)rr r#r0r1r2r3r4irsursrrrr$r new_versionnew_record_idsnew_canonical_idsrBrrCdataset_batch_updates2    z,LLMObsExperimentsClient.dataset_batch_updaterr4c CsH||}|d}td||t|pdd|dt|}|d|}|jdkrs                                             5 &