ܘh\ddlZddlZddlmZddlmZmZmZmZm Z m Z m Z  ddl m Z mZmZn#e$r ddlm Z mZmZYnwxYwddlmZddlmZmZmZmZGdde ZGd d e d ZGd dZGddeZdeeeege eeefffdZdS)N)abstractmethod)CallableDictListOptional TypedDictUnioncast) BaseModelFieldValidationError)wraps) SCORE_TYPE VALUE_TYPEExampleRunc"eZdZUdZeed< dZeed< dZe ed< dZ e eed< dZ e e ed< eeZe ed < dZe eejefed < dZe eejefed < Gd d ZdS)EvaluationResultzEvaluation result.keyNscorevaluecomment correction)default_factoryevaluator_info source_run_id target_run_idceZdZdZdZdS)EvaluationResult.ConfigzPydantic model configuration.FN)__name__ __module__ __qualname____doc__ allow_extraf/var/www/html/services/rag-service/venv/lib/python3.11/site-packages/langsmith/evaluation/evaluator.pyConfigr's++ r&r()r r!r"r#str__annotations__rrrrrrrrr dictrrr uuidUUIDrr(r%r&r'rrs  HHH@E:0E:8!GXc]!!!2!%J%%%: 5666ND666559M8E$)S.12999659M8E$)S.12999 r&rc*eZdZUdZeeed<dS)EvaluationResultszQBatch evaluation results, if your evaluator wishes to return multiple scores.resultsN)r r!r"r#rrr*r%r&r'r/r/-s0""" ####!!r&r/F)totalc eZdZdZe ddedeedee e ffdZ ddedeedee e ffdZ dS) RunEvaluatorzEvaluator interface class.NrunexamplereturncdS)zEvaluate an example.Nr%selfr4r5s r' evaluate_runzRunEvaluator.evaluate_run8sr&cnKtjd|j||d{VS)z#Evaluate an example asynchronously.N)asyncioget_running_looprun_in_executorr:r8s r' aevaluate_runzRunEvaluator.aevaluate_run>sS-//?? $#S'         r&N) r r!r"r#rrrrr rr/r:r?r%r&r'r3r35s$$59###!)'!2# !22 3###^# 6:   !)'!2 !22 3      r&r3c eZdZdZdeeeegee e e fffdZ ddee e fde de fdZd ee e fdee e ffd Z dd ed eedee e ffdZ dd ed eedee e ffdZd S)DynamicRunEvaluatora A dynamic evaluator that wraps a function and transforms it into a `RunEvaluator`. This class is designed to be used with the `@run_evaluator` decorator, allowing functions that take a `Run` and an optional `Example` as arguments, and return an `EvaluationResult` or `EvaluationResults`, to be used as instances of `RunEvaluator`. Attributes: func (Callable): The function that is wrapped by this evaluator. funccDt||||_dS)a Initialize the DynamicRunEvaluator with a given function. Args: func (Callable): A function that takes a `Run` and an optional `Example` as arguments, and returns an `EvaluationResult` or `EvaluationResults`. N)rrC)r9rCs r'__init__zDynamicRunEvaluator.__init__Ss% d D r&Fresult allow_no_keyr6ct|tr|S d|vr|r|jj|d<tdi|S#t$r}t d||d}~wwxYw)Nrz[Expected an EvaluationResult object, or dict with a metric 'key' and optional 'score'; got r%) isinstancerrCr r ValueError)r9rFrGes r'_coerce_evaluation_resultz-DynamicRunEvaluator._coerce_evaluation_resultcs f. / / M F""7$(I$6F5M#--f-- -   =4:==  s : AAAr0cd|vr7|}fd|dD|d<tdi|Stt|dS)Nr0c:g|]}|Sr%)rL).0rr9s r' zBDynamicRunEvaluator._coerce_evaluation_results..{s467..q11r&T)rGr%)copyr/rLr r+)r9r0cps` r'_coerce_evaluation_resultsz.DynamicRunEvaluator._coerce_evaluation_resultsus   B;B9;MByM%**r** *--d4.A.APT-UUUr&Nr4r5c|||}t|tr|St|tst d|||S)a Evaluate a run using the wrapped function. This method directly invokes the wrapped function with the provided arguments. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: Union[EvaluationResult, EvaluationResults]: The result of the evaluation. z=Expected a dict, EvaluationResult, or EvaluationResults, got )rCrIrr+rJrT)r9r4r5rFs r'r:z DynamicRunEvaluator.evaluate_runsu3(( f. / / M&$'' XPVXX ..v666r&c.|||S)a Make the evaluator callable, allowing it to be used like a function. This method enables the evaluator instance to be called directly, forwarding the call to `evaluate_run`. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: Union[EvaluationResult, EvaluationResults]: The result of the evaluation. )r:r8s r'__call__zDynamicRunEvaluator.__call__s   g...r&)Fr@)r r!r"r#rrrrr rr/r+rEboolrLrTr:rWr%r&r'rBrBGsv   (7# $e,<>OQU,U&V V &#&,-  $ Vt../ V !22 3 V V V V6:777!)'!27 !22 3777726:///!)'!2/ !22 3//////r&rBrCc t|S)z4Decorator to create a run evaluator from a function.)rB)rCs r' run_evaluatorrZs t $ $$r&)r<r,abcrtypingrrrrrr r pydantic.v1r r r ImportErrorpydantic functoolsrlangsmith.schemasrrrrrr/r3rBr+rZr%r&r'rbs IIIIIIIIIIIIIIIIII;===========;;;::::::::::::;BBBBBBBBBBBBy:""""" """"        $c/c/c/c/c/,c/c/c/L%  hw %(8:KT(Q"RR %%%%%%s -??