蘺h0dZddlmZmZmZmZmZddlmZddl m Z ddl m Z ddl mZmZddlmZddlmZdd lmZdd lmZmZdd lmZGd d eZGddeZGddeZdS)z!Configuration for run evaluators.)AnyDictListOptionalUnion) Embeddings)BaseLanguageModel)BasePromptTemplate) BaseModelField) RunEvaluator) CRITERIA_TYPE)EmbeddingDistance) EvaluatorTypeStringEvaluator)StringDistancec:eZdZUdZeed<deeeffdZ dS) EvalConfigaConfiguration for a given run evaluator. Parameters ---------- evaluator_type : EvaluatorType The type of evaluator to use. Methods ------- get_kwargs() Get the keyword arguments for the evaluator configuration. evaluator_typereturnc8i}|D]\}}|dkr ||||<|S)zGet the keyword arguments for the load_evaluator call. Returns ------- Dict[str, Any] The keyword arguments for the load_evaluator call. r)selfkwargsfieldvals i/var/www/html/services/rag-service/venv/lib/python3.11/site-packages/langchain/smith/evaluation/config.py get_kwargszEvalConfig.get_kwargs&sC  JE3(((F5MM N) __name__ __module__ __qualname____doc__r__annotations__rstrrrrrrrrsQ  "!!!DcNrrceZdZUdZdZeeed< dZeeed< dZ eeed< de ee fffd Z xZ S)SingleKeyEvalConfigzBConfiguration for a run evaluator that only requires a single key.N reference_keyprediction_key input_keyrc~t}dD]}||d|S)N)r(r)r*)superrpop)rrkey __class__s rrzSingleKeyEvalConfig.get_kwargsGsC##%%C " "C JJsD ! ! ! ! r)r r!r"r#r(rr%r$r)r*rrr __classcell__r/s@rr'r'9sLL#'M8C='''@$(NHSM((( $Ix}###BDcNrr'ceZdZUdZeeZeee e e fe d< dZ eeeeefe d< dZee e d< dZee e d< dZee e d< dZeee d < Gd d ZGd d eZGddeZGddeZGddeZGddeZGddeZGddeZGddeZGdde Z GddeZ!Gd d!eZ"Gd"d#eZ#Gd$d%e#Z$dS)& RunEvalConfigaConfiguration for a run evaluation. Parameters ---------- evaluators : List[Union[EvaluatorType, EvalConfig]] Configurations for which evaluators to apply to the dataset run. Each can be the string of an :class:`EvaluatorType `, such as EvaluatorType.QA, the evaluator type string ("qa"), or a configuration for a given evaluator (e.g., :class:`RunEvalConfig.QA `). custom_evaluators : Optional[List[Union[RunEvaluator, StringEvaluator]]] Custom evaluators to apply to the dataset run. reference_key : Optional[str] The key in the dataset run to use as the reference string. If not provided, it will be inferred automatically. prediction_key : Optional[str] The key from the traced run's outputs dictionary to use to represent the prediction. If not provided, it will be inferred automatically. input_key : Optional[str] The key from the traced run's inputs dictionary to use to represent the input. If not provided, it will be inferred automatically. eval_llm : Optional[BaseLanguageModel] The language model to pass to any evaluators that use a language model. )default_factory evaluatorsNcustom_evaluatorsr(r)r*eval_llmceZdZdZdS)RunEvalConfig.ConfigTNr r!r"arbitrary_types_allowedrrrConfigr9s"&rr<ceZdZUdZdZeeed<dZee ed<e j Z e ed< d deede ddffd ZxZS) RunEvalConfig.Criteriaa"Configuration for a reference-free criteria evaluator. Parameters ---------- criteria : Optional[CRITERIA_TYPE] The criteria to evaluate. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. Ncriteriallmrrrc >tjdd|i|dSNr?rr,__init__rr?rr/s rrDzRunEvalConfig.Criteria.__init__- EGG  9 9h 9& 9 9 9 9 9rN)r r!r"r#r?rrr$r@r rCRITERIArrrDr0r1s@rCriteriar>s -1(=)000+/X' (///(5(> >>>7; : :$]3 :FI :  : : : : : : : : : :rrIceZdZUdZdZeeed<dZee ed<e j Z e ed< d deede ddffd ZxZS) RunEvalConfig.LabeledCriteriaa,Configuration for a labeled (with references) criteria evaluator. Parameters ---------- criteria : Optional[CRITERIA_TYPE] The criteria to evaluate. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. Nr?r@rrrc >tjdd|i|dSrBrCrEs rrDz&RunEvalConfig.LabeledCriteria.__init__rFrrG)r r!r"r#r?rrr$r@r rLABELED_CRITERIArrrDr0r1s@rLabeledCriteriarKs  -1(=)000+/X' (///(5(F FFF7; : :$]3 :FI :  : : : : : : : : : :rrNczeZdZUdZejZeed<dZe e ed<dZ e e ed<GddZ dS)RunEvalConfig.EmbeddingDistanceaIConfiguration for an embedding distance evaluator. Parameters ---------- embeddings : Optional[Embeddings] The embeddings to use for computing the distance. distance_metric : Optional[EmbeddingDistanceEnum] The distance metric to use for computing the distance. rN embeddingsdistance_metricceZdZdZdS)&RunEvalConfig.EmbeddingDistance.ConfigTNr:rrrr<rTs&* # # #rr<)r r!r"r#rEMBEDDING_DISTANCErr$rQrrrREmbeddingDistanceEnumr<rrrrrPs )6(H HHH+/ HZ(///;?"78??? + + + + + + + + + +rrcVeZdZUdZejZeed<dZe e ed< dZ e ed<dS)RunEvalConfig.StringDistancezConfiguration for a string distance evaluator. Parameters ---------- distance : Optional[StringDistanceEnum] The string distance metric to use. rNdistanceTnormalize_score) r r!r"r#rSTRING_DISTANCErr$rYrStringDistanceEnumrZboolrrrrrXsd  )6(E EEE15(-.555 !%$$$ N Nrrc`eZdZUdZejZeed<dZe e ed<dZ e e ed<dS)RunEvalConfig.QAa-Configuration for a QA evaluator. Parameters ---------- prompt : Optional[BasePromptTemplate] The prompt template to use for generating the question. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. rNr@prompt) r r!r"r#rQArr$r@rr r`r rrrrar_s]  )6(8 888+/X' (////3+,33333rrac`eZdZUdZejZeed<dZe e ed<dZ e e ed<dS)RunEvalConfig.ContextQA<Configuration for a context-based QA evaluator. Parameters ---------- prompt : Optional[BasePromptTemplate] The prompt template to use for generating the question. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. rNr@r` r r!r"r#r CONTEXT_QArr$r@rr r`r rrr ContextQArc] )6(@ @@@+/X' (////3+,33333rrgc`eZdZUdZejZeed<dZe e ed<dZ e e ed<dS)RunEvalConfig.CoTQArdrNr@r`rerrrCoTQArjrhrrkc,eZdZUdZejZeed<dS)RunEvalConfig.JsonValidityz\Configuration for a json validity evaluator. Parameters ---------- rN)r r!r"r#r JSON_VALIDITYrr$rrr JsonValidityrm 1  )6(C CCCCCrroc,eZdZUdZejZeed<dS)#RunEvalConfig.JsonEqualityEvaluatorz\Configuration for a json equality evaluator. Parameters ---------- rN)r r!r"r#r JSON_EQUALITYrr$rrrJsonEqualityEvaluatorrrrprrtcVeZdZUdZejZeed<dZe ed<dZ e ed<dZ e ed<dS)RunEvalConfig.ExactMatchaxConfiguration for an exact match string evaluator. Parameters ---------- ignore_case : bool Whether to ignore case when comparing strings. ignore_punctuation : bool Whether to ignore punctuation when comparing strings. ignore_numbers : bool Whether to ignore numbers when comparing strings. rF ignore_caseignore_punctuationignore_numbersN) r r!r"r#rr[rr$rwr]rxryrrr ExactMatchrvsb )6(E EEE! T!!!#(D((($$$$$$rrzc:eZdZUdZejZeed<dZe ed<dS)RunEvalConfig.RegexMatchzConfiguration for a regex match string evaluator. Parameters ---------- flags : int The flags to pass to the regex. Example: re.IGNORECASE. rrflagsN) r r!r"r#r REGEX_MATCHrr$r}intrrr RegexMatchr|0s<  )6(A AAAsrrc eZdZUdZejZeed<dZe e ed<dZ e e ed<dZ e eed<dZe eed< d de e de eded dffd ZxZS) RunEvalConfig.ScoreStringaConfiguration for a score string evaluator. This is like the criteria evaluator but it is configured by default to return a score on the scale from 1-10. It is recommended to normalize these scores by setting `normalize_by` to 10. Parameters ---------- criteria : Optional[CRITERIA_TYPE] The criteria to evaluate. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. normalize_by: Optional[int] = None If you want to normalize the score, the denominator to use. If not provided, the score will be between 1 and 10 (by default). prompt : Optional[BasePromptTemplate] rNr?r@ normalize_byr`rrc @tjd||d|dS)N)r?rrrC)rr?rrr/s rrDz"RunEvalConfig.ScoreString.__init__Ws/ EGG  Th\ T TV T T T T Tr)NN)r r!r"r#r SCORE_STRINGrr$r?rrr@r rfloatr`r rrDr0r1s@r ScoreStringr<s  ()6(B BBB,0(=)000+/X' (///(, huo,,,/3+,33315,0 U U}- U#5/ U  U   U U U U U U U U U Urrc(eZdZUejZeed<dS) RunEvalConfig.LabeledScoreStringrN)r r!r"rLABELED_SCORE_STRINGrr$rrrLabeledScoreStringr_s%(5(J JJJJJrr)%r r!r"r#r listr5rrrr%rr$r6rr rr(r)r*r7r r<r'rIrNrrrargrkrortrzrrrrrrr3r3Osx<?De???JU=#z9:;XOSxU<+H%I JKRRR8#'M8C='''@$(NHSM((( $Ix}###B,0Hh()000H'''''''':::::&:::*:::::-:::(+++++/+++(NNNNN,NNN, 4 4 4 4 4 4 4 444444'444 44444#444 DDDDD*DDDDDDDD DDD%%%%%(%%%$     (   !U!U!U!U!U)!U!U!UFKKKKK[KKKKKrr3N) r#typingrrrrrlangchain_core.embeddingsrlangchain_core.language_modelsr langchain_core.promptsr langchain_core.pydantic_v1r r langsmithr (langchain.evaluation.criteria.eval_chainr,langchain.evaluation.embedding_distance.baserrVlangchain.evaluation.schemarr)langchain.evaluation.string_distance.baserr\rr'r3rrrrs''33333333333333000000<<<<<<55555577777777""""""BBBBBBGFFFFFFF !!!!!!!!H*,QKQKQKQKQKIQKQKQKQKQKr