蘺h*dZddlmZddlZddlZddlmZmZmZm Z m Z ddl m Z ddl mZddlmZddlmZdd lmZdd lmZmZmZdd lmZmZdd lmZddZddZGddeeeZ GddeeeZ!Gdde!Z"dS)z-LLM Chains for evaluating question answering.) annotationsN)AnyListOptionalSequenceTuple)BaseLanguageModel)PromptTemplate)Extra) Callbacks)LLMChain)CONTEXT_PROMPT COT_PROMPTPROMPT) LLMEvalChainStringEvaluator)RUN_KEYtextstrreturnOptional[Tuple[str, int]]ctjd|tj}|rZ|ddkrdS|ddkrdS |dt ddtj }|dkrdS|dkrdS|d t ddtj }|dkrdS|dkrdSn#t$rYnwxYwdS) Nzgrade:\s*(correct|incorrect)CORRECT)rr INCORRECT)rrr) researchstrip IGNORECASEgroupuppersplit translater maketransstring punctuation IndexError)rmatch first_word last_words j/var/www/html/services/rag-service/venv/lib/python3.11/site-packages/langchain/evaluation/qa/eval_chain.py _get_scorer.s I5tzz||R] S SE " ;;q>>   ! !Y . .< [[^^ ! ! # #{ 2 2!> JJLL    # - -cmmBFDV.W.W X X       * *<     ; . .!> JJLL UWWR  Ys}}RV-?@@ A A  ??   ) )< __  + - -!>.      4s&A;F? F?'A;F?$F?? G  G dictcn|}t|}|d\}}n|\}}|||dS)zParse the output text. Args: text (str): The output text to parse. Returns: Any: The parsed output. N)NN) reasoningvaluescore)r r.)rr1 parsed_scoresr2r3s r-_parse_string_eval_outputr50sP Iy))M! uu$ u  ceZdZUdZdZded<GddZed.d Ze d/d Z e d.d Z e d.d Z e d0d1dZ d2d dd3d"Zd4d$Zd d d d%d&d5d,Zd d d d%d&d5d-Zd S)6 QAEvalChainz,LLM Chain for evaluating question answering.resultsr output_keyc eZdZdZejZdS)QAEvalChain.Config"Configuration for the QAEvalChain.N__name__ __module__ __qualname____doc__r ignoreextrar6r-Configr<K00 r6rFrboolcdSNFrEclss r-is_lc_serializablezQAEvalChain.is_lc_serializablePur6cdS)N correctnessrEselfs r-evaluation_namezQAEvalChain.evaluation_nameTs}r6cdSNTrErQs r-requires_referencezQAEvalChain.requires_referenceXtr6cdSrUrErQs r-requires_inputzQAEvalChain.requires_input\rWr6Nllmr promptOptional[PromptTemplate]kwargsrc |pt}hd}|t|jkrtd|d|j|d||d|S)aLoad QA Eval Chain from LLM. Args: llm (BaseLanguageModel): the base language model to use. prompt (PromptTemplate): A prompt template containing the input_variables: 'input', 'answer' and 'result' that will be used as the prompt for evaluation. Defaults to PROMPT. **kwargs: additional keyword arguments. Returns: QAEvalChain: the loaded QA eval chain. >queryanswerresultInput variables should be , but got rZr[rE)rsetinput_variables ValueError)rLrZr[r]expected_input_varss r-from_llmzQAEvalChain.from_llm`s,!6;;; #f&<"="= = =4-@44!144 s4s644V444r6r_r`ra callbacksexamplesSequence[dict] predictions question_key answer_keyprediction_keyrkr List[dict]ctfdt|D}|||S)5Evaluate question answering examples and predictions.cRg|]#\}}|||d$S)r_r`rarE).0iexamplerprqrnros r- z(QAEvalChain.evaluate..sR   7 !.!*-%a.8     r6rj enumerateapply)rRrlrnrorprqrkinputss ```` r-evaluatezQAEvalChain.evaluate_       (11    zz&Iz666r6r/cvt||j}t|vr|t|t<|SNr5r:rrRra parsed_results r-_prepare_outputzQAEvalChain._prepare_output51&2IJJ f  %+G_M' "r6F referenceinputrkinclude_run_info predictionr Optional[str]rrc P||||d||}||S)aEvaluate Chain or LLM output, based on optional input and label. Args: prediction (str): the LLM or chain prediction to evaluate. reference (Optional[str], optional): the reference label to evaluate against. input (Optional[str], optional): the input to consider during evaluation callbacks (Callbacks, optional): the callbacks to use for tracing. include_run_info (bool, optional): whether to include run info in the returned results. **kwargs: additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. rvrkrrrRrrrrkrr]ras r-_evaluate_stringszQAEvalChain._evaluate_stringssK0#$    -   ##F+++r6ctK||||d||d{V}||S)Nrvr~rkracallrrs r-_aevaluate_stringszQAEvalChain._aevaluate_stringssgzz"i:NN-"         ##F+++r6rrHrrr)rZr r[r\r]rrr8rv)rlrmrnrmrorrprrqrrkr rrrrar/rr/rrrrrrrkr rrHr]rrr/)r?r@rArBr:__annotations__rF classmethodrMpropertyrSrVrYrirrrrrEr6r-r8r8Fs66J [XXX,05555[5D$"& 7 $777777,$(##!&!,!,!,!,!,!,N$(##!&,,,,,,,,r6r8ceZdZdZed.dZed.dZed.dZGddZ ed/d Z ed0dZ e d1d2dZ d3ddd4d"Z d5d$Zdddd%d&d6d,Zdddd%d&d6d-ZdS)7ContextQAEvalChainz3LLM Chain for evaluating QA w/o GT based on contextrrHcdSrJrErKs r-rMz%ContextQAEvalChain.is_lc_serializablerNr6cdS)z.Whether the chain requires a reference string.TrErQs r-rVz%ContextQAEvalChain.requires_reference tr6cdS)z+Whether the chain requires an input string.TrErQs r-rYz!ContextQAEvalChain.requires_inputrr6c eZdZdZejZdS)ContextQAEvalChain.Configr=Nr>rEr6r-rFrrGr6rFr[r Nonecrhd}|t|jkrtd|d|jdS)N>r_racontextrbrc)rerfrg)rLr[rhs r-_validate_input_varsz'ContextQAEvalChain._validate_input_varss`<<< #f&<"="= = =4-@44!144  > =r6rcdS)NzContextual AccuracyrErQs r-rSz"ContextQAEvalChain.evaluation_names$$r6NrZr r\r]rc T|pt}|||d||d|S)aLoad QA Eval Chain from LLM. Args: llm (BaseLanguageModel): the base language model to use. prompt (PromptTemplate): A prompt template containing the input_variables: 'query', 'context' and 'result' that will be used as the prompt for evaluation. Defaults to PROMPT. **kwargs: additional keyword arguments. Returns: ContextQAEvalChain: the loaded QA eval chain. rdrE)rrrLrZr[r]s r-rizContextQAEvalChain.from_llms?,)>   (((s4s644V444r6r_rrarjrlrrrnro context_keyrqrkr ctfdt|D}|||S)rtcRg|]#\}}|||d$S)r_rrarE)rwrxryrrqrnros r-rzz/ContextQAEvalChain.evaluate..sR   7 !.";/%a.8     r6rjr{)rRrlrnrorrqrkr~s ```` r-rzContextQAEvalChain.evaluate rr6r/cvt||j}t|vr|t|t<|Srrrs r-rz"ContextQAEvalChain._prepare_output#rr6Frrrrrrc P||||d||}||S)Nrrrrs r-rz$ContextQAEvalChain._evaluate_strings)sK$$    -   ##F+++r6ctK||||d||d{V}||S)Nrrrrs r-rz%ContextQAEvalChain._aevaluate_strings>sgzz"yJOO-"         ##F+++r6r)r[r rrrr)rZr r[r\r]rrrr)rlrrrnrrrorrrrqrrkr rrrrr)r?r@rArBrrMrrVrYrFrrSrirrrrrEr6r-rrs==[XX [%%%X%,05555[5:$$& 7 $777777,$(##!&,,,,,,2$(##!&,,,,,,,,r6rc^eZdZdZeddZeddZe dddZdS)CotQAEvalChainz=LLM Chain for evaluating QA using chain of thought reasoning.rrHcdSrJrErKs r-rMz!CotQAEvalChain.is_lc_serializableSrNr6rcdS)NzCOT Contextual AccuracyrErQs r-rSzCotQAEvalChain.evaluation_nameWs((r6NrZr r[r\r]rc T|pt}|||d||d|S)zLoad QA Eval Chain from LLM.rdrE)rrrs r-rizCotQAEvalChain.from_llm[s?%:   (((s4s644V444r6rrr)rZr r[r\r]rrr) r?r@rArBrrMrrSrirEr6r-rrPsGG[)))X),0 5 5 5 5[ 5 5 5r6r)rrrr)rrrr/)#rB __future__rrr'typingrrrrrlangchain_core.language_modelsr langchain_core.promptsr langchain_core.pydantic_v1r langchain.callbacks.managerr langchain.chains.llmr #langchain.evaluation.qa.eval_promptrrrlangchain.evaluation.schemarrlangchain.schemarr.r5r8rrrEr6r-rs33"""""" 77777777777777<<<<<<111111,,,,,,111111))))))RRRRRRRRRREEEEEEEE$$$$$$:,G,G,G,G,G,(O\G,G,G,T},},},},},?L},},},@55555'55555r6