蘺h d dl Z d dlZd dlmZmZ d dlmZ d dlmZ d dl m
Z
d dl m Z d dl
mZ d dlmZ d d lmZmZ d d
lmZmZ d d lmZ d d lmZ d d
lm Z m Z d dl m Z d dl m!Z! d dl"m#Z# d dl$m%Z%m&Z& e j' e( Z) G d de Z* edgd Z+ edgd Z, G d de Z- G d de# Z. G d d e Z/dS ) N)ListOptional)AsyncHtmlLoader)Html2TextTransformer)LlamaCpp)GoogleSearchAPIWrapper)Document)BaseLLM)BasePromptTemplatePromptTemplate) BaseModelField)
BaseRetriever) VectorStore)#AsyncCallbackManagerForRetrieverRun CallbackManagerForRetrieverRun)LLMChain)ConditionalPromptSelector)PydanticOutputParser) RecursiveCharacterTextSplitter TextSplitterc D e Zd ZU dZ edd Zee ed< dS )
SearchQueriesz/Search queries to research for the user's goal..z+List of search queries to look up on Google descriptionqueriesN) __name__
__module__ __qualname____doc__r r r str__annotations__ i/var/www/html/services/rag-service/venv/lib/python3.11/site-packages/langchain/retrievers/web_research.pyr r sM 99 F GT#Y r$ r questiona! <>
You are an assistant tasked with improving Google search results.
<>
[INST] Generate THREE Google search queries that are similar to this question. The output should be a numbered list of questions and each should have a question mark at the end:
{question} [/INST])input_variablestemplatezYou are an assistant tasked with improving Google search results. Generate THREE Google search queries that are similar to this question. The output should be a numbered list of questions and each should have a question mark at the end: {question}c B e Zd ZU dZ ed Zee ed< dS )LineListzList of questions. Questionsr linesN) r r r r r r, r r! r" r# r$ r% r* r* 5 s9 u555E4955555r$ r* c 4 e Zd ZdZd fd ZdedefdZ xZS )QuestionListOutputParserz/Output parser for a list of numbered questions.returnNc V t t d S )N)pydantic_object)super__init__r* )self __class__s r% r3 z!QuestionListOutputParser.__init__> s$
22222r$ textc L t j d| }t | S )Nz\d+\..*?(?:\n|$))r, )refindallr* )r4 r6 r, s r% parsez QuestionListOutputParser.parseA s%
.55e$$$$r$ )r/ N) r r r r r3 r! r* r:
__classcell__)r5 s @r% r. r. ; se 993 3 3 3 3 3%# %( % % % % % % % %r$ r. c e Zd ZU dZ edd Zeed< eed< edd Z e
ed< ed d
Z e ed < e e
d d
d Zeed< eed Zee ed< edd e
d d fdedede
dee d e de
dd fd ZdedefdZd ded e dee fd Zded e dee fd Z ded e dee fd Z dS )!WebResearchRetrieverz `Google Search API` retriever..z"Vector store for storing web pagesr vectorstore llm_chainzGoogle Search API Wrappersearch z!Number of pages per Google searchnum_search_resultsi 2 )
chunk_size
chunk_overlapz1Text splitter for splitting web pages into chunks
text_splitterzList of processed URLs)default_factoryr url_databaseN llmpromptr/ c |s4t t d t fg }| | }t ||t } | ||||| S )a Initialize from llm using default template.
Args:
vectorstore: Vector store for storing web pages
llm: llm for search question generation
search: GoogleSearchAPIWrapper
prompt: prompt to generating search questions
num_search_results: Number of pages per Google search
text_splitter: Text splitter for splitting web pages into chunks
Returns:
WebResearchRetriever
c , t | t S N)
isinstancer )rJ s r% z/WebResearchRetriever.from_llm..v s C!:!: r$ )default_prompt conditionals)rJ rK
output_parser)r> r? r@ rB rF )r DEFAULT_SEARCH_PROMPTDEFAULT_LLAMA_SEARCH_PROMPT
get_promptr r. ) clsr> rJ r@ rK rB rF QUESTION_PROMPT_SELECTORr? s r% from_llmz WebResearchRetriever.from_llmX s 4 >'@ 4::>#&& ' !#2#JE{{}} r$ c f | | }| j || }|S )z3Returns num_search_results pages per Google search.)rc r@ results)r4 rZ rB query_cleanresults r% search_toolz WebResearchRetriever.search_tool s2 --e44 $$[2DEE
r$ run_managerc v t d | d|i }t d| t |d dg }t d| t d g }|D ]}| || j }t d t d| |D ]3}| d d
r| |d 4t | }t |
| j } t d | | rt | d
}
t } t d |
} t | | } | j | } | j | | j | t d g } |D ]/}| | j | 0d | D }
t |
}|S )zSearch Google for documents related to the query input.
Args:
query: user query
Returns:
Relevant documents from all various urls.
z*Generating questions for Google Search ...r& z#Questions for Google Search (raw): r6 r, z Questions for Google Search: z Searching for relevant urls...zSearch results: linkNzNew URLs to load: T)ignore_load_errorszIndexing new urls...z*Grabbing most relevant splits from urls...c i | ]=}|j t t |j f|>S r# ) page_contenttuplesortedmetadataitems).0docs r%
z@WebResearchRetriever._get_relevant_documents.. sM !
!
!
MPS
uVCL,>,>,@,@%A%A B B CS!
!
!
r$ )loggerinfor? getattrrh rB getappendsetlist
differencerH r r loadtransform_documentsrF split_documentsr>
add_documentsextendsimilarity_searchvalues)r4 rZ ri rg questions urls_to_looksearch_resultsresurlsnew_urlsloader html2textdocsunique_documents_dictunique_documentss r% _get_relevant_documentsz,WebResearchRetriever._get_relevant_documents s @AAA U 344 B&BBCCCF6NGR88 ?I??@@@ 4555 5 5E ! - -eT5L M MN KK8 9 9 9 KK;>;; < < <%
5
57764(( 5 ''F 444
5
< (9::;; 333444 /$X$GGGF,..I KK. / / /;;==D 55d;;< >?? r$ c K t rN )NotImplementedError)r4 rZ ri s r% _aget_relevant_documentsz-WebResearchRetriever._aget_relevant_documents s "!r$ )rA )!r r r r r r> r r" r r@ r rB intr rF r r| rH r r! classmethodr
r r rY rc dictrh r r r r r r# r$ r% r= r= F sa (( %u = K %*U3r s} ! ! ! ! ! ! ! ! @ @ @ @ @ @ J J J J J J - - - - - - @ @ @ @ @ @ - - - - - - 2 2 2 2 2 2 E E E E E E E E 7 7 7 7 7 7 7 7 3 3 3 3 3 3 3 3 3 3 3 3 & % % % % % F F F F F F B B B B B B P P P P P P P P 8 $ $ I -n LL ' L6 6 6 6 6 6y 6 6 6 % % % % % 3 % % %b" b" b" b" b"= b" b" b" b" b"r$