蘺hVtdZddlmZddlZddlZddlZddlmZddlm Z m Z m Z m Z m Z mZmZmZmZmZmZmZmZmZmZmZddlmZddlmZddlmZdd lm Z dd l!m"Z"m#Z#ed Z$d?dZ%d@dZ&GddeZ'dAdZ(dBdZ)dCd"Z*dDd&Z+Gd'd(eZ,d)ddd*d+d,dEd9Z-dFd<Z.d)ddd*d+d,dGd>Z/dS)Hz@Module contains logic for indexing documents into vector stores.) annotationsN)islice)Any AsyncIterable AsyncIteratorCallableDictIterableIteratorListLiteralOptionalSequenceSet TypedDictTypeVarUnioncast) BaseLoader)Document)root_validator) VectorStore)NAMESPACE_UUID RecordManagerT input_stringstrreturn uuid.UUIDctj|d}t jt |S)z3Hashes a string and returns the corresponding UUID.utf-8)hashlibsha1encode hexdigestuuiduuid5r)r hash_values ^/var/www/html/services/rag-service/venv/lib/python3.11/site-packages/langchain/indexes/_api.py_hash_string_to_uuidr*%s=l11'::;;EEGGJ :nj 1 11datadict[Any, Any]ctj|d}tj|d}t jt|S)z>Hashes a nested dictionary and returns the corresponding UUID.T) sort_keysr!) jsondumpsr"r#r$r%r&r'r)r,serialized_datar(s r)_hash_nested_dict_to_uuidr3+sQj666Oo44W==>>HHJJJ :nj 1 11r+ceZdZUdZded<ded< ded< ded< edd Zed ddZddZ eddddZ dS)_HashedDocumentz#A hashed document with a unique ID.ruidhash_ content_hash metadata_hashrboolcdS)NF)clss r)is_lc_serializablez"_HashedDocument.is_lc_serializable=sur+T)prevaluesDict[str, Any]c|dd}|di}d}|D]}||vrtd|dtt|} tt |}n%#t $r}td|dd }~wwxYw||d <||d <tt||z|d <|d d } | |d |d <|S)z6Root validator to calculate content and metadata hash. page_contentmetadata)r7r8r9zMetadata cannot contain key z$ as it is reserved for internal use.zFailed to hash metadata: z6. Please use a dict that can be serialized using json.Nr8r9r7r6)get ValueErrorrr*r3 Exception) r=r@contentrEforbidden_keyskeyr8r9e_uids r)calculate_hashesz _HashedDocument.calculate_hashesAsS**^R00::j"--C!  Ch 53555 /8899   9( C CDDMM   HAHHH   ".~"/2<-3OPPQQwzz%&& <"7OF5M s(B B'B""B'rc8t|j|jS)zReturn a Document object.rCrE)rrCrE)selfs r) to_documentz_HashedDocument.to_documentds$*]    r+Nr6document Optional[str]c2|||j|jS)z(Create a HashedDocument from a Document.)r6rCrErP)r=rTr6s r) from_documentz_HashedDocument.from_documentks, s!.&    r+)rr:)r@rArrA)rr)rTrr6rUrr5) __name__ __module__ __qualname____doc____annotations__ classmethodr>rrNrRrWr<r+r)r5r52s-- HHHJJJB+,[^    D    9=     [   r+r5sizeintiterable Iterable[T]Iterator[List[T]]c#rKt|} tt||}|sdS|V&)Utility batching function.TN)iterlistr)r^r`itchunks r)_batchriwsH hBVB%%&&  F r+AsyncIterable[T]AsyncIterator[List[T]]cKg}|23d{V}t||kr||t||kr|WVg}J6|r|WVdSdS)rdN)lenappend)r^r`batchelements r)_abatchrqsE!g u::   LL ! ! ! u::  KKKKE "  sA source_id_key+Union[str, Callable[[Document], str], None]&Callable[[Document], Union[str, None]]cdSttrfdStrStddt d)z$Get the source id from the document.NcdSNr<)docs r)z)_get_source_id_assigner..s4r+c|jSrw)rE)rxrrs r)ryz)_get_source_id_assigner..s3< 6r+zAsource_id_key should be either None, a string or a callable. Got z of type .) isinstancercallablerGtype)rrs`r)_get_source_id_assignerrs M3 ' ' 66666 -   B  B B+/ +>+> B B B   r+hashed_documentsIterable[_HashedDocument]Iterator[_HashedDocument]c#~Kt}|D])}|j|vr||j|V*dS)z>Deduplicate a list of hashed documents while preserving order.N)setr7add)rseen hashed_docs r)_deduplicate_in_orderrsZUUD&  4 ' ' HHZ% & & &   r+cBeZdZUdZded< ded< ded< ded<dS)IndexingResultzFReturn a detailed a breakdown of the result of the indexing operation.r_ num_added num_updated num_deleted num_skippedN)rXrYrZr[r\r<r+r)rrsNPPNNN$G&KKr+rdiF) batch_sizecleanuprrcleanup_batch_size force_update docs_source%Union[BaseLoader, Iterable[Document]]record_managerr vector_storerrr$Literal['incremental', 'full', None]rrr:cl|dvrtd|d|dkr|tdddg}|D]'} t|| std |d | (t|jtjkrtd t |t rG |} n@#t$r$t| } YnwxYwt|} t|| } d } d } d }d }t|| D]}ttd |D}fd|D}|dkrXt!||D]'\}}| td|jddd(t%t&t(|}|d|D}g}g}g}t-}t!||D]\}}|r8|r||jn||j?||j|||r)||| | t9|z } |rK|||| t9|t9|z z } |t9|z }|d|D|| |dkr|D]}|t=dt%t&t(|}||| }|r<||| ||t9|z }|dkrn|| |x}rU||| ||t9|z }|| |x}U| || |dS)Index data from the loader into the vector store. Indexing functionality uses a manager to keep track of which documents are in the vector store. This allows us to keep track of which documents were updated, and which documents were deleted, which documents should be skipped. For the time being, documents are indexed using their hashes, and users are not able to specify the uid of the document. IMPORTANT: if auto_cleanup is set to True, the loader should be returning the entire dataset, and not just a subset of the dataset. Otherwise, the auto_cleanup will remove documents that it is not supposed to. Args: docs_source: Data loader or iterable of documents to index. record_manager: Timestamped set to keep track of which documents were updated. vector_store: Vector store to index the documents into. batch_size: Batch size to use when indexing. cleanup: How to handle clean up of documents. - Incremental: Cleans up all documents that haven't been updated AND that are associated with source ids that were seen during indexing. Clean up is done continuously during indexing helping to minimize the probability of users seeing duplicated content. - Full: Delete all documents that haven to been returned by the loader. Clean up runs after all documents have been indexed. This means that users may see duplicated content during indexing. - None: Do not delete any documents. source_id_key: Optional key that helps identify the original source of the document. cleanup_batch_size: Batch size to use when cleaning up documents. force_update: Force update documents even if they are present in the record manager. Useful if you are re-indexing with updated embeddings. Returns: Indexing result which contains information about how many documents were added, updated, deleted, or skipped. >Nfull incrementalzindex..&III..s33IIIr+c&g|] }|Sr<r<rrxsource_id_assigners r)rzindex..#2/ / / (+  s # #/ / / r+]Source ids are required when cleanup mode is incremental. Document that starts with content: r was not assigned as source id.cg|] }|j Sr<rSrs r)rzindex..4s-M-M-M#cg-M-M-Mr+ time_at_leastidscg|] }|j Sr<rSrs r)rzindex..U , , ,SW , , ,r+ group_idsrSource ids cannot be None here.rbeforerrlimitrrrr)!rGhasattrr~rrr|r lazy_loadNotImplementedErrorreloadrget_timerirfrziprCrrrexistsrrr6rnrRupdatermrAssertionError list_keys delete_keys)rrrrrrrrrmethodsmethod doc_iteratorindex_start_dtrrrr doc_batch hashed_docs source_ids source_idr exists_batchuids docs_to_indexuids_to_refresh seen_docs doc_exists _source_idsuids_to_deleters @r)indexrsnn333       -M$9VWWW)G|V,, T|TTFTT    L K$666LMMM+z**) 4&0022LL" 4 4 4 0 0 2 233LLL 4K(( 0??$,,..NIKKKJ 55R3R3  !IIyIII    / / / / /:/ / /  m # #),Z)E)E  % :$$)$.$;DSD$A)))%hsmZ88J%,,-M-M-M-M-MNN  !ee &)+|&D&D ; ; "J  MM*.1111#**:>::: KK  ' ' '  !7!7!9!9 : : : :  0  ! !/ ! P P P 3// /K  *  & &}$ & ? ? ? ]++c)nn< > )K  , , , , , (     m # # ( L L $()JKKK%x}j99K+55%n6N 3##N333**>:::s>222 & . 8 8!);!9! !  n /    / / /  & &~ 6 6 6 3~.. .K!/ 8 8!);!9! !  n /"""   s%B::+C('C(iteratorAsyncIterator[T]cK|D]}|WVdS)z)Convert an iterable to an async iterator.Nr<)ritems r)_to_async_iteratorrs, r+2Union[Iterable[Document], AsyncIterator[Document]]cK|dvrtd|d|dkr|tdddg}|D]'} t|| std |d | (t|jtjkrtd t |t rtd t|d r|} nt|} t|| d{V} d} d} d}d}t|| 23d{V}ttd|D}fd|D}|dkrXt||D]'\}}| td|jddd(t!t"t$|}|d|Dd{V}g}g}g}t)}t||D]\}}|r8|r||jn||j?||j|||r/||| d{V| t5|z } |rQ|||d{V| t5|t5|z z } |t5|z }|d|D|| d{V|dkr|D]}|t9dt!t"t$|}||| d{V}|rH||d{V||d{V|t5|z }6|dkr|| |d{Vx}rg||d{V||d{V|t5|z }|| |d{Vx}g| || |dS)r>Nrrrr{rNradeleteaadd_documentsrrrz>Not supported yet. Please pass an async iterator of documents. __aiter__rcBg|]}t|Sr<rrs r)rzaindex..rr+c&g|] }|Sr<r<rs r)rzaindex..rr+rrrcg|] }|j Sr<rSrs r)rzaindex..s4T4T4TSW4T4T4Tr+rrcg|] }|j Sr<rSrs r)rzaindex..%rr+rrrrrr)rGrr~rrr|rrrr aget_timerqrfrrrCrrraexistsrrr6rnrRaupdatermrr alist_keys adelete_keys)rrrrrrrrrrrasync_doc_iteratorrrrrrrrrrrrrrrrrrrrs @r)aindexrsp333       -M$9VWWW*+G|V,, T|TTFTT    L![%888LMMM+z** ! L   {K((=(/ <<0??*3355555555NIKKK":/ABBS3S3S3S3S3S3S3i !IIyIII    / / / / /:/ / /  m # #),Z)E)E  % :$$)$.$;DSD$A)))%hsmZ88J+334T4T 4T4T4TUUUUUUUU (* !ee &)+|&D&D ; ; "J  MM*.1111#**:>::: KK  ' ' '  !7!7!9!9 : : : :  0 (((WW W W W W W W W 3// /K  *--m-FF F F F F F F F ]++c)nn< > )K $$ , , , , , (%          m # # ( L L $()JKKK%x}j99K#1#<#<%n$=$$N 3"**>:::::::::$11.AAAAAAAAAs>222 gCj&&4&?&?!);'@' ' ! ! ! ! ! !  n /&&~66 6 6 6 6 6 6 6 --n== = = = = = = = 3~.. .K'5&?&?!);'@' ' ! ! ! ! ! !  n /"""   sO )rrrr)r,r-rr)r^r_r`rarrb)r^r_r`rjrrk)rrrsrrt)rrrr)rrrrrrrr_rrrrrsrr_rr:rr)rrarr)rrrrrrrr_rrrrrsrr_rr:rr)0r[ __future__rr"r0r& itertoolsrtypingrrrrr r r r r rrrrrrr)langchain_community.document_loaders.baserlangchain_core.documentsrlangchain_core.pydantic_v1rlangchain_core.vectorstoresrlangchain.indexes.baserrrr*r3r5rirqrrrrrrr<r+r)rsFF"""""" &A@@@@@------555555333333@@@@@@@@ GCLL2222 2222B B B B B hB B B J        "     L L L L LY L L L$48AE#BBBBBBL48AE#GGGGGGGGr+