昺h0#ddlmZddlZddlZddlmZddlmZmZm Z m Z m Z ddl m Z ddlmZer ddlZddlmZdd ZGd deZddZGddeZdS)) annotationsN)Path) TYPE_CHECKINGDictListOptionalUnionDocument) BaseLoader) EntityLikerowdictreturnstrcH|d}|d}|d}|d|d|dS)zBCombine message information in a readable format ready to be used.datefromtextz on z:  )rrsenderrs u/var/www/html/services/rag-service/venv/lib/python3.11/site-packages/langchain_community/document_loaders/telegram.pyconcatenate_rowsrs< v;D [F v;D , ,$ , ,$ , , ,,c"eZdZdZd dZd dZdS) TelegramChatFileLoaderzLoad from `Telegram chat` dump.pathrc||_dS)zInitialize with a path.N) file_path)selfrs r__init__zTelegramChatFileLoader.__init__s rrList[Document]c4t|j}t|d5}tj|}dddn #1swxYwYdd|dD}dt |i}t||gS) Load documents.utf8encodingNc3K|]:}|ddkt|dt)t|V;dS)typemessagerN) isinstancerr).0r,s r z.TelegramChatFileLoader.load..'sX  v)++ 76?C0P0P+ W % %++++  rmessagessource page_contentmetadata)rr openjsonloadjoinrr )r!pfdrr4s rr7zTelegramChatFileLoader.load s   !f % % %  ! A               ww  Z=      c!ff%dX>>>??sAA A N)rrrr#)__name__ __module__ __qualname____doc__r"r7rrrrrsH))@@@@@@rrrUnion[str, List[str]]r#cddlm}t|tr|g}d|D}t |D]\}}|dz|jd<g}|D]}|dgdd }||j}t |D]^\}}t||jd|d  }|jdd |jd |jd<| |_|S)zIConvert a string or list of strings to a list of Documents with metadata.r)RecursiveCharacterTextSplitterc.g|]}t|S))r3r )r.pages r z text_to_docs..8s#>>>t,,,>>>rrEi )r .!?, r)) chunk_size separators chunk_overlap)rEchunkr2-rRr1) langchain.text_splitterrCr-r enumerater4 split_textr3r append) rrC page_docsidoc doc_chunks text_splitterchunksrRs r text_to_docsr^1sVFFFFFF$v>>>>>II&&%%3 1u VJ # #66BBB   ))#*:;;!&)) # #HAu"cl66JUV-W-WC), V(<%V%Vs|G?T%V%VCL "   c " " " "  # rcFeZdZdZ ddd ZddZddZddZddZdS) TelegramChatApiLoaderz)Load `Telegram` chat json directory dump.Ntelegram_data.json chat_entityOptional[EntityLike]api_id Optional[int]api_hash Optional[str]usernamer rcL||_||_||_||_||_dS)aIInitialize with API parameters. Args: chat_entity: The chat entity to fetch data from. api_id: The API ID. api_hash: The API hash. username: The username. file_path: The file path to save the data to. Defaults to "telegram_data.json". N)rbrdrfrhr )r!rbrdrfrhr s rr"zTelegramChatApiLoader.__init__Us,$'     "rrNonec 4Kddlm}g}||j|j|j4d{V}||j23d{V}|jdu}|r |jjnd}| |j |j |j |j||dc6 dddd{Vn#1d{VswxYwYt|jdd5}t#j||dd ddddS#1swxYwYdS) z8Fetch data from Telegram API and save it as a JSON file.r)TelegramClientN) sender_idrr message.idis_reply reply_to_idwzutf-8r'F) ensure_asciiindent) telethon.syncrlrhrdrf iter_messagesrbreply_toreply_to_msg_idrWrmrr isoformatidr5r r6dump)r!rldataclientr,rorpr:s rfetch_data_from_telegramz.TelegramChatApiLoader.fetch_data_from_telegramms(000000!>$-dmLL       PV!'!5!5d6F!G!G       g"+47BJTg.>>PT  %,%6 ' ' 6 6 8 8&-j$,'2     "H!G                           $.# 8 8 8 =A IdAE! < < < < = = = = = = = = = = = = = = = = = =s0B?B+AB?? C  C 'D  DDr| pd.DataFramercd fd ||d}||dd g d td <fd |d D}|S)a Create a dictionary of message threads from the given data. Args: data (pd.DataFrame): A DataFrame containing the conversation data with columns: - message.sender_id - text - date - message.id - is_reply - reply_to_id Returns: dict: A dictionary where the key is the parent message ID and the value is a list of message IDs in ascending order. parent_idint reply_datarr List[int]c||d|kd}g}|D]}||g||zz }|S)a^ Recursively find all replies to a given parent message ID. Args: parent_id (int): The parent message ID. reply_data (pd.DataFrame): A DataFrame containing reply messages. Returns: list: A list of message IDs that are replies to the parent message ID. rprn)tolist)rrdirect_replies all_repliesreply_id find_repliess rrz@TelegramChatApiLoader._get_message_threads..find_repliessj( =(AY(NOfhh  K* O OzLL:,N,NNN  rrorp)subsetc2i|]}||g|zSrr)r.rrreply_messagess r z>TelegramChatApiLoader._get_message_threads..s>     {\\)^%L%LL   rrn)rrrrrr)dropnaastyper)r!r|parent_messagesmessage_threadsrrs @@r_get_message_threadsz*TelegramChatApiLoader._get_message_threadss$      0Z 001d:./66}o6NN)7}(E(L(LS(Q(Q}%     ,\:    rrDict[int, List[int]]cJd}|D]y\}}||d|dd}d|D}|d|dzz }z|S) aw Combine the message texts for each parent message ID based on the list of message threads. Args: message_threads (dict): A dictionary where the key is the parent message ID and the value is a list of message IDs in ascending order. data (pd.DataFrame): A DataFrame containing the conversation data: - message.sender_id - text - date - message.id - is_reply - reply_to_id Returns: str: A combined string of message texts sorted by date. r)rnr)byrc,g|]}t|Sr)r)r.elems rrFz@TelegramChatApiLoader._combine_message_texts..sAAA4SYYAAArrMz. )itemsisin sort_valuesrr8strip)r!rr| combined_textr message_ids message_textss r_combine_message_textsz,TelegramChatApiLoader._combine_message_textss* '6&;&;&=&= = = "I{T,',,[99:''0  BA=AAAM SXXm44u< AA" B++B/2B/7B<<C)NNNNra) rbrcrdrerfrgrhrgr r)rrj)r|rrr)rrr|rrrr<) r=r>r?r@r"r~rrr7rrrr`r`Rs33-1 $"&"&- #####0====.9999v$%$%$%$%L",",",",",",rr`)rrrr)rrArr#) __future__rrr6pathlibrtypingrrrrr langchain_core.documentsr )langchain_community.document_loaders.baser rrtelethon.hintsr rrr^r`rrrrsH"""""" ==============------@@@@@@*))))))----@@@@@Z@@@0Bu,u,u,u,u,Ju,u,u,u,u,r