hndZddlZddlZddlZddlmZmZmZddlm Z ddl m Z ddl m Z ddlmZddlmZdd lmZmZdd lmZdd lmZdd lmZdd lmZmZmZedgdZGddZ GddZ!GddZ"GddZ#Gdde#Z$dZ%e&dkr e%gdZ'dS)a  This module brings together a variety of NLTK functionality for text analysis, and provides simple, interactive interfaces. Functionality includes: concordancing, collocation discovery, regular expression search over tokenized strings, and distributional similarity. N)Counter defaultdict namedtuple)reduce)log)BigramCollocationFinder)MLE)padded_everygram_pipeline)BigramAssocMeasures f_measure)ConditionalFreqDist)FreqDist) sent_tokenize)LazyConcatenation cut_string tokenwrapConcordanceLine)leftqueryrightoffset left_print right_printlinecTeZdZdZedZdddfdZdZdZd d Z d d Z dS) ContextIndexa A bidirectional index between words and their 'contexts' in a text. The context of a word is usually defined to be the words that occur in a fixed window around the word; but other definitions may also be used by providing a custom context function. c|dkr||dz nd}|t|dz kr||dznd}||fS)z;One left token and one right token, normalized to lowercaser*START**END*)lowerlen)tokensirrs Q/var/www/html/services/rag-service/venv/lib/python3.11/site-packages/nltk/text.py_default_contextzContextIndex._default_context/sf)*Qva!e}""$$$I)*c&kkAo)=)=q1u ##%%%7e}Nc|SNxs r%zContextIndex.6sQr'c,|__|r|_n j_rfdDt fdt D_t fdt D_dS)Nc*g|]}| |Sr*r*).0tfilters r% z)ContextIndex.__init__..>s&555A66!995a555r'c3tK|]2\}}||fV3dSr))_key _context_funcr0r$wselfr#s r% z(ContextIndex.__init__..?sW% % >BaTYYq\\4--fa88 9% % % % % % r'c3tK|]2\}}||fV3dSr))r6r5r7s r%r:z(ContextIndex.__init__..BsW% % >BaT   * *DIIaLL 9% % % % % % r')r5_tokensr6r&CFD enumerate_word_to_contexts_context_to_words)r9r# context_funcr2keys`` ` r%__init__zContextIndex.__init__6s   7!-D  !%!6D   65555555F!$% % % % % FOPVFWFW% % % " " "%% % % % % FOPVFWFW% % % " " r'c|jS)zw :rtype: list(str) :return: The document that this context index was created from. r<r9s r%r#zContextIndex.tokensF |r'c||}t|j|}i}|jD]%\}}t |t|||<&|S)z Return a dictionary mapping from words to 'similarity scores,' indicating how often these two words occur in the same context. )r5setr?itemsr )r9word word_contextsscoresr8 w_contextss r%word_similarity_dictz!ContextIndex.word_similarity_dictNsq yyD24899 !399;; B BMAz!-ZAAF1II r'cDtt}|j||D]M}|j|D]=}||kr5||xx|j|||j||zz cc<>Nt ||jdd|S)NT)rBreverse)rintr?r5r@sortedget)r9rKnrMcr8s r% similar_wordszContextIndex.similar_words]sS!!' $8  A+A.  991III.q1$7$:PQR:STU:VVIII  f&*d;;;BQB??r'FcrfdDfdDfdttD}ttj|r%|r#t ddstStfdD}|S)a Find contexts where the specified words can all appear; and return a frequency distribution mapping each context to the number of times that context was used. :param words: The words used to seed the similarity search :type words: str :param fail_on_unknown: If true, then raise a value error if any of the given words do not occur at all in the index. c:g|]}|Sr*)r5r0r8r9s r%r3z0ContextIndex.common_contexts..rs#---!1---r'cDg|]}tj|Sr*)rIr?r[s r%r3z0ContextIndex.common_contexts..ss)BBBqC.q122BBBr'c0g|]}| |Sr*r*)r0r$contextswordss r%r3z0ContextIndex.common_contexts..ts&HHHaHQKHqHHHr'z%The following word(s) were not found: c3DK|]}j|D] }|v|V dSr))r?)r0r8rWcommonr9s r%r:z/ContextIndex.common_contexts..|sI$*@*C%&qF{{{{{{{r')ranger"rrI intersection ValueErrorjoinr)r9r_fail_on_unknownemptyfdrbr^s`` @@r%common_contextszContextIndex.common_contextsgs.---u---BBBBEBBBHHHHH5U#4#4HHH((33  _ DchhuooVV V ::  BIr'rP)F) __name__ __module__ __qualname____doc__ staticmethodr&rCr#rOrXrjr*r'r%rr's\ -1;;       @@@@r'rc@eZdZdZdfdZdZdZdZd dZd d Z d S)ConcordanceIndexzs An index that can be used to look up the offset locations at which a given word occurs in a document. c|Sr)r*r+s r%r-zConcordanceIndex.sQr'c||_ ||_ tt|_ t |D]:\}}||}|j||;dS)a Construct a new concordance index. :param tokens: The document (list of tokens) that this concordance index was created from. This list can be used to access the context of a given word occurrence. :param key: A function that maps each token to a normalized version that will be used as a key in the index. E.g., if you use ``key=lambda s:s.lower()``, then the index will be case-insensitive. N)r<r5rlist_offsetsr>append)r9r#rBindexrKs r%rCzConcordanceIndex.__init__s   D#D)) L$V,, . .KE499T??D M$  & &u - - - - . .r'c|jS)z{ :rtype: list(str) :return: The document that this concordance index was created from. rErFs r%r#zConcordanceIndex.tokensrGr'cF||}|j|S)z :rtype: list(int) :return: A list of the offset positions at which the given word occurs. If a key function was specified for the index, then given word's key will be looked up. )r5rvr9rKs r%offsetszConcordanceIndex.offsetss yy}T""r'cXdt|jt|jfzS)Nz+)r"r<rvrFs r%__repr__zConcordanceIndex.__repr__s/<      @   r'Pc t|tr|}n|g}d|}td|D}||z dz dz}|dz}g}||d} t |ddD]H\}fd||D} t | | } I| r| D]d|jt|z} |jtd|z } |jt|z|z} td| |  |}td| |}d|| |g}t| | | |||}|||S) z Find all concordance lines given the query word. Provided with a list of words, these will be found as a phrase. r`c3BK|]}tj|dVdS)rN) unicodedata combining)r0chars r%r:z4ConcordanceIndex.find_concordance..s2UUt9Nt9T9TUUUUUUUr'rrNc h|] }|z dz  S)rr*)r0rr$s r% z4ConcordanceIndex.find_concordance..s!LLLvFQJNLLLr') isinstancerurfsumr|r>rTrdr<r"maxrrjustrrw)r9rKwidthphrase phrase_str phrase_len half_widthcontextconcordance_listr| word_offsets query_word left_context right_contextrr line_printconcordance_liner$s @r%find_concordancez!ConcordanceIndex.find_concordances dD ! ! FFVFXXf%% UUzUUUUU j(1,2 1*,,vay)) ,, A AGAtLLLLd9K9KLLLL\66w??@@GG  : : : XXdl1q3v;;3F&GHH #|C1w;,?,?!,CD $ QV_q7{-J K '(>(> LLRR )-)@)@*MM  XXz:{&KLL #2 !$$ !''(89999r'c<|||}|stddSt|t|}td|dt|dt |d|D]\}}t|jdS)a Print concordance lines given the query word. :param word: The target word or phrase (a list of strings) :type word: str or list :param lines: The number of lines to display (default=25) :type lines: int :param width: The width of each line, in characters (default=80) :type width: int :param save: The option to save the concordance. :type save: bool )rz no matchesz Displaying z of z matches:N)rprintminr"r>r)r9rKrlinesrr$rs r%print_concordancez"ConcordanceIndex.print_concordances 00U0CC - ,     s#34455E KKK3/?+@+@KKK L L L'01A&5&1I'J'J - -##&+,,,, - -r'N)r)rr) rlrmrnrorCr#r|r~rrr*r'r%rrrrs $/;....4###   . . . . `------r'rrceZdZdZdZdZdS) TokenSearchera A class that makes it easier to use regular expressions to search over tokenized strings. The tokenized string is converted to a string where tokens are marked with angle brackets -- e.g., ``''``. The regular expression passed to the ``findall()`` method is modified to treat angle brackets as non-capturing parentheses, in addition to matching the token boundaries; and to have ``'.'`` not match the angle brackets. cNdd|D|_dS)Nc3&K|] }d|zdzV dS)<>Nr*)r0r8s r%r:z)TokenSearcher.__init__.. s*::aC!GcM::::::r')rf_raw)r9r#s r%rCzTokenSearcher.__init__ s(GG::6::::: r'c~tjdd|}tjdd|}tjdd|}tjdd|}tj||j}|D];}|ds$|drt d >> from nltk.text import TokenSearcher >>> from nltk.book import text1, text5, text9 >>> text5.findall("<.*><.*>") you rule bro; telling you bro; u twizted bro >>> text1.findall("(<.*>)") monied; nervous; dangerous; white; white; white; pious; queer; good; mature; white; Cape; great; wise; wise; butterless; white; fiendish; pale; furious; better; certain; complete; dismasted; younger; brave; brave; brave; brave >>> text9.findall("{3,}") thread through those; the thought that; that the thing; the thing that; that that thing; through these than through; them that the; through the thick; them that they; thought that the :param regexp: A regular expression :type regexp: str z\srrz(?:<(?:rz)>)z (?]z$Bad regexp for TokenSearcher.findallcHg|]}|ddd S)rz><splitr0hs r%r3z)TokenSearcher.findall..5s,222!B$ d##222r')resubfindallr startswithendswithre)r9regexphitsrs r%rzTokenSearcher.findalls0r6**i00eV,, ff55z&$),, I IA<<$$ IC I !GHHH32T222 r'N)rlrmrnrorCrr*r'r%rrs<;;;'''''r'rceZdZdZdZd!dZdZdZd"d Zd"d Z d#d Z d#dZ dZ dZ dZd$dZd$dZdZd%dZd&dZdZdZdZejdZdZdZd ZdS)'Texta A wrapper around a sequence of simple (string) tokens, which is intended to support initial exploration of texts (via the interactive console). Its methods perform a variety of analyses on the text's contexts (e.g., counting, concordancing, collocation discovery), and display the results. If you wish to write a program which makes use of these analyses, then you should bypass the ``Text`` class, and use the appropriate analysis function or class directly instead. A ``Text`` is typically initialized from a given document or corpus. E.g.: >>> import nltk.corpus >>> from nltk.text import Text >>> moby = Text(nltk.corpus.gutenberg.words('melville-moby_dick.txt')) TNcb|jrt|}||_|r ||_dSd|ddvrK|ddd}dd|d|D|_dSdd|ddDd z|_dS) zv Create a Text object. :param tokens: The source text. :type tokens: sequence of str ]NrPr`c34K|]}t|VdSr)strr0toks r%r:z Text.__init__..bs( C CcS C C C C C Cr'rc34K|]}t|VdSr)rrs r%r:z Text.__init__..ds( @ @cS @ @ @ @ @ @r'z...) _COPY_TOKENSrur#namerxrf)r9r#rends r%rCz Text.__init__Ss   "&\\F  IDIII F3B3K  "+##C((C C CVAcE] C C CCCDIII @ @VBQBZ @ @ @@@5HDIIIr'c|j|Sr))r#)r9r$s r% __getitem__zText.__getitem__js{1~r'c*t|jSr))r"r#rFs r%__len__z Text.__len__ms4;r'Orcd|jvrt|jd|_|j|||S)a Prints a concordance for ``word`` with the specified context window. Word matching is not case-sensitive. :param word: The target word or phrase (a list of strings) :type word: str or list :param width: The width of each line, in characters (default=80) :type width: int :param lines: The number of lines to display (default=25) :type lines: int :seealso: ``ConcordanceIndex`` _concordance_indexc*|Sr)r!ss r%r-z"Text.concordance..17799r'rB)__dict__rrr#rrr9rKrrs r% concordancezText.concordancetsP t} 4 4&6 !4!4'''D #&88ueLLLr'cd|jvrt|jd|_|j||d|S)a Generate a concordance for ``word`` with the specified context window. Word matching is not case-sensitive. :param word: The target word or phrase (a list of strings) :type word: str or list :param width: The width of each line, in characters (default=80) :type width: int :param lines: The number of lines to display (default=25) :type lines: int :seealso: ``ConcordanceIndex`` rc*|Sr)rrs r%r-z'Text.concordance_list..rr'rN)rrrr#rrrs r%rzText.concordance_listsW t} 4 4&6 !4!4'''D #&77eDDVeVLLr'rPrcd|jvr|j|kr |j|ks||_||_ddlm}|dt j|j|}| d| fdt}t| |j||_|jS)a Return collocations derived from the text, ignoring stopwords. >>> from nltk.book import text4 >>> text4.collocation_list()[:2] [('United', 'States'), ('fellow', 'citizens')] :param num: The maximum number of collocations to return. :type num: int :param window_size: The number of tokens spanned by a collocation (default=2) :type window_size: int :rtype: list(tuple(str, str)) _collocationsr) stopwordsenglishrcVt|dkp|vS)N)r"r!)r8 ignored_wordss r%r-z'Text.collocation_list..s#s1vvz/WQWWYY-=Wr')r_num _window_size nltk.corpusrr_r from_wordsr#apply_freq_filterapply_word_filterr runbestlikelihood_ratior)r9num window_sizerfinderbigram_measuresrs @r%collocation_listzText.collocation_lists t} , , S  ![00DI +D  . - - - - -%OOI66M,7 [QQF  $ $Q ' ' '  $ $%W%W%W%W X X X133O!% _=sCC""D !!r'cd|||D}tt|ddS)a Print collocations derived from the text, ignoring stopwords. >>> from nltk.book import text4 >>> text4.collocations() # doctest: +NORMALIZE_WHITESPACE United States; fellow citizens; years ago; four years; Federal Government; General Government; American people; Vice President; God bless; Chief Justice; one another; fellow Americans; Old World; Almighty God; Fellow citizens; Chief Magistrate; every citizen; Indian tribes; public debt; foreign nations :param num: The maximum number of collocations to print. :type num: int :param window_size: The number of tokens spanned by a collocation (default=2) :type window_size: int c$g|] \}}|dz|zSr`r*r0w1w2s r%r3z%Text.collocations..s1   $b"BHrM   r'; ) separatorN)rrr)r9rrcollocation_stringss r% collocationszText.collocationssU&  (,(=(=c;(O(O    i+t<<<=====r'c6|j|S)zJ Count the number of times this word appears in the text. )r#countr{s r%rz Text.count{  &&&r'c6|j|S)zQ Find the index of the first occurrence of the word in the text. )r#rxr{s r%rxz Text.indexrr'ctr))NotImplementedError)r9methods r% readabilityzText.readabilitys!!r'cd|jvrt|jdd|_|jjvrttfdD}d| |D}tt|dStddS) a~ Distributional similarity: find other words which appear in the same contexts as the specified word; list most similar words first. :param word: The word used to seed the similarity search :type word: str :param num: The number of words to generate (default=20) :type num: int :seealso: ContextIndex.similar_words() _word_context_indexc*|Sr))isalphar+s r%r-zText.similar..saiikkr'c*|Sr)rrs r%r-zText.similar..sr')r2rBc3FK|]}|D]}|v|k |VdSr)r*)r0r8rWr^wcirKs r%r:zText.similar..sWQ==d*3 r'cg|]\}}|Sr*r*r0r8_s r%r3z Text.similar..s77741aQ777r'z No matchesN) rrr#rr!r? conditionsrIr most_commonrr)r9rKrrir_r^rs ` @@r%similarz Text.similars ! 5 5'3 $9$9?R?R(((D $ zz||&8 3>>## # #3t9~~H))B 872>>##6#6777E )E"" # # # # # ,     r'cd|jvrt|jd|_ |j|d}|st ddSd||D}t td|DdS#t$r}t |Yd}~dSd}~wwxYw) aY Find contexts where the specified words appear; list most frequent common contexts first. :param words: The words used to seed the similarity search :type words: str :param num: The number of words to generate (default=20) :type num: int :seealso: ContextIndex.common_contexts() rc*|Sr)rrs r%r-z&Text.common_contexts..rr'rTzNo common contexts were foundcg|]\}}|Sr*r*rs r%r3z(Text.common_contexts.. s"E"E"EA1"E"E"Er'c3,K|]\}}|dz|zVdS)rNr*rs r%r:z'Text.common_contexts..!s.LL&"bS2 LLLLLLr'N) rrr#rrjrrrre)r9r_rriranked_contextses r%rjzText.common_contexts s ! 5 5'3 !4!4(((D $ )99%FFB N566666"E"E1D1D"E"E"EiLLOLLLLLMMMMM    !HHHHHHHHH s,BAB C&B;;Cc*ddlm}|||dS)z Produce a plot showing the distribution of the words through the text. Requires pylab to be installed. :param words: The words to be plotted :type words: list(str) :seealso: nltk.draw.dispersion_plot() r)dispersion_plotN) nltk.drawr)r9r_rs r%rzText.dispersion_plot&s. .-----e$$$$$r'rcxt||\}}t|}||||S)N)order)r r fit)r9tokenized_sentsrV train_data padded_sentsmodels r%_train_default_ngram_lmzText._train_default_ngram_lm3s<#.Is/! ! ! $DJJsOO! ! ! r'r`_trigram_modelzBuilding ngram index...)filer)rVrz!The `length` must be more than 0.) text_seed random_seedzzrrN)rrfr#_tokenized_sentshasattrrsysstderrrr r"r>generaterwr) r9lengthr"r#generated_tokensidxtokenprefix output_strs r%r(z Text.generate9s ! ! (5chht{6K6K(L(L! ! ! t-..  +#* = = = ="&">">%#?##D zzz>zzz"##f,,'#,,i[- / / U E>>F??E ''.... 1 K"##f,,/8?)$$s**Ri(8&(ABBB  jr'c:|j|S)zc See documentation for FreqDist.plot() :seealso: nltk.prob.FreqDist.plot() )vocabplot)r9argss r%r1z Text.plotgs !tzz|| $''r'cJd|jvrt||_|jS)z. :seealso: nltk.prob.FreqDist _vocab)rrr4rFs r%r0z Text.vocabns& 4= ( ("4..DK{r'cd|jvrt||_|j|}d|D}t t |ddS)a Find instances of the regular expression in the text. The text is a list of tokens, and a regexp pattern to match a single token must be surrounded by angle brackets. E.g. >>> from nltk.book import text1, text5, text9 >>> text5.findall("<.*><.*>") you rule bro; telling you bro; u twizted bro >>> text1.findall("(<.*>)") monied; nervous; dangerous; white; white; white; pious; queer; good; mature; white; Cape; great; wise; wise; butterless; white; fiendish; pale; furious; better; certain; complete; dismasted; younger; brave; brave; brave; brave >>> text9.findall("{3,}") thread through those; the thought that; that the thing; the thing that; that that thing; through these than through; them that the; through the thick; them that they; thought that the :param regexp: A regular expression :type regexp: str _token_searcherc8g|]}d|Sr)rfrs r%r3z Text.findall..s"*** ***r'rN)rrr6rrr)r9rrs r%rz Text.findallwsh. DM 1 1#0#6#6D #++F33**T*** id##$$$$$r'z \w+|[\.\!\?]c|dz }|dkrK|j||s+|dz}|dkr |j||+|dkr||nd}|dz}|t|krX|j||s8|dz }|t|kr |j||8|t|kr||nd}||fS)z One left & one right token, both case-normalized. Skip over non-sentence-final punctuation. Used by the ``ContextIndex`` that is created for ``similar()`` and ``common_contexts()``. rrrr ) _CONTEXT_REmatchr")r9r#r$jrrs r%_contextz Text._contexts E1ffT-33F1I>>f FA1ffT-33F1I>>fFFvayy  E#f++ood&6&<&rrFs r%__str__z Text.__str__di''r'cd|jzSr>r?rFs r%r~z Text.__repr__rAr'r))rr)rPrrk)r)rNr)rlrmrnrorrCrrrrrrrrxrr rjrrr(r1r0rrcompiler9r<r@r~r*r'r%rr9s.LIIII.   MMMM*MMMM(!"!"!"!"F>>>>0''' ''' """     D8 % % % ,,,,\(((%%%D"*_--K0((((((((r'rc*eZdZdZdZdZdZdZdS)TextCollectiona;A collection of texts, which can be loaded with list of texts, or with a corpus consisting of one or more texts, and which supports counting, concordancing, collocation discovery, etc. Initialize a TextCollection as follows: >>> import nltk.corpus >>> from nltk.text import TextCollection >>> from nltk.book import text1, text2, text3 >>> gutenberg = TextCollection(nltk.corpus.gutenberg) >>> mytexts = TextCollection([text1, text2, text3]) Iterating over a TextCollection produces all the tokens of all the texts in order. ctdr fdD|_t|t i|_dS)Nr_c:g|]}|Sr*)r_)r0fsources r%r3z+TextCollection.__init__..s#@@@!fll1oo@@@r')r%fileids_textsrrCr _idf_cache)r9rIs `r%rCzTextCollection.__init__sh 67 # # A@@@@v~~/?/?@@@F  d-f55666r'cL||t|z S)z"The frequency of the term in text.)rr"r9termtexts r%tfzTextCollection.tfszz$#d))++r'c2|j}|ytfd|jD}t|jdkrt d|r$t t|j|z nd}||j<|S)zThe number of texts in the corpus divided by the number of texts that the term appears in. If a term does not appear in the corpus, 0.0 is returned.Ncg|]}|vd S)Tr*)r0rPrOs r%r3z&TextCollection.idf..sHHHD44<<4<<rnsS 8888888888555555;;;;;;77777777777777%%%%%%''''''>>>>>>>>>>*MMM XXXXXXXXv|-|-|-|-|-|-|-|-~55555555p~(~(~(~(~(~(~(~(D +4+4+4+4+4T+4+4+4\999< zDFFF   r'