Kh ddlmZesdevrddlmZnddlZ ddlZn#e $rddlZYnwxYwdZ dZ dZ d Z Gd d eZGd d eZejeGddeZejeGddeZejeGddeZejedZdZGddeZejeGddeZejedZddl Z ddl!Z!ddlZddl"Z"ddl#Z$ddl%m&Z&ddl%m'Z'dZ(dZ)ej*Z+ej*Z,e-edej.e-edej.ej/e_0ej1e_2d D]Z3e)ee3e(ee(ee(eeZ4eZ5dd!l6m7Z7ee"j89e:e$j;<d"d#Gd$d%eZ=dS)&) version_info.)_sentencepieceNc d|jz}n#tj$rd}YnwxYwd|jjd|jjd|dS)Nz proxy of )this__repr__ __builtin__ Exception __class__ __module____name__)selfstrthiss ^/var/www/html/services/rag-service/venv/lib/python3.11/site-packages/sentencepiece/__init__.py _swig_reprrss 2 2 4 44  !^6668O8O8OQXQXQX ZZs 33cfd}|S)Nc4|dkr|||dS|dkr|j|dSt||r?tt t ||t r|||dStd|z)Nr thisownz(You cannot add instance attributes to %s)r ownhasattr isinstancegetattrtypepropertyAttributeError)rnamevaluesets rset_instance_attrzE_swig_setattr_nondynamic_instance_variable..set_instance_attrs 6>> CdE " " " " " Y   IMM% T4  TZT D0I0I8%T%T T CdE " " " " " !Kd!RSS S)r!r"s` r*_swig_setattr_nondynamic_instance_variabler%s)TTTTT r#cfd}|S)Nct||r2tt||ts|||dSt d|z)Nz%You cannot add class attributes to %s)rrrrr)clsrr r!s rset_class_attrz?_swig_setattr_nondynamic_class_variable..set_class_attr)s` 3   Pjd1C1CX&N&N P CT5 ! ! ! ! ! !H3!NOO Or#r$)r!r)s` r'_swig_setattr_nondynamic_class_variabler*(s)PPPPP r#cfd}|S)zlClass decorator for adding a metaclass to a SWIG wrapped class - a slimmed down version of six.add_metaclassc`|j|j|jSN)r __bases____dict__copy)r( metaclasss rwrapperz$_swig_add_metaclass..wrapper3s)ys}cl6G6G6I6IJJJr#r$)r1r2s` r_swig_add_metaclassr31s&KKKKK Nr#c2eZdZdZeejZdS)_SwigNonDynamicMetazKMeta class to enforce nondynamic attributes (no new attributes) for a classN)rr __qualname____doc__r*r __setattr__r$r#rr5r58s'UU99$:JKKKKKr#r5c eZdZedddZeZdZej Z dZ dZ dZ d Zd Zd Zd Zee ZeeZee ZeeZee ZeeZeeZd ZdZdZeZdS)1ImmutableSentencePieceText_ImmutableSentencePiecec4|jSr-r rxs rz:ImmutableSentencePieceText_ImmutableSentencePiece.>r#c6|j|Sr-r<r>vs rr?z:ImmutableSentencePieceText_ImmutableSentencePiece.>AFJJqMMr#The membership flagdoccRtj|tjdSr-)r:ImmutableSentencePieceText_ImmutableSentencePiece_swiginit5new_ImmutableSentencePieceText_ImmutableSentencePiecers r__init__z:ImmutableSentencePieceText_ImmutableSentencePiece.__init__AsAQRVXfY]Y_Y_ ` ` ` ` `r#c*tj|Sr-)r8ImmutableSentencePieceText_ImmutableSentencePiece__piecerKs r_piecez8ImmutableSentencePieceText_ImmutableSentencePiece._pieceEVW[\\\r#c*tj|Sr-)r:ImmutableSentencePieceText_ImmutableSentencePiece__surfacerKs r_surfacez:ImmutableSentencePieceText_ImmutableSentencePiece._surfaceHsXY]^^^r#c*tj|Sr-)r5ImmutableSentencePieceText_ImmutableSentencePiece__idrKs r_idz5ImmutableSentencePieceText_ImmutableSentencePiece._idKsSTXYYYr#c*tj|Sr-)r8ImmutableSentencePieceText_ImmutableSentencePiece__beginrKs r_beginz8ImmutableSentencePieceText_ImmutableSentencePiece._beginNrPr#c*tj|Sr-)r6ImmutableSentencePieceText_ImmutableSentencePiece__endrKs r_endz6ImmutableSentencePieceText_ImmutableSentencePiece._endQsTUYZZZr#c*tj|Sr-)rCImmutableSentencePieceText_ImmutableSentencePiece__surface_as_bytesrKs r_surface_as_byteszCImmutableSentencePieceText_ImmutableSentencePiece._surface_as_bytesTsabfgggr#c*tj|Sr-)rAImmutableSentencePieceText_ImmutableSentencePiece__piece_as_bytesrKs r_piece_as_byteszAImmutableSentencePieceText_ImmutableSentencePiece._piece_as_bytesWs_`deeer#cfd|j|j|j|j|jS)Nz3piece: "{}" id: {} surface: "{}" begin: {} end: {} )formatpieceidsurfacebeginendrKs r__str__z9ImmutableSentencePieceText_ImmutableSentencePiece.__str__bs2"6$*dgt|"&*dh88 8r#c|j|jko?|j|jko/|j|jko|j|jko|j|jkSr-)rerfrgrhrirothers r__eq__z8ImmutableSentencePieceText_ImmutableSentencePiece.__eq__jsv Z5; &Y47eh+>Y4vr@r#c6|j|Sr-r<rBs rr?z#ImmutableSentencePieceText.vrDr#rErFcRtj|tjdSr-)r#ImmutableSentencePieceText_swiginitnew_ImmutableSentencePieceTextrKs rrLz#ImmutableSentencePieceText.__init__ys$:4AnApApqqqqqr#c*tj|Sr-)r'ImmutableSentencePieceText__pieces_sizerKs r _pieces_sizez'ImmutableSentencePieceText._pieces_size}sEdKKKr#c,tj||Sr-)r"ImmutableSentencePieceText__piecesrindexs r_piecesz"ImmutableSentencePieceText._piecess@uMMMr#c*tj|Sr-)r ImmutableSentencePieceText__textrKs r_textz ImmutableSentencePieceText._texts>tDDDr#c*tj|Sr-)r!ImmutableSentencePieceText__scorerKs r_scorez!ImmutableSentencePieceText._scores?EEEr#c*tj|Sr-)r,ImmutableSentencePieceText_SerializeAsStringrKs rSerializeAsStringz,ImmutableSentencePieceText.SerializeAsStringJ4PPPr#c*tj|Sr-)r)ImmutableSentencePieceText__text_as_bytesrKs r_text_as_bytesz)ImmutableSentencePieceText._text_as_bytessGMMMr#c*eZdZdZdZdZdZeZdS)9ImmutableSentencePieceText.ImmutableSentencePieceIteratorcP||_|j|_dSr-)protorlenrrs rrLzBImmutableSentencePieceText.ImmutableSentencePieceIterator.__init__" :**,,r#c|jSr-rrKs r__len__zAImmutableSentencePieceText.ImmutableSentencePieceIterator.__len__ xr#c2t|tr8fdtjD|j|j|jS|dkr |jz}|dks |jkrtdj |S)NcDg|]}j|Sr$)rr.0irs r zYImmutableSentencePieceText.ImmutableSentencePieceIterator.__getitem__..)AAAA$*$$Q''AAAr#rzpiece index is out of range) rslicerangerstartstopstep IndexErrorrrrs` r __getitem__zEImmutableSentencePieceText.ImmutableSentencePieceIterator.__getitem__s eU # # eAAAAtxAAA%+ejY^YcBcd d 199$("% 199))899 9z!!%(((r#c@dd|DS)N cRg|]$}dt|%Szpieces {{ {}}}rdrqrr>s rrzUImmutableSentencePieceText.ImmutableSentencePieceIterator.__str__..-IIIq+223q66::IIIr#joinrKs rrjzAImmutableSentencePieceText.ImmutableSentencePieceIterator.__str__#yyIIDIIIJJJr#Nrrr6rLrrrjr r$r#rImmutableSentencePieceIteratorrT---)))KKKhhhr#rc6t|Sr-)rxrrKs rpiecesz!ImmutableSentencePieceText.piecess ' F Ft L LLr#cV||kSr-rrls rrnz!ImmutableSentencePieceText.__eq__%  # # % %)@)@)B)B BBr#cDt|Sr-rprrKs rrrz#ImmutableSentencePieceText.__hash__ $((** + ++r#cd|j|jdd|jDS)Nztext: "{}" score: {} {}rcRg|]$}dt|%Srrrs rrz6ImmutableSentencePieceText.__str__..s-%\%\%\1&7&>&>s1vv&F&F%\%\%\r#)rdtextscorerrrKs rrjz"ImmutableSentencePieceText.__str__sHF49dj99%\%\PTP[%\%\%\]]___r#N)rrr6rrrr rLr!delete_ImmutableSentencePieceTextrtrrrrrrr text_as_bytesrrrrnrrrjr$r#rrxrxusfh--/I/IOdeeeGHrrr%GLLLNNNEEEFFFQQQNNN 8E??DH^,,M HV  E,MMXMCCC,,,___ HHHr#rxceZdZedddZeZdZej Z dZ dZ dZ Gd d Zed Zd Zd ZdZeZdS)ImmutableNBestSentencePieceTextc4|jSr-r<r=s rr?z(ImmutableNBestSentencePieceText.r@r#c6|j|Sr-r<rBs rr?z(ImmutableNBestSentencePieceText.rDr#rErFcRtj|tjdSr-)r(ImmutableNBestSentencePieceText_swiginit#new_ImmutableNBestSentencePieceTextrKs rrLz(ImmutableNBestSentencePieceText.__init__s$?nFxFzFz{{{{{r#c*tj|Sr-)r,ImmutableNBestSentencePieceText__nbests_sizerKs r _nbests_sizez,ImmutableNBestSentencePieceText._nbests_sizerr#c,tj||Sr-)r'ImmutableNBestSentencePieceText__nbestsrs r_nbestsz'ImmutableNBestSentencePieceText._nbestssEdERRRr#c*tj|Sr-)r1ImmutableNBestSentencePieceText_SerializeAsStringrKs rrz1ImmutableNBestSentencePieceText.SerializeAsStringsOPTUUUr#c*eZdZdZdZdZdZeZdS)BImmutableNBestSentencePieceText.ImmutableSentencePieceTextIteratorcP||_|j|_dSr-)rrrrs rrLzKImmutableNBestSentencePieceText.ImmutableSentencePieceTextIterator.__init__rr#c|jSr-rrKs rrzJImmutableNBestSentencePieceText.ImmutableSentencePieceTextIterator.__len__rr#c2t|tr8fdtjD|j|j|jS|dkr |jz}|dks |jkrtdj |S)NcDg|]}j|Sr$)rrrs rrzbImmutableNBestSentencePieceText.ImmutableSentencePieceTextIterator.__getitem__..rr#rznbests index is out of range) rrrrrrrrrrrs` rrzNImmutableNBestSentencePieceText.ImmutableSentencePieceTextIterator.__getitem__s eU # # eAAAAtxAAA%+ejY^YcBcd d 199$("% 199))9:: :z!!%(((r#c@dd|DS)NrcRg|]$}dt|%Sznbests {{ {}}}rrs rrz^ImmutableNBestSentencePieceText.ImmutableSentencePieceTextIterator.__str__..rr#rrKs rrjzJImmutableNBestSentencePieceText.ImmutableSentencePieceTextIterator.__str__rr#Nrr$r#r"ImmutableSentencePieceTextIteratorrrr#rc6t|Sr-)rrrKs rnbestsz&ImmutableNBestSentencePieceText.nbestss , O OPT U UUr#cV||kSr-rrls rrnz&ImmutableNBestSentencePieceText.__eq__rr#cDt|Sr-rrKs rrrz(ImmutableNBestSentencePieceText.__hash__rr#cJdd|jDS)NrcRg|]$}dt|%Srrrs rrz;ImmutableNBestSentencePieceText.__str__..s-NNNQ)00Q88NNNr#)rrrKs rrjz'ImmutableNBestSentencePieceText.__str__s% YYNN$+NNN O OOr#N)rrr6rrrr rLr&delete_ImmutableNBestSentencePieceTextrtrrrrrrnrrrjr$r#rrrsh--/I/IOdeeeGH|||%LQQQSSSVVV,VVXVCCC,,,PPPHHHr#rc eZdZedddZeZdZej Z dZ dZ dZ d Zd Zd Zd Zd ZdZdZdZdZdZdZdZdZdZdZdZdZdZdZ dZ!dZ"dZ#dZ$d Z%d!Z&d"Z'd#Z(d$Z)d%Z*d&Z+d'Z,d(Z-d)Z.d*Z/d+Z0d,Z1d-Z2d.Z3d/Z4d0Z5d1Z6d2Z7d3Z8d4Z9d5Z:d6Z;d7Zd:Z?d;Z@d<ZAd=ZBd>d>eCd?d?d?d?d?d@dAd@f dBZD didCZEdDZFdEZGdFZHdGZIdjdHZJdjdIZKdjdJZLdjdKZM dkdLZNdldMZOdldNZPdldOZQdldPZR didQZSdjdRZTdjdSZUdjdTZVdjdUZWeXd>fdVZYeXfdWZZeXfdXZ[dmdZZ\dmd[Z]dnd]Z^dnd^Z_dld_Zdld`Z`daZadbZbdcZcddZddeZedfZfdgZgdjdhZhd>S)oSentencePieceProcessorc4|jSr-r<r=s rr?zSentencePieceProcessor.r@r#c6|j|Sr-r<rBs rr?zSentencePieceProcessor.rDr#rErFcRtj|tjdSr-)rSentencePieceProcessor_swiginitnew_SentencePieceProcessorrKs rrLzSentencePieceProcessor.__init__s$6t^=f=h=hiiiiir#c,tj||Sr-)r.SentencePieceProcessor_LoadFromSerializedProtor serializeds rLoadFromSerializedProtoz.SentencePieceProcessor.LoadFromSerializedProtosLTS]^^^r#c,tj||Sr-)r,SentencePieceProcessor_SetEncodeExtraOptionsr extra_options rSetEncodeExtraOptionsz,SentencePieceProcessor.SetEncodeExtraOptionsJ4Q]^^^r#c,tj||Sr-)r,SentencePieceProcessor_SetDecodeExtraOptionsrs rSetDecodeExtraOptionsz,SentencePieceProcessor.SetDecodeExtraOptionsrr#c,tj||Sr-)r$SentencePieceProcessor_SetVocabulary)r valid_vocabs r SetVocabularyz$SentencePieceProcessor.SetVocabularysB4UUUr#c*tj|Sr-)r&SentencePieceProcessor_ResetVocabularyrKs rResetVocabularyz&SentencePieceProcessor.ResetVocabulary sDTJJJr#c.tj|||Sr-)r%SentencePieceProcessor_LoadVocabulary)rfilename thresholds rLoadVocabularyz%SentencePieceProcessor.LoadVocabularysCD(T]^^^r#c&tj|g|RSr-)r'SentencePieceProcessor_CalculateEntropyrargss rCalculateEntropyz'SentencePieceProcessor.CalculateEntropysEdRTRRRRr#c*tj|Sr-)r#SentencePieceProcessor_GetPieceSizerKs r GetPieceSizez#SentencePieceProcessor.GetPieceSizesA$GGGr#c,tj||Sr-)r SentencePieceProcessor_PieceToIdrres r PieceToIdz SentencePieceProcessor.PieceToIds>tUKKKr#c,tj||Sr-)r SentencePieceProcessor_IdToPiecerrfs r IdToPiecez SentencePieceProcessor.IdToPiece>tRHHHr#c,tj||Sr-)rSentencePieceProcessor_GetScorers rGetScorezSentencePieceProcessor.GetScore=dBGGGr#c,tj||Sr-)r SentencePieceProcessor_IsUnknownrs r IsUnknownz SentencePieceProcessor.IsUnknown rr#c,tj||Sr-)r SentencePieceProcessor_IsControlrs r IsControlz SentencePieceProcessor.IsControl#rr#c,tj||Sr-)rSentencePieceProcessor_IsUnusedrs rIsUnusedzSentencePieceProcessor.IsUnused&rr#c,tj||Sr-)rSentencePieceProcessor_IsByters rIsBytezSentencePieceProcessor.IsByte)s;D"EEEr#c*tj|Sr-)rSentencePieceProcessor_unk_idrKs runk_idzSentencePieceProcessor.unk_id,;DAAAr#c*tj|Sr-)rSentencePieceProcessor_bos_idrKs rbos_idzSentencePieceProcessor.bos_id/r,r#c*tj|Sr-)rSentencePieceProcessor_eos_idrKs reos_idzSentencePieceProcessor.eos_id2r,r#c*tj|Sr-)rSentencePieceProcessor_pad_idrKs rpad_idzSentencePieceProcessor.pad_id5r,r#c*tj|Sr-)r-SentencePieceProcessor_serialized_model_protorKs rserialized_model_protoz-SentencePieceProcessor.serialized_model_proto8sKDQQQr#c,tj||Sr-)r#SentencePieceProcessor_LoadFromFilerargs r LoadFromFilez#SentencePieceProcessor.LoadFromFile;sA$LLLr#c :tj||||||||| Sr-)r#SentencePieceProcessor__EncodeAsIds rrenable_sampling nbest_sizealphaadd_bosadd_eosreverseemit_unk_pieces r _EncodeAsIdsz#SentencePieceProcessor._EncodeAsIds>s>A$o_ikpry|CELN\]] ]r#c :tj||||||||| Sr-)r&SentencePieceProcessor__EncodeAsPiecesr@s r_EncodeAsPiecesz&SentencePieceProcessor._EncodeAsPiecesAs?DT4Q`blnsu|FHOQ_`` `r#c :tj||||||||| Sr-)r/SentencePieceProcessor__EncodeAsSerializedProtor@s r_EncodeAsSerializedProtoz/SentencePieceProcessor._EncodeAsSerializedProtoDsDMdTXZikuw|FHOQXZhii ir#c :tj||||||||| Sr-)r.SentencePieceProcessor__EncodeAsImmutableProtor@s r_EncodeAsImmutableProtoz.SentencePieceProcessor._EncodeAsImmutableProtoGsDLTSWYhjtv{~EGNPWYghh hr#c <tj|||||||||| Sr-)r(SentencePieceProcessor__EncodeAsIdsBatch rins num_threadsrArBrCrDrErFrGs r_EncodeAsIdsBatchz(SentencePieceProcessor._EncodeAsIdsBatchJsJFtSR]_npz}BDKMTV]_mnn nr#c <tj|||||||||| Sr-)r+SentencePieceProcessor__EncodeAsPiecesBatchrTs r_EncodeAsPiecesBatchz+SentencePieceProcessor._EncodeAsPiecesBatchMsKI$PSU`bqs}@EGNPWY`bpqq qr#c <tj|||||||||| Sr-)r4SentencePieceProcessor__EncodeAsSerializedProtoBatchrTs r_EncodeAsSerializedProtoBatchz4SentencePieceProcessor._EncodeAsSerializedProtoBatchPsPRSWY\^ikz}GINPWY`bikyzz zr#c <tj|||||||||| Sr-)r3SentencePieceProcessor__EncodeAsImmutableProtoBatchrTs r_EncodeAsImmutableProtoBatchz3SentencePieceProcessor._EncodeAsImmutableProtoBatchSsPQRVX[]hjy|FHMOVX_ahjxyy yr#c,tj||Sr-)r!SentencePieceProcessor__DecodeIdsridss r _DecodeIdsz!SentencePieceProcessor._DecodeIdsVs?cJJJr#c,tj||Sr-)r(SentencePieceProcessor__DecodeIdsAsBytesrcs r_DecodeIdsAsBytesz(SentencePieceProcessor._DecodeIdsAsBytesYsFtSQQQr#c,tj||Sr-)r$SentencePieceProcessor__DecodePiecesrrs r _DecodePiecesz$SentencePieceProcessor._DecodePieces\sB4PPPr#c,tj||Sr-)r2SentencePieceProcessor__DecodeIdsAsSerializedProtorcs r_DecodeIdsAsSerializedProtoz2SentencePieceProcessor._DecodeIdsAsSerializedProto_sPQUWZ[[[r#c,tj||Sr-)r5SentencePieceProcessor__DecodePiecesAsSerializedProtorks r_DecodePiecesAsSerializedProtoz5SentencePieceProcessor._DecodePiecesAsSerializedProtobsSTXZ`aaar#c,tj||Sr-)r1SentencePieceProcessor__DecodeIdsAsImmutableProtorcs r_DecodeIdsAsImmutableProtoz1SentencePieceProcessor._DecodeIdsAsImmutableProtoesOPTVYZZZr#c,tj||Sr-)r4SentencePieceProcessor__DecodePiecesAsImmutableProtorks r_DecodePiecesAsImmutableProtoz4SentencePieceProcessor._DecodePiecesAsImmutableProtohsRSWY_```r#c.tj|||Sr-)r&SentencePieceProcessor__DecodeIdsBatchrrUrVs r_DecodeIdsBatchz&SentencePieceProcessor._DecodeIdsBatchksDT3P[\\\r#c.tj|||Sr-)r-SentencePieceProcessor__DecodeIdsAsBytesBatchr{s r_DecodeIdsAsBytesBatchz-SentencePieceProcessor._DecodeIdsAsBytesBatchnsKDRUWbcccr#c.tj|||Sr-)r7SentencePieceProcessor__DecodeIdsAsSerializedProtoBatchr{s r _DecodeIdsAsSerializedProtoBatchz7SentencePieceProcessor._DecodeIdsAsSerializedProtoBatchqsUVZ\_almmmr#c.tj|||Sr-)r6SentencePieceProcessor__DecodeIdsAsImmutableProtoBatchr{s r_DecodeIdsAsImmutableProtoBatchz6SentencePieceProcessor._DecodeIdsAsImmutableProtoBatchtsTUY[^`klllr#c.tj|||Sr-)r)SentencePieceProcessor__DecodePiecesBatchr{s r_DecodePiecesBatchz)SentencePieceProcessor._DecodePiecesBatchwsGcS^___r#c.tj|||Sr-)r:SentencePieceProcessor__DecodePiecesAsSerializedProtoBatchr{s r#_DecodePiecesAsSerializedProtoBatchz:SentencePieceProcessor._DecodePiecesAsSerializedProtoBatchzsXY]_bdopppr#c.tj|||Sr-)r9SentencePieceProcessor__DecodePiecesAsImmutableProtoBatchr{s r"_DecodePiecesAsImmutableProtoBatchz9SentencePieceProcessor._DecodePiecesAsImmutableProtoBatch}sWX\^acnooor#c 6tj|||||||Sr-)r(SentencePieceProcessor__NBestEncodeAsIdsrrrBrDrErFrGs r_NBestEncodeAsIdsz(SentencePieceProcessor._NBestEncodeAsIdss1FtTS]_fhoqx{IJJ Jr#c 6tj|||||||Sr-)r+SentencePieceProcessor__NBestEncodeAsPiecesrs r_NBestEncodeAsPiecesz+SentencePieceProcessor._NBestEncodeAsPiecess2I$PTV`bikrt{~LMM Mr#c 6tj|||||||Sr-)r4SentencePieceProcessor__NBestEncodeAsSerializedProtors r_NBestEncodeAsSerializedProtoz4SentencePieceProcessor._NBestEncodeAsSerializedProtos7RSWY]_ikrt{~EGUVV Vr#c 6tj|||||||Sr-)r3SentencePieceProcessor__NBestEncodeAsImmutableProtors r_NBestEncodeAsImmutableProtoz3SentencePieceProcessor._NBestEncodeAsImmutableProtos7QRVX\^hjqsz}DFTUU Ur#c <tj|||||||||| Sr-)r1SentencePieceProcessor__SampleEncodeAndScoreAsIds rr num_samplesrCwor include_bestrDrErFrGs r_SampleEncodeAndScoreAsIdsz1SentencePieceProcessor._SampleEncodeAndScoreAsIdssLOPTVZ\ginpsvBDKMTV]_mnn nr#c <tj|||||||||| Sr-)r4SentencePieceProcessor__SampleEncodeAndScoreAsPiecesrs r_SampleEncodeAndScoreAsPiecesz4SentencePieceProcessor._SampleEncodeAndScoreAsPiecessLRSWY]_jlqsvyEGNPWY`bpqq qr#c <tj|||||||||| Sr-)r=SentencePieceProcessor__SampleEncodeAndScoreAsSerializedProtors r&_SampleEncodeAndScoreAsSerializedProtoz=SentencePieceProcessor._SampleEncodeAndScoreAsSerializedProtosL[\`bfhsuz|BNPWY`bikyzz zr#c <tj|||||||||| Sr-)r to the result (Default = false) add_eos: Add to the result (Default = false) / is added after reversing (if enabled). reverse: Reverses the tokenized sequence (Default = false) emit_unk_piece: Emits the unk literal string (Default = false) nbest_size: sampling parameters for unigram. Invalid in BPE-Dropout. nbest_size = {0,1}: No sampling is performed. nbest_size > 1: samples from the nbest_size results. nbest_size < 0: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. alpha: Soothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. num_threads: number of threads in batch processing (Default = -1, auto-detected) ) model_file model_protoN) $_sentencepiece_processor_init_native _out_type_add_bos_add_eos_reverse_emit_unk_piece_enable_sampling _nbest_size_alpha _num_threadsLoad) rrrout_typerDrErFrGrArBrCrVs rInitzSentencePieceProcessor.InitsD+4000dndmdmdm+d-d#ddk%d B{B Z[ AAAAABBr#c ||j}||j}||j}||j}||j}||j}||j}| |j} | |j} |dkr||dks|dks| td| t| turtdt|tur|tur| || ||| |||| S|tur||| ||| |||| S|dks|dkr||| ||| |||| S|d kr||| ||| |||| S|tur||||| ||||S|tur||||| ||||S|dks|dkr||||| ||||S|d kr||||| ||||Std |) a~Encode text input to segmented ids or tokens. Args: input: input string. accepsts list of string. out_type: output type. int or str. add_bos: Add to the result (Default = false) add_eos: Add to the result (Default = false) / is added after reversing (if enabled). reverse: Reverses the tokenized sequence (Default = false) emit_unk_piece: Emits the unk literal string (Default = false) nbest_size: sampling parameters for unigram. Invalid in BPE-Dropout. nbest_size = {0,1}: No sampling is performed. nbest_size > 1: samples from the nbest_size results. nbest_size < 0: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. alpha: Soothing parameter for unigram sampling, and merge probability for BPE-dropout (probablity 'p' in BPE-dropout paper). num_threads: the number of threads used in the batch processing (Default = -1). NTrraWhen enable_sampling is True, We must specify "nbest_size > 1" or "nbest_size = -1", and "alpha". "nbest_size" is enabled only on unigram mode ignored in BPE-dropout. when "nbest_size = -1" , this method samples from all candidates on the lattice instead of nbest segmentations.num_threads must be intserialized_protorimmutable_protozunknown out_type={})rrrrrrrrr RuntimeErrorrintlistrWrqrZr]r`rHrKrNrQrd) rinputrrDrErFrGrArBrCrVs rEncodezSentencePieceProcessor.Encodes@  > - - -  -  /  %    ' D j&8J!OO&0Aoo .     [ 1 1 < <4555 e   s??''{OZ(-w.ZZ Z s??**5+PZ+0'7G^]] ] ) ) )X-@-@33E;Yc497GWVdff f ( ( (225+Xb38'7GUcee e S  !&'>SS S S##E?J$)7GWnVV V ' ' '8w+>+>,,UOZ-2GWg~__ _ & & &++E?J,17GWn^^ ^ .55h?? @ @@r#c ,|jd|td|SNrrr$rrqrrkwargss rEncodeAsPiecesz%SentencePieceProcessor.EncodeAsPieces5! T[ =us = =f = ==r#c ,|jd|td|Srrrrs r EncodeAsIdsz"SentencePieceProcessor.EncodeAsIds9rr#c "|jd|dd|S)Nrrr$rrs rEncodeAsSerializedProtoz.SentencePieceProcessor.EncodeAsSerializedProto=s" T[ Lu/A L LV L LLr#c "|jd|dd|S)Nrrr$rrs rEncodeAsImmutableProtoz-SentencePieceProcessor.EncodeAsImmutableProtoAs" T[ Ku/@ K KF K KKr#c 2|jd|||tdd|SNTrrBrCrrAr$rrrrBrCrs rSampleEncodeAsPiecesz+SentencePieceProcessor.SampleEncodeAsPiecesE; T[Gu5"%tGG?EGGGr#c 2|jd|||tdd|Srrrs rSampleEncodeAsIdsz(SentencePieceProcessor.SampleEncodeAsIdsJrr#c (|jd|||ddd|S)NrTrr$rrs rSampleEncodeAsSerializedProtoz4SentencePieceProcessor.SampleEncodeAsSerializedProtoOs; T[Vu5"4dVVNTVVVr#c (|jd|||ddd|S)NrTrr$rrs rSampleEncodeAsImmutableProtoz3SentencePieceProcessor.SampleEncodeAsImmutableProtoTs; T[Uu5"3TUUMSUUUr#cjjjjjjdkrdfdt |turfd|DS|S)aNBestEncode text input to segmented ids or tokens. Args: input: input string. accepsts list of string. out_type: output type. int or str. add_bos: Add to the result (Default = false) add_eos: Add to the result (Default = false) / is added after reversing (if enabled). reverse: Reverses the tokenized sequence (Default = false) emit_unk_piece: Emits the unk literal string (Default = false) nbest_size: nbest size Nrrc:tur|Stur|Sdksdkr|Sdkr|St d)Nrrrzunknown out_type)rrrqrrrr)rrDrErGrBrrFrs r_encodez3SentencePieceProcessor.NBestEncode.._encode}s s??''j(/'>SS S s??**4+2GWnVV V ) ) )X-@-@33D*4;Wg~__ _ ( ( (2243:GWn^^ ^-...r#c&g|] }|Sr$r$rnrs rrz6SentencePieceProcessor.NBestEncode..!***q ***r#)rrrrrrrr) rrrrDrErFrGrBrs ` ``````@r NBestEncodez"SentencePieceProcessor.NBestEncodeYs(  > - - -  -  % q /////////// e  ****E**** WU^^r#c .|jd||td|SNrrBrr$)rrqrrrBrs rNBestEncodeAsPiecesz*SentencePieceProcessor.NBestEncodeAsPieces2 T 6Ej'*66.4666r#c .|jd||td|Sr)rrrs rNBestEncodeAsIdsz'SentencePieceProcessor.NBestEncodeAsIdsrr#c $|jd||dd|S)Nrrr$rrs rNBestEncodeAsSerializedProtoz3SentencePieceProcessor.NBestEncodeAsSerializedProtos8 T EEj'9EE=CEEEr#c $|jd||dd|S)Nrrr$rrs rNBestEncodeAsImmutableProtoz2SentencePieceProcessor.NBestEncodeAsImmutableProtos8 T DEj'8DD to the result (Default = false) add_eos: Add to the result (Default = false) / is added after reversing (if enabled). reverse: Reverses the tokenized sequence (Default = false) emit_unk_piece: Emits the unk literal string (Default = false) num_samples: How many samples to return (Default = 1) alpha: inverse temperature for sampling wor: whether to sample without replacement (Default = false) include_best: whether to include the best tokenization, requires wor=True (Default = false) Nrg?Frznum_examples must be positivez8When include_best is True, We must specify "wor = True".c R tur |  Stur |  Sdksdkr |  Sdkr |  St d)Nrrrzunknown output type)rrrqrrrr) rrDrErCrGrrrrFrrs rrz._encodes  s??00{E3P\18'7N\\ \ s??33D+ucS_4;Wg~__ _ ) ) )X-@-@<.rr#)rrrrrrrr) rrrrDrErFrGrrCrrrs ` `````````@rSampleEncodeAndScorez+SentencePieceProcessor.SampleEncodeAndScores34  > - - -  -         :;;; WcWUVVV22222222222222& e  ****E**** WU^^r#c 0|jd|||td|SNrrrCrr$)r rqrrrrCrs rSampleEncodeAndScoreAsPiecesz3SentencePieceProcessor.SampleEncodeAndScoreAsPieces5 &T &?U SX03??7=???r#c 0|jd|||td|Sr )r rrs rSampleEncodeAndScoreAsIdsz0SentencePieceProcessor.SampleEncodeAndScoreAsIdsrr#c &|jd|||dd|S)Nrr r$r rs r%SampleEncodeAndScoreAsSerializedProtoz">677 7**5%EEE  # #E5 1 11r#crfdt|turfd|DS|S)Nc\r|S|Sr-rrrr with_offsetss r _normalizez4SentencePieceProcessor.Normalize.._normalize1  2++D11 1t$$$r#c&g|] }|Sr$r$rr>r0s rrz4SentencePieceProcessor.Normalize..!---! 1 ---r#rrrrr/r0s` `@r Normalizez SentencePieceProcessor.Normalize`%%%%%% e  ----u---- Z  r#c i}|D]\}}t|||<||Sr-)itemsrqr)rr new_kwargskeyr s rOverrideNormalizerSpecz-SentencePieceProcessor.OverrideNormalizerSpecsGj %%*#ue** 3  ) )* 5 55r#c*|Sr-rrKs r piece_sizez!SentencePieceProcessor.piece_size     r#c*|Sr-r?rKs r vocab_sizez!SentencePieceProcessor.vocab_sizerAr#c*|Sr-r8rKs r __getstate__z#SentencePieceProcessor.__getstate__  ( ( * **r#cX|||dSr-rLrrr8s r __setstate__z#SentencePieceProcessor.__setstate__* mmooo ""#9:::::r#c*|Sr-r?rKs rrzSentencePieceProcessor.__len__rAr#c,||Sr-)rrs rrz"SentencePieceProcessor.__getitem__s ^^E " ""r#c|r|rtd|r||S||S)aOverwride SentencePieceProcessor.Load to support both model_file and model_proto. Args: model_file: The sentencepiece model file path. model_proto: The sentencepiece model serialized proto. Either `model_file` or `model_proto` must be set. z-model_file and model_proto must be exclusive.)rrr=)rrrs rrzSentencePieceProcessor.LoadsT L LJKKK 9++K888   z * **r#) NNNNNNNNNNN)NNNNNNr-)r)r)irrr6rrrr rLrdelete_SentencePieceProcessorrtrrrrrrr rrrrrr"r%r(r+r/r2r5r8r=rHrKrNrQrWrZr]r`rerhrlrorrrurxr|rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr rrrrrqrrr!r#r%r'r)r7r=r@rCrFrKrrrr$r#rrrsh--/I/IOdeeeGHjjj%C_________VVVKKK___SSSHHHLLLIIIHHHIIIIIIHHHFFFBBBBBBBBBBBBRRRMMM]]]```iiihhhnnnqqqzzzyyyKKKRRRQQQ\\\bbb[[[aaa]]]dddnnnmmm```qqqpppJJJMMMVVVUUUnnnqqqzzzyyyLLLWWWZZZkkkYYY!"-B-B-B-Bd"#[[[[|>>>>>>MMMLLLGGGG GGGG VVVV UUUU"   #'#7777t6666 6666 EEEE DDDD'+%)%)%),0)-#'!%*.JJJJZ???? ???? NNNN MMMM &)d\\\\~,/CCCC),CCCCCCCCCCCCCCCCCCCC 2 2 2 2666!!!!!!+++;;; !!!### + + + + + +r#rc*tj|Sr-)rSetRandomGeneratorSeed)seeds rrSrSs  0 6 66r#c*tj|Sr-)rSetMinLogLevel)rCs rrVrVs  ( + ++r#ceZdZedddZdZeZedZ edZ edZ ed Z ed Z edd Zedd Zd S)SentencePieceTrainerc4|jSr-r<r=s rr?zSentencePieceTrainer.r@r#c6|j|Sr-r<rBs rr?zSentencePieceTrainer.rDr#rErFc td)NzNo constructor defined)r)rr rs rrLzSentencePieceTrainer.__init__s5666r#c*tj|Sr-)r%SentencePieceTrainer__TrainFromString)r<s r_TrainFromStringz%SentencePieceTrainer._TrainFromStringsCCHHHr#c*tj|Sr-)r"SentencePieceTrainer__TrainFromMapr s r _TrainFromMapz"SentencePieceTrainer._TrainFromMaps@FFFr#c,tj||Sr-)r#SentencePieceTrainer__TrainFromMap2r iters r_TrainFromMap2z#SentencePieceTrainer._TrainFromMap2A$MMMr#c*tj|Sr-)r#SentencePieceTrainer__TrainFromMap3ras r_TrainFromMap3z#SentencePieceTrainer._TrainFromMap3sA$GGGr#c,tj||Sr-)r#SentencePieceTrainer__TrainFromMap4res r_TrainFromMap4z#SentencePieceTrainer._TrainFromMap4rhr#Nc |0t|turt|Sd}d}d}i}|D]!\}}|dvr|} |dvr|}||||<"|rN|rt||}nt|}||n7|rt||St |SdS)zDTrain Sentencepiece model. Accept both kwargs and legacy string arg.NcDt|tur|tjddkrt }nt }t j|d}|d|D| St|S)zEncode value to CSV..rr)lineterminatorc,g|]}t|Sr$)rq)rrCs rrz@SentencePieceTrainer._Train.._encode..s111a3q66111r#) rrsysrStringIOBytesIOcsvwriterwriterowgetvaluerq)r frxs rrz,SentencePieceTrainer._Train.._encodes ;;$    a A % % AA A:a333& //115111 2 2 2 U r#)sentence_iteratorsentence_reader) model_writer) rrqrXr^r:rnrkwritergrb) r<rrr|r~r;r<r rs r_TrainzSentencePieceTrainer._Trains9 T#YY#--#44S999   lj ++*#u : : :#   $ $ $,,#GENN*S//  @  H,;;J;LNN++-;;JGG+;''''  @%44ZARSS S%33J?? ? Tr#c ~t|5tjdd|i|ddddS#1swxYwYdS)N)ostreamr<r$) _LogStreamrXr)r< logstreamrs rTrainzSentencePieceTrainer.Trains i ( ( (77#666v666777777777777777777s 266r-rP)rrr6rrrLrr staticmethodr^rbrgrkrnrrr$r#rrXrXsh--/I/IOdeeeG777HII\IGG\GNN\NHH\HNN\N***\*X777\777r#rXceZdZedddZeZdZej Z dZ dZ dZ d Zd Zd Zd Zd Z ddZddZdZdZdS)SentencePieceNormalizerc4|jSr-r<r=s rr?z SentencePieceNormalizer.r@r#c6|j|Sr-r<rBs rr?z SentencePieceNormalizer.rDr#rErFcRtj|tjdSr-)r SentencePieceNormalizer_swiginitnew_SentencePieceNormalizerrKs rrLz SentencePieceNormalizer.__init__ s$7n>h>j>jkkkkkr#c,tj||Sr-)r/SentencePieceNormalizer_LoadFromSerializedProtors rrz/SentencePieceNormalizer.LoadFromSerializedProto$sMdT^___r#c,tj||Sr-)r'SentencePieceNormalizer_LoadFromRuleTSV)rrs rLoadFromRuleTSVz'SentencePieceNormalizer.LoadFromRuleTSV'sEdHUUUr#c,tj||Sr-)r(SentencePieceNormalizer_LoadFromRuleName)rrs rLoadFromRuleNamez(SentencePieceNormalizer.LoadFromRuleName*sFtTRRRr#c*tj|Sr-)r.SentencePieceNormalizer_serialized_model_protorKs rr8z.SentencePieceNormalizer.serialized_model_proto-sLTRRRr#c,tj||Sr-)r$SentencePieceNormalizer_LoadFromFiler;s rr=z$SentencePieceNormalizer.LoadFromFile0sB4MMMr#c,tj||Sr-)r"SentencePieceNormalizer__Normalizers rrz"SentencePieceNormalizer._Normalize3s@tLLLr#c,tj||Sr-)r-SentencePieceNormalizer__NormalizeWithOffsetsrs rrz-SentencePieceNormalizer._NormalizeWithOffsets6sKDRVWWWr#c.tj|||Sr-)r&SentencePieceNormalizer__SetProtoField)rrr s r_SetProtoFieldz&SentencePieceNormalizer._SetProtoField9sDT4QVWWWr#NFct||r||}nW|r||}n?|r||}n'|r||}nt d|rD|d||d||d|dSdS)aInitialzie sentencePieceNormalizer. Args: model_file: The sentencepiece model file path. model_proto: The sentencepiece model serialized proto. rule_tsv: The normalization rule file in TSV format. rule_name: Pre-defined normalization name. add_dummy_prefix: add dummy prefix. escape_whitespaces: escape whitespaces. remove_extra_whitespaces: remove extra whitespaces. zno model is specifiedadd_dummy_prefixescape_whitespacesremove_extra_whitespacesN)%_sentencepiece_normalizer_init_nativer=rrrrr) rrrrule_tsv rule_namerrrstatuss rrzSentencePieceNormalizer.Init<s(,D111  4"":.. 4--k:: 4%%h// 4&&y112333 R .0@AAA 02DEEE 68PQQQQQRRr#crfdt|turfd|DS|S)Nc\r|S|Sr-r-r.s rr0z5SentencePieceNormalizer.Normalize.._normalizecr1r#c&g|] }|Sr$r$r3s rrz5SentencePieceNormalizer.Normalize..ir4r#r5r6s` `@rr7z!SentencePieceNormalizer.Normalizebr8r#c*|Sr-rErKs rrFz$SentencePieceNormalizer.__getstate__mrGr#cX|||dSr-rIrJs rrKz$SentencePieceNormalizer.__setstate__qrLr#)NNNNFFFr-)rrr6rrrr rLrdelete_SentencePieceNormalizerrtrrrr8r=rrrrr7rFrKr$r#rrrsEh--/I/IOdeeeGHlll%D```VVVSSSSSSNNNMMMXXXXXX# %&+$R$R$R$RL+++;;;;;r#rc*tj|Sr-)r SetDataDir)data_dirs rrrys  $X . ..r#)ru)rvcJi}|jD][\}}tjd|rAtjdd|dd}|||<\|D]\}}t|||dS)z1Added snake_cased method from CammelCased method.z^[A-Z]+z(?._funcsK Aww#~~1q55A$7$7 2 3 33 41::r#cht|turfd|DS|S)Nc(g|]}|Sr$r$)rrrrs rrz5_batchnize.._batched_func..s# * * *eeD!nn * * *r#r5)rr<rs` r _batched_funcz!_batchnize.._batched_funcsD CyyD * * * * *c * * ** U4  r#)rr)rrrrrs @@r _batchnizersa D$ ' '$   )T=)))))r#rL)rrrrr"r%r() __version__ sentencepiece package_datac"eZdZddZdZdZdS)rNcl||_|j%tj|_dSdSr-)rrtstderrfilenoorig_stream_fileno)rrs rrLz_LogStream.__init__s5DL | # 1 1 3 3d r#c|jQtj|j|_tj|j|jdSdSr-)rosduprorig_stream_dupdup2rrKs r __enter__z_LogStream.__enter__sP |VD$;<>> r#r)>rtr_swig_python_version_info __package__rrrbuiltinsr ImportErrorrr%r*r3rr5objectr:>ImmutableSentencePieceText_ImmutableSentencePiece_swigregisterrx'ImmutableSentencePieceText_swigregisterr,ImmutableNBestSentencePieceText_swigregisterr#SentencePieceProcessor_swigregisterrSrVrX!SentencePieceTrainer_swigregisterr$SentencePieceNormalizer_swigregisterrrrwrimportlib.resources importlibiorurvrrrLrrrrrTokenizer Detokenizemset_random_generator_seedset_min_log_level_versionrpathrrq resourcesfilesrr$r#rrs:99999#//       """""[[[   LLLLL$LLL 33333333nNMNAAADDDDDDDDP767QRRR44444f444p<;<[\\\J +J +J +J +J +VJ +J +J +\323IJJJ777,,,K7K7K7K7K76K7K7K7^101EFFFW;W;W;W;W;fW;W;W;v434KLLL///    ***"(>'F$(?(H% ,B,GHHH-D-IJJJ"8"?$:$A! ((A * #Q''''&'''$%%%'(((2"!!!!!! 27<<I/55oFFGG X XYYYs  **