ޘhxRddlZddlZddlmZmZmZmZddlZddl m Z ddl m Z m Z mZmZddlmZmZmZmZer ddlZddlmZer ddlZdd lmZGd d eZGd d e Ze eddGddeZeZdS)N)AnyOptionalUnionoverload)BasicTokenizer) ExplicitEnumadd_end_docstringsis_tf_availableis_torch_available)ArgumentHandler ChunkPipelineDatasetbuild_pipeline_init_args)/TF_MODEL_FOR_TOKEN_CLASSIFICATION_MAPPING_NAMES),MODEL_FOR_TOKEN_CLASSIFICATION_MAPPING_NAMESc:eZdZdZdeeeeffdZdS)"TokenClassificationArgumentHandlerz5 Handles arguments for token classification. inputsc |dd}|d}|Nt|ttfr2t |dkrt|}t |}nft|t r|g}d}nKt t|t st|tjr||d|fStd|d}|rUt|trt|dtr|g}t ||krtd||||fS) Nis_split_into_wordsF delimiterrr zAt least one input is required.offset_mappingz;offset_mapping should have the same batch size as the input) get isinstancelisttuplelenstrrtypes GeneratorType ValueError)selfrkwargsrr batch_sizers s/var/www/html/services/rag-service/venv/lib/python3.11/site-packages/transformers/pipelines/token_classification.py__call__z+TokenClassificationArgumentHandler.__call__ sN$jj)>FFJJ{++  *VdE]"C"C F VW&\\FVJJ  $ $ @XFJJ  Z%@%@ JvW\WjDkDk .i? ?>?? ?$455  `.$// 2J~a?PRW4X4X 2"0!1>""j00 !^___*NIEEN)__name__ __module__ __qualname____doc__rr rr(r)r'rrsMFuS$s)^4FFFFFFr)rc&eZdZdZdZdZdZdZdZdS)AggregationStrategyzDAll the valid aggregation strategies for TokenClassificationPipelinenonesimplefirstaveragemaxN) r*r+r,r-NONESIMPLEFIRSTAVERAGEMAXr.r)r'r0r08s-NN D F EG CCCr)r0T) has_tokenizera ignore_labels (`list[str]`, defaults to `["O"]`): A list of labels to ignore. grouped_entities (`bool`, *optional*, defaults to `False`): DEPRECATED, use `aggregation_strategy` instead. Whether or not to group the tokens corresponding to the same entity together in the predictions or not. stride (`int`, *optional*): If stride is provided, the pipeline is applied on all the text. The text is split into chunks of size model_max_length. Works only with fast tokenizers and `aggregation_strategy` different from `NONE`. The value of this argument defines the number of overlapping tokens between chunks. In other words, the model will shift forward by `tokenizer.model_max_length - stride` tokens each step. aggregation_strategy (`str`, *optional*, defaults to `"none"`): The strategy to fuse (or not) tokens based on the model prediction. - "none" : Will simply not do any aggregation and simply return raw results from the model - "simple" : Will attempt to group entities following the default schema. (A, B-TAG), (B, I-TAG), (C, I-TAG), (D, B-TAG2) (E, B-TAG2) will end up being [{"word": ABC, "entity": "TAG"}, {"word": "D", "entity": "TAG2"}, {"word": "E", "entity": "TAG2"}] Notice that two consecutive B tags will end up as different entities. On word based languages, we might end up splitting words undesirably : Imagine Microsoft being tagged as [{"word": "Micro", "entity": "ENTERPRISE"}, {"word": "soft", "entity": "NAME"}]. Look for FIRST, MAX, AVERAGE for ways to mitigate that and disambiguate words (on languages that support that meaning, which is basically tokens separated by a space). These mitigations will only work on real words, "New york" might still be tagged with two different entities. - "first" : (works only on word based models) Will use the `SIMPLE` strategy except that words, cannot end up with different tags. Words will simply use the tag of the first token of the word when there is ambiguity. - "average" : (works only on word based models) Will use the `SIMPLE` strategy except that words, cannot end up with different tags. scores will be averaged first across tokens, and then the maximum label is applied. - "max" : (works only on word based models) Will use the `SIMPLE` strategy except that words, cannot end up with different tags. Word entity will simply be the token with the maximum score.ceZdZdZdZdZdZdZdZe ffd Z d)de e de e d e e d e eeeefd e e d e ed e efdZedededeeeeffdZedeededeeeeeffdZdeeeefdedeeeeefeeeeeffffd Zd*dZdZe jdfdZdZ d+dedejdejd e eeeefdejd e de ee ede eeeefdeefdZd eed e deefd!Z d"eed e defd#Z!d"eed e deefd$Z"d"eedefd%Z#d&edeeeffd'Z$d"eedeefd(Z%xZ&S),TokenClassificationPipelineuv Named Entity Recognition pipeline using any `ModelForTokenClassification`. See the [named entity recognition examples](../task_summary#named-entity-recognition) for more information. Example: ```python >>> from transformers import pipeline >>> token_classifier = pipeline(model="Jean-Baptiste/camembert-ner", aggregation_strategy="simple") >>> sentence = "Je m'appelle jean-baptiste et je vis à montréal" >>> tokens = token_classifier(sentence) >>> tokens [{'entity_group': 'PER', 'score': 0.9931, 'word': 'jean-baptiste', 'start': 12, 'end': 26}, {'entity_group': 'LOC', 'score': 0.998, 'word': 'montréal', 'start': 38, 'end': 47}] >>> token = tokens[0] >>> # Start and end provide an easy way to highlight words in the original text. >>> sentence[token["start"] : token["end"]] ' jean-baptiste' >>> # Some models use the same idea to do part of speech. >>> syntaxer = pipeline(model="vblagoje/bert-english-uncased-finetuned-pos", aggregation_strategy="simple") >>> syntaxer("My name is Sarah and I live in London") [{'entity_group': 'PRON', 'score': 0.999, 'word': 'my', 'start': 0, 'end': 2}, {'entity_group': 'NOUN', 'score': 0.997, 'word': 'name', 'start': 3, 'end': 7}, {'entity_group': 'AUX', 'score': 0.994, 'word': 'is', 'start': 8, 'end': 10}, {'entity_group': 'PROPN', 'score': 0.999, 'word': 'sarah', 'start': 11, 'end': 16}, {'entity_group': 'CCONJ', 'score': 0.999, 'word': 'and', 'start': 17, 'end': 20}, {'entity_group': 'PRON', 'score': 0.999, 'word': 'i', 'start': 21, 'end': 22}, {'entity_group': 'VERB', 'score': 0.998, 'word': 'live', 'start': 23, 'end': 27}, {'entity_group': 'ADP', 'score': 0.999, 'word': 'in', 'start': 28, 'end': 30}, {'entity_group': 'PROPN', 'score': 0.999, 'word': 'london', 'start': 31, 'end': 37}] ``` Learn more about the basics of using a pipeline in the [pipeline tutorial](../pipeline_tutorial) This token recognition pipeline can currently be loaded from [`pipeline`] using the following task identifier: `"ner"` (for predicting the classes of tokens in a sequence: person, organisation, location or miscellaneous). The models that this pipeline can use are models that have been fine-tuned on a token classification task. See the up-to-date list of available models on [huggingface.co/models](https://huggingface.co/models?filter=token-classification). sequencesFTctj|i|||jdkrtnt t d|_||_dS)NtfF) do_lower_case) super__init__check_model_type frameworkrrr_basic_tokenizer _args_parser)r$ args_parserargsr% __class__s r'rCz$TokenClassificationPipeline.__init__sr$)&))) ~%% < ;=   !/U C C C'r)Ngrouped_entitiesignore_subwordsaggregation_strategyrrstriderc i} || d<|r |dn|| d<||| d<i} ||b|r|r tj}n|r|s tj}n tj}|t jd|d|t jd|d|yt |trt|}|tjtj tj hvr|j j std|| d <||| d <|i||j jkrtd |tjkrtd |d |j j r dd|d} | | d<ntd| i| fS)Nr rrzl`grouped_entities` is deprecated and will be removed in version v5.0.0, defaulted to `aggregation_strategy="z "` instead.zk`ignore_subwords` is deprecated and will be removed in version v5.0.0, defaulted to `aggregation_strategy="z{Slow tokenizers cannot handle subwords. Please set the `aggregation_strategy` option to `"simple"` or use a fast tokenizer.rM ignore_labelszl`stride` must be less than `tokenizer.model_max_length` (or even lower if the tokenizer adds special tokens)zI`stride` was provided to process all the text but `aggregation_strategy="z&"`, please select another one instead.T)return_overflowing_tokenspaddingrNtokenizer_paramszm`stride` was provided to process all the text but you're using a slow tokenizer. Please use a fast tokenizer.)r0r8r7r6warningswarnrr upperr:r9 tokenizeris_fastr#model_max_length) r$rQrKrLrMrrrNrpreprocess_paramspostprocess_paramsrTs r'_sanitize_parametersz0TokenClassificationPipeline._sanitize_parameterssn3F/0  U4=4ESS9 k *  %2@ . /  '?+F @O @':'@$$! @/ @':'A$$':'?$+ Q/CQQQ* Q/CQQQ +.44 Y':;O;U;U;W;W'X$$'-/B/FH[Hcdee.e!>:N 5 6  $2?  /  888 C$':'??? U,UUU >) 59#'"((($ =M%&899$8!"&888r)rr%returnc dSNr.r$rr%s r'r(z$TokenClassificationPipeline.__call__sLOCr)c dSr`r.ras r'r(z$TokenClassificationPipeline.__call__sX[X[r)c |j|fi|\}}}}||d<||d<|r4td|Dstj|gfi|S|r||d<tj|fi|S)a Classify each token of the text(s) given as inputs. Args: inputs (`str` or `List[str]`): One or several texts (or one list of texts) for token classification. Can be pre-tokenized when `is_split_into_words=True`. Return: A list or a list of list of `dict`: Each result comes as a list of dictionaries (one for each token in the corresponding input, or each entity if this pipeline was instantiated with an aggregation_strategy) with the following keys: - **word** (`str`) -- The token/word classified. This is obtained by decoding the selected tokens. If you want to have the exact string in the original sentence, use `start` and `end`. - **score** (`float`) -- The corresponding probability for `entity`. - **entity** (`str`) -- The entity predicted for that token/word (it is named *entity_group* when *aggregation_strategy* is not `"none"`. - **index** (`int`, only present when `aggregation_strategy="none"`) -- The index of the corresponding token in the sentence. - **start** (`int`, *optional*) -- The index of the start of the corresponding entity in the sentence. Only exists if the offsets are available within the tokenizer - **end** (`int`, *optional*) -- The index of the end of the corresponding entity in the sentence. Only exists if the offsets are available within the tokenizer rrc3@K|]}t|tVdSr`)rr).0inputs r' z7TokenClassificationPipeline.__call__..s,*W*Wu:eT+B+B*W*W*W*W*W*Wr)r)rGallrBr()r$rr%_inputsrrrrJs r'r(z$TokenClassificationPipeline.__call__s:CT$BSTZBeBe^dBeBe?$ni(;$%'{  8s*W*WPV*W*W*W'W'W 8#577#VH7777 7  6'5F# $uww11&111r)c+(K|di}|jjo|jjdk}d}|d}|r|d}t|tst d|} || }g}t|} d} | D]>} || | t| zf| t| | zz } ?| } d|d<n&t|tst d|} |j| f|j |d|jj d|}|r|jj st d |d dt|d }t|D]|j d kr!fd | D}n fd| D}|||d<dkr|nd|d<|dz k|d<|||d<||d<|VdS)NrTrrrzEWhen `is_split_into_words=True`, `sentence` must be a list of tokens.TzKWhen `is_split_into_words=False`, `sentence` must be an untokenized string.)return_tensors truncationreturn_special_tokens_maskreturn_offsets_mappingz@is_split_into_words=True is only supported with fast tokenizers.overflow_to_sample_mapping input_idsr@cNi|]!\}}|tj|d"Sr)r@ expand_dimsrekvis r' z:TokenClassificationPipeline.preprocess..Fs/VVVtq!2>!A$#:#:VVVr)cNi|]!\}}||d"Srr) unsqueezerts r'rxz:TokenClassificationPipeline.preprocess..Hs/PPPA1Q4>>!#4#4PPPr)rsentencer is_lastword_idsword_to_chars_map)poprXrZrrr#joinrappendr rErYrangeitemsr})r$r{rr[rTrlr~rrwords delimiter_len char_offsetwordtext_to_tokenizer num_chunks model_inputsrws @r' preprocessz&TokenClassificationPipeline.preprocesss,001CRHH^4\9X[\9\  /0EF  ()+6Ih-- j !hiiiE ~~e,,H "  NNMK 9 9!((+{SYY7N)OPPPs4yy=88  % 6: 2 3 3h,, p !nooo'   >!'+#'>#9        at~'= a_`` ` /666 ,-- z""  A~%%VVVVv||~~VVV PPPPPPP )1? -.3466xxtL $&':>&9L # ,+1??1+=+= Z(4E 01      r)c|d}|dd}|d}|d}|dd}|dd}|jdkr|jd i|d}n2|jd i|} t| tr| d n| d}|||||||d |S) Nspecial_tokens_maskrr{r|r}r~r@rlogits)rrrr{r|r}r~r.)rrEmodelrdict) r$rrrr{r|r}r~routputs r'_forwardz$TokenClassificationPipeline._forwardTs*../DEE%))*:DAA##J//""9--##J55(,,-@$GG >T ! !TZ//,//2FFTZ//,//F)3FD)A)APVH%%vayF#6,  !2     r)c Rdgg}|dd}|D]}|jdkri|ddjtjtjfvr>|ddtj}n |dd}|dd}|dd} |d|ddnd} |d d} |d } tj |d d } tj || z }|| d d z }|jdkr,| } | | nd} | || || | || |}|||}fd|D}||t!|}|dkr||}|S)NOrr~ptrr{rprrr}T)axiskeepdimsr@)r}r~ctg|]4}|ddvr|ddv2|5S)entityN entity_group)r)rerrQs r' z;TokenClassificationPipeline.postprocess..sX::h--]BBJJ~t44MIIJIIr)r )rrEdtypetorchbfloat16float16tofloat32numpynpr5expsumgather_pre_entities aggregateextendraggregate_overlapping_entities)r$ all_outputsrMrQ all_entitiesr~ model_outputsrr{rprrr}maxes shifted_expscores pre_entitiesrKentitiesrs ` r' postprocessz'TokenClassificationPipeline.postprocessns]   EM (N../BCC(( *( *M~%%-*A!*D*Ju~_d_lNm*m*m&x0366u}EEKKMM&x0399;;"1~j1H%k215I6CDT6U6a ./22gk #00E"Fq"I"O"O"Q"Q $((44HF6T:::E&%00K ;??T?#J#JJF~%%%OO-- ;I;U!5!5!7!7!7[_33#$!"34  L $~~l>>>|LLLr)ct|dkr|St|d}g}|d}|D]~}|d|dcxkr |dkrFnnC|d|dz }|d|dz }||ks||kr|d|dkr|}g|||}|||S)Nrc|dS)Nstartr.)xs r'zLTokenClassificationPipeline.aggregate_overlapping_entities..s !G*r)keyrendscore)rsortedr)r$raggregated_entitiesprevious_entityrcurrent_lengthprevious_lengths r'rz:TokenClassificationPipeline.aggregate_overlapping_entitiess x==A  O((<(<=== "1+ ) )Fw'6'?SSSS_U=SSSSSS!'!@"1%"8?7;S"S"_44%88w/'*BBB&,O#**?;;;"(""?333""r)r{rprrr}r~c g} t|D]\} } || r|jt|| } |V|| \} }|!||| }|||\}}| |z } ||z }t | ts3|jdkr(| } |}|| |}t|jddrAt|jjj ddr!t| t|k}nW|tj tj tjhvrtjdt"| dkod|| dz | dzv}t|| |jjkr|} d }nd} d}d }| | | || |d }| || S) zTFuse various numpy arrays into dicts with all the information needed for aggregationNr _tokenizercontinuing_subword_prefixz?Tokenizer does not support real words, using fallback heuristicrrPr F)rrrrindex is_subword) enumeraterXconvert_ids_to_tokensintrrEitemgetattrrrrr0r8r9r:rUrV UserWarning unk_token_idr)r$r{rprrrrMr}r~ridx token_scoresr start_indend_ind word_index start_char_word_refr pre_entitys r'rz/TokenClassificationPipeline.gather_pre_entitiess, !*6!2!29 ,9 , C"3' >77IcN8K8KLLD)%3C%8" 7',=,I!)#J!-(9*(E A!Z/ :-!)S111~--$-NN$4$4 "),,..#Ig$564><>>f7N-35PRVDDf "%Tc(mm!;JJ,+1+3+/0 ! ]'"+Q!e3hyST}W`cdWdGd>e3eJy~&&$.*EEE#D!&J " &"( J    + + + +r)rc|tjtjhvr{g}|D]u}|d}|d|}|jjj|||d|d|d|dd}||vn|||}|tjkr|S| |S)Nrrrrr)rrrrrr) r0r6r7argmaxrconfigid2labelraggregate_wordsgroup_entities)r$rrMrr entity_idxrrs r'rz%TokenClassificationPipeline.aggregates $7$<>Q>X#Y Y YH* ( ( '188:: "8,Z8"j/8D"'0&v.'0%e, '''' (++L:NOOH #6#; ; ;O""8,,,r)rc|jd|D}|tjkrB|dd}|}||}|jjj|}n|tjkrNt|d}|d}|}||}|jjj|}n|tj krhtj d|D}tj |d} | } |jjj| }| | }ntd||||dd |d d d } | S) Ncg|] }|d Srr.rers r'rz>TokenClassificationPipeline.aggregate_word..s7^7^7^6v7^7^7^r)rrc6|dS)Nr)r5)rs r'rz.!s&:J:N:N:P:Pr)rcg|] }|d S)rr.rs r'rz>TokenClassificationPipeline.aggregate_word..'sGGGFvh/GGGr))rzInvalid aggregation_strategyrrr)rrrrr)rXconvert_tokens_to_stringr0r8rrrrr:r5r9rstacknanmeanr#) r$rrMrrrrr max_entityaverage_scoresr new_entitys r'aggregate_wordz*TokenClassificationPipeline.aggregate_words{~667^7^U]7^7^7^__ #6#< < <a[*F--//C3KEZ&/4FF !%8%< < <X+P+PQQQJ)F--//C3KEZ&/4FF !%8%@ @ @XGGhGGGHHFZQ777N'..00JZ&/ ;F":.EE;<< <a[)B<&   r)c`|tjtjhvrtdg}d}|D]R}||g}|dr||&|||||g}S|)|||||S)z Override tokens from a given word that disagree to force agreement on word boundaries. Example: micro|soft| com|pany| B-ENT I-NAME I-ENT I-ENT will be rewritten with first strategy as microsoft| company| B-ENT I-ENT z;NONE and SIMPLE strategies are invalid for word aggregationNr)r0r6r7r#rr)r$rrM word_entities word_grouprs r'rz+TokenClassificationPipeline.aggregate_words7s  $  &$   Z[[ [   & &F!$X  % &!!&))))$$T%8%8EY%Z%Z[[[$X  !  !4!4ZAU!V!V W W Wr)c>|ddddd}tjd|D}d|D}|tj||j||dd|dd d }|S) z Group together the adjacent tokens with the same entity predicted. Args: entities (`dict`): The entities predicted by the pipeline. rr-r rcg|] }|d S)rr.rs r'rzBTokenClassificationPipeline.group_sub_entities..\sDDDVG_DDDr)cg|] }|d Srr.rs r'rzBTokenClassificationPipeline.group_sub_entities..]s888V&.888r)rr)rrrrr)splitrrmeanrXr)r$rrrtokensrs r'group_sub_entitiesz.TokenClassificationPipeline.group_sub_entitiesSs!X&,,S!44R8DD8DDDEE88x888#WV__N;;FCCa[)B<&   r) entity_namec|dr d}|dd}n&|dr d}|dd}nd}|}||fS)NzB-BrzI-I) startswith)r$rbitags r'get_tagz#TokenClassificationPipeline.get_taghsk  ! !$ ' ' Babb/CC  # #D ) ) Babb/CCBC3wr)cg}g}|D]}|s||||d\}}||dd\}}||kr|dkr||~||||g}|r(||||S)z Find and group together the adjacent tokens with the same entity predicted. Args: entities (`dict`): The entities predicted by the pipeline. rrr)rrr) r$r entity_groupsentity_group_disaggrrrlast_bilast_tags r'rz*TokenClassificationPipeline.group_entitiesvs   / /F& #**6222 ll6(#344GB $ -@-DX-N O O GXh299#**62222$$T%<%<=P%Q%QRRR'-h##  O  !8!89L!M!M N N Nr))NNNNNFNNr`)NN)'r*r+r,r-default_input_names_load_processor_load_image_processor_load_feature_extractor_load_tokenizerrrCrboolr0rrrr r]rrrr(rrrr6rrrndarrayrrrrrrr __classcell__)rJs@r'r=r=Bs-F""H&O!#O#E#E#G#G ( ( ( ( ( (+/*.>B:>.3 $#'N9N9#4.N9"$ N9 '':; N9 !eCHo!67 N9&d^N9 N9C=N9N9N9N9`OsOcOd4S>6JOOOXO [tCy[C[Dd3PS8nAU%?? @%2%2%2%2%2%2N9999v   4=Prs 111111111111:::::: TSSSSSSSSSSS?_^^^^^^YLLLXXXXXXFFFFFFFF:,4000n!!Duuuuu-uuE!!Dup* r)