ݘh|fddlZddlZddlZddlZddlZddlZddlZddlZddlm Z m Z ddl m Z ddl mZmZddlmZddlmZddlZddlmZmZddlmZmZmZdd lmZdd lmZm Z dd l!m"Z"m#Z#ej$d krddl%Z%e"rdd l&m'Z'ddl(m)Z)ddl*m+Z+e#rddl,Z,ddlm-Z-mZm.Z.mZe/ej0ej1zZ2e/ej0ej3zej1zdzZ4ddiddiddiddiddiddiddidZ5dZ6dd7e58dZ9Gdd Z:eGd!d"Z;d#e fd$Z<Gd%d&eZ=e>d'kr9e;Z?d(e?_@d)e?_@e=e?ZAeABdSdS)*N)ArgumentParser Namespace) AsyncIterator) dataclassfield)Thread)Optional)AsyncInferenceClientChatCompletionStreamOutput) AutoTokenizerGenerationConfigPreTrainedTokenizer)BaseTransformersCLICommand)ServeArguments ServeCommand)is_rich_availableis_torch_availableWindows)Console)Live)Markdown)AutoModelForCausalLMr BitsAndBytesConfigr z .!\"#$%&'()*+,\-/:<=>?@[]^_`{|}~textz5There is a Llama in my lawn, how can I get rid of it?zyWrite a Python function that integrates any Python function f(x) numerically over an arbitrary interval [x_start, x_end].z4How many helicopters can a human eat in one sitting?z4Count to 10 but skip every number ending with an 'e'zWhy aren't birds real?z2Why is it important to eat socks after meditating?z$Which number is larger, 9.9 or 9.11?)llamacode helicopternumbersbirdssocksnumbers2a **TRANSFORMERS CHAT INTERFACE** Chat interface to try out a model. Besides chatting with the model, here are some basic commands: - **!help**: shows all available commands (set generation settings, save chat, etc.) - **!status**: shows the current status of the model and generation settings - **!clear**: clears the current conversation and starts a new one - **!exit**: closes the interface am **TRANSFORMERS CHAT INTERFACE HELP** Full command list: - **!help**: shows this help message - **!clear**: clears the current conversation and starts a new one - **!status**: shows the current status of the model and generation settings - **!example {NAME}**: loads example named `{NAME}` from the config and uses it as the user input. Available example names: `z`, `a%` - **!set {ARG_1}={VALUE_1} {ARG_2}={VALUE_2}** ...: changes the system prompt or generation settings (multiple settings are separated by a space). Accepts the same flags and format as the `generate_flags` CLI argument. If you're a new user, check this basic flag guide: https://huggingface.co/docs/transformers/llm_tutorial#common-options - **!save {SAVE_NAME} (optional)**: saves the current chat and settings to file by default to `./chat_history/{MODEL_NAME}/chat_{DATETIME}.yaml` or `{SAVE_NAME}` if provided - **!exit**: closes the interface ceZdZddeedeefdZdeedeee ffdZ defdZ d Z d efd Z d ed efd ZddefdZdededefdZdS) RichInterfaceN model_name user_namecrt|_|d|_n||_| d|_dS||_dS)N assistantuser)r_consoler$r%)selfr$r%s b/var/www/html/services/rag-service/venv/lib/python3.11/site-packages/transformers/commands/chat.py__init__zRichInterface.__init__ss?   )DOO(DO  #DNNN&DNNNstreamreturncK|jd|jdt|jd5}d}|d{V23d{V}|jdjj}|s!tjdd|}||z }g}| D]W}| || d r| d B| d Xtd |d }||d6 dddn #1swxYwY|j|S)Nz [bold blue]:)consolerefresh_per_secondrz <(/*)(\w*)>z\<\1\2\>z``` z z github-dark) code_themeT)refresh)r)printr$rchoicesdeltacontentresub splitlinesappend startswithrjoinstripupdate) r*r.livertokenoutputslineslinemarkdowns r+ stream_outputzRichInterface.stream_output~s >4?>>>??? $-A > > >& 4$D%+||||||$ 4$ 4$ 4$ 4$ 4$ 4$ 4e-*08&gFF OO----DLL&&&u--- T**** V,,,,#BGGENN$8$8$:$:}UUU Hd 3333I ,|& 4& 4& 4& 4& 4& 4& 4& 4& 4& 4& 4& 4& 4& 4& 4P  s E D? C3E  EEc~|jd|jd}|j|S)z!Gets user input from the console. [bold red]: )r)inputr%r8)r*rMs r+rMzRichInterface.inputs> ##$F$.$F$F$FGG  r-c8|jdS)zClears the console.N)r)clearr*s r+rOzRichInterface.clears r-rc|jd|jd||jdS)z%Prints a user message to the console.rLz>:[/ bold red] N)r)r8r%)r*rs r+print_user_messagez RichInterface.print_user_messagesE P$.PP$PPQQQ r-colorcx|jd|d||jdS)z,Prints text in a given color to the console.z[bold ]Nr)r8)r*rrSs r+ print_colorzRichInterface.print_colorsC 3U33T33444 r-Fminimalc|jt|rtnt|jdS)z'Prints the help message to the console.N)r)r8rHELP_STRING_MINIMAL HELP_STRING)r*rXs r+ print_helpzRichInterface.print_helpsE HG%T%8%8UUVVV r-generation_config model_kwargsc|jd|d|r|jd||jd||jdS)zFPrints the status of the model and generation settings to the console.z[bold blue]Model: r5z[bold blue]Model kwargs: z [bold blue]NrV)r*r$r]r^s r+ print_statuszRichInterface.print_statuss ????@@@  L M   JL J J K K K =*;==>>> r-)NN)F)__name__ __module__ __qualname__r strr,rr tupleintrJrMrOrRrWboolr\r dictr`r-r+r#r#rs+ ' '8C= 'HSM ' ' ' ',-8R*S,X]^acf^fXg,,,,\s s C $ s?O_cr-r#ceZdZUdZedddiZeeed<edddiZ eeed<eddd iZ eeed <ed dd iZ eed <edddiZ eeed<edddiZ eed<edddiZeeed<edddiZeed<edddiZeed<eddgddZeeed<eddgddZeeed <eddd!iZeed"<eddd#iZeeed$<eddd%iZeed&<eddd'iZeed(<ed)d*d+d)gdZeed,<eddd-iZeed.<ed/dd0iZeed1<ed2dd3iZeed4<d5ZdS)6 ChatArgumentsz Arguments for the chat CLI. See the metadata arg for each argument's description -- the medatata will be printed with `transformers chat --help` Nhelpz_Name of the pre-trained model. The positional argument will take precedence if both are passed.)defaultmetadatamodel_name_or_pathzKUsername to display in chat interface. Defaults to the current user's name.r(zSystem prompt. system_promptz./chat_history/zFolder to save chat history. save_folderz"Path to a yaml file with examples. examples_pathFz7Whether to show runtime warnings in the chat interface.verbosezPath to a local generation config file or to a HuggingFace repo containing a `generation_config.json` file. Other generation settings passed as CLI arguments will be applied on top of this generation config.r]mainzLSpecific model version to use (can be a branch name, tag name or commit id).model_revisionautozDevice to use for inference.devicezA`torch_dtype` is deprecated! Please use `dtype` argument instead.)rvbfloat16float16float32)rlr9 torch_dtypezOverride the default `torch.dtype` and load the model under this dtype. If `'auto'` is passed, the dtype will be automatically derived from the model's weights.dtypez2Whether to trust remote code when loading a model.trust_remote_codezWhich attention implementation to use; you can run --attn_implementation=flash_attention_2, in which case you must install this manually by running `pip install flash-attn --no-build-isolation`.attn_implementationzIWhether to use 8 bit precision for the base model - works only with LoRA. load_in_8bitzIWhether to use 4 bit precision for the base model - works only with LoRA. load_in_4bitnf4zQuantization type.fp4bnb_4bit_quant_typez#Whether to use nested quantization.use_bnb_nested_quant localhostz%Interface the server will listen to..host@zPort the server will listen to.portcJ|j|jdkr|j|_dSdSdS)z(Only used for BC `torch_dtype` argument.Nrv)r{r|rPs r+ __post_init__zChatArguments.__post_init__!s5   'DJ&,@,@)DJJJ ( ',@,@r-)rarbrc__doc__rror rd__annotations__r(rprqrrrsrgr]rurwr{r|r}r~rrrrrrrfrrir-r+rkrks). u )))   %ghD(3-$)5IY@Z#[#[#[M8C=[[[u%6&JhAijjjKjjj#(5Im@n#o#o#oM8C=oooE%6;t2uvvvGTvvv(-u 1  ( ( (x}    %hiNC%&:X1YZZZFCZZZ!&WAAA  """K#!5PAAA  E8C=$e)] ^t*/ r ***#efL$efL$ %uUFZhmotgu=v=vwwwwww!&uHm?n!o!o!o$oookV=d4efffD#fffdf6W-XYYYD#YYY*****r-rkargsc t|S)z; Factory function used to chat with a local model. ) ChatCommand)rs r+chat_command_factoryr(s t  r-c"eZdZedefdZdZedefdZed"de de edefd Z ed"d e ede e fd Zd e ede fd Zde dedeee ffdZededede ede edeee eff dZede de dfdZde dedeffdZdede dede ee eeffdede de e dee e ee ffdZd Zd!ZdS)#rparserctf}|d|}|d}|dtdd|dtdd d |t dS) z Register this command to argparse so it's available for the transformer-cli Args: parser: Root parser to register command-specific arguments chat)dataclass_typeszPositional argumentsmodel_name_or_path_or_addressNz7Name of the pre-trained model or address to connect to.)typermrlgenerate_flagsaFlags to pass to `generate`, using a space as a separator between flags. Accepts booleans, numbers, and lists of integers, more advanced parameterization should be set through --generation-config. Example: `transformers chat max_new_tokens=100 do_sample=False eos_token_id=[1,2]`. If you're a new user, check this basic flag guide: https://huggingface.co/docs/transformers/llm_tutorial#common-options*)rrmrlnargs)func)rk add_parseradd_argument_group add_argumentrd set_defaultsr)rr chat_parsergroups r+register_subcommandzChatCommand.register_subcommand0s)*'''PP ../EFF  +J     W      &: ;;;;;r-c|j|j}|ds*|ds|drkd|_|jdks |jdkrt d|jdd\|_|_|jt d nd |_|j|_ts$ts|jrtd tstd ts|jrtd ||_ dS)NhttphttpsrFruuLooks like you’ve set both a server address and a custom host/port. Please pick just one way to specify the server.:z\When connecting to a server, please specify a model name with the --model_name_or_path flag.TzYou need to install rich to use the chat interface. Additionally, you have not specified a remote endpoint and are therefore spawning a backend. Torch is required for this: (`pip install rich torch`)zHYou need to install rich to use the chat interface. (`pip install rich`)zYou have not specified a remote endpoint and are therefore spawning a backend. Torch is required for this: (`pip install rich torch`)) rr@ spawn_backendrr ValueErrorrsplitrorr ImportErrorr)r*rnames r+r,zChatCommand.__init__Qs  - 95Dv&& M$//'*B*B MdooVaFbFb M%*"9 ++tyD/@/@$J (,'I'P'PQTVW'X'X$ 49*2$v3 &*"*.*L' "" ,>,@,@ TEW x #$$ hii i#%% $*< 7   r-r/ctjdkrtjSt jtjjS)z)Returns the username of the current user.r)platformsystemosgetloginpwdgetpwuidgetuidpw_namerir-r+ get_usernamezChatCommand.get_usernamexs< ?   ) );== < ,,4 4r-Nrfilenameci}t||d<||d<|j}|Atjd}|jd|d}t j||}t jt j |dt|d 5}tj ||d dddn #1swxYwYt j |S) z!Saves the chat history to a file.settings chat_historyNz%Y-%m-%d_%H-%M-%Sz/chat_.jsonT)exist_okwr1)indent)varsrqtimestrftimerrpathrAmakedirsdirnameopenjsondumpabspath)rrr output_dictfoldertime_strfs r+ save_chatzChatCommand.save_chats "&t** J&* N#!  }%899H<SSHSSSHw||FH55H BGOOH--==== (C  0A Ik1Q / / / / 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0wx(((s'C  CCrpc|g}nd|dg}|S)zClears the chat history.Nrroler;ri)rprs r+clear_chat_historyzChatCommand.clear_chat_historys&  DD%-@@AD r-rct|dkriSd|D}d|D}d|D}dtdtfdfd|D}d d |D}d |zd z}|d d}|dd}|dd}|dd}|dd}|dd} t j|}n"#t j$rtdwxYw|S)zUParses the generate flags from the user input into a dictionary of `generate` kwargs.rci|]=}d|ddzdz|dd>S)"=rr)split).0flags r+ z4ChatCommand.parse_generate_flags..sD!p!p!pY]# 3(:":S"@$**S//RSBT!p!p!pr-cni|]2\}}||dvr|n|3S))truefalse)lowerrkvs r+rz4ChatCommand.parse_generate_flags..sJ" " " FJaAAGGII):::qwwyyy" " " r-c*i|]\}}||dkrdn|S)Nonenullrirs r+rz4ChatCommand.parse_generate_flags..s*!m!m!mda!qF{{VV!m!m!mr-sr/c|dr |dd}|dddS)N-r.r4)r@replaceisdigit)rs r+ is_numberz3ChatCommand.parse_generate_flags..is_numbersE||C   abbE99S"a((0022 2r-c>i|]\}}||sd|dn|S)rri)rrrrs r+rz4ChatCommand.parse_generate_flags..s9!t!t!tTQPQ!YYq\\%HXXXXXq!t!t!tr-z, c"g|] \}}|d| S)z: rirs r+ z4ChatCommand.parse_generate_flags..s&*b*b*b41aa;;1;;*b*b*br-{}z"null"rz"true"rz"false"rz"[[z]"rUrrzFailed to convert `generate_flags` into a valid JSON object. `generate_flags` = {generate_flags} Converted JSON string = {generate_flags_string}) lenitemsrdrgrArrloadsJSONDecodeErrorr)r*rgenerate_flags_as_dictgenerate_flags_stringprocessed_generate_flagsrs @r+parse_generate_flagsz ChatCommand.parse_generate_flagss ~  ! # #I "q!pao!p!p!p" " NdNjNjNlNl" " " "n!mNdNjNjNlNl!m!m!m 3 3 3 3 3 3 "u!t!t!tUkUqUqUsUs!t!t!t !% *b*bCYC_C_CaCa*b*b*b c c!$&; ;c A!6 = =h O O 5 = =h O O 5 = =i Q Q 5 = =dC H H 5 = =dC H H!6 = =c3 G G '+z2G'H'H $ $#   D   ('s E$$Fmodel_generation_configc|jd|jvr^tj|j}tj|j}t j||}n>t j|j}n$tj|}|j diddd| |j }|j di|}||fS)zj Returns a GenerationConfig object holding the generation parameters for the CLI command. NrT) do_samplemax_new_tokensri) r]rrrbasenamer from_pretrainedcopydeepcopyrCrr)r*rrrrr]parsed_generate_flagsr^s r+get_generation_parameterizationz+ChatCommand.get_generation_parameterizations  ! -$000'//$*@AA7++D,BCC$4$DWh$W$W!!$4$DTE[$\$\!! !% .E F F  $  $ R RTS'Q'Q R R R!% 9 9$:M N N/(/HH2GHH !,..r- tokenizerr] eos_tokens eos_token_idscv|j|j}n|j}g}|;|||d|2|d|dDt |dkr||j||fS)z:Retrieves the pad token ID and all possible EOS token IDs.N,c,g|]}t|Sri)rf)rtoken_ids r+rz0ChatCommand.parse_eos_tokens..s%]%]%]c(mm%]%]%]r-r) pad_token_id eos_token_idextendconvert_tokens_to_idsrrr?)rr]rrr all_eos_token_idss r+parse_eos_tokenszChatCommand.parse_eos_tokenss  ) 1,9LL,9L  !  $ $Y%D%DZEUEUVYEZEZ%[%[ \ \ \  $  $ $%]%]MDWDWX[D\D\%]%]%] ^ ^ ^  ! !Q & &  $ $%6%C D D D...r- model_argsrc|jr)td|j|j|j|j}n|jrtd}nd}|S)NT)rbnb_4bit_compute_dtyperbnb_4bit_use_double_quantbnb_4bit_quant_storage)r)rrr|rrr)rquantization_configs r+get_quantization_configz#ChatCommand.get_quantization_config sv  " '"4!'1'7$.$B*4*I'1'7 ###   $ '"4!###  #' ""r-rctj|j|j|j}|jdvr|jnt t|j}||}|j|j |d|d}tj|jfd|ji|}t |dd| |j }||fS)N)revisionr})rvNrv)rr~r| device_maprr} hf_device_map) r rmodel_name_or_path_positionalrur}r|getattrtorchrr~rtorw)r*rrr|rr^models r+load_model_and_tokenizerz$ChatCommand.load_model_and_tokenizer s!1  .("4   #jN:: tz@Z@Z"::4@@+#'#; #6   %4  .  BFBX \h   5/4 0 0 8HHT[))Eir- user_input interfaceexamplesr^rcd}|dkr0||j}|n|dkr|n|drt |dkrf|} t | dkr | d} nd} |||| } |d| d d n|d r|d d } | } | D]#} d| vr|d| dd n0$| | } |j di| }|j di|nI|drt |dkr|d}||vr\|g}| ||d| d||ddnd|dt|d}||d nU|dkr||j||n1d}|d|dd |||||fS)z Handles all user commands except for `!exit`. May update the chat history (e.g. reset it) or the generation config (e.g. set a new flag). Tz!clearz!helpz!saverNzChat saved in !green)rrSz!setr1rz(Invalid flag format, missing `=` after `z;`. Please use the format `arg_1=value_1 arg_2=value_2 ...`.red!examplerr(rzExample z* not found in list of available examples: rz!status)r$r]r^F'z/' is not a valid command. Showing help message.ri)rrprOr\r@rrrrWrBrrCrRr?listkeysr`ro)r*r"rr#r$r]r^r valid_command split_inputrnew_generate_flagsrparsed_new_generate_flagsnew_model_kwargs example_name example_errors r+handle_non_exit_user_commandsz)ChatCommand.handle_non_exit_user_commands;s|  ! !**4+=>>D OO     7 " "  " " " "  " "7 + +7 #J4D4D4F4F0G0G!0K0K$**,,K;1$$&q>~~dD(;;H  ! !'C'C'C'C7 ! S S S S  " "6 * *- #",ABB!5!5!7!7 !3!9!9!;!; * 8 8d??))AtAAA$ *E#-1,E,EFX,Y,Y)#;#4#;#X#X>W#X#X # #77&6777  " ": . . #3z7G7G7I7I3J3Ja3O3O%++--a0Lx''!!!,,Xl-CF-KLLL V8Nv8VWWXXXXp|ooW[\d\i\i\k\kWlWlooo%%=%FFFF 9 $ $  " "2"3) #    "M  ! !'f:'f'f'fns ! t t t  " " "]$5|CCr-cRtj|dSN)asynciorun _inner_runrPs r+r9zChatCommand.runs" DOO%%&&&&&r-c PK|jrt|jj|jj|jj|jj|jj|jj|jj |jj |jj |jj d }t|}t|j}d|_||jjdz|jjz}|jj dkrdn |jj }t)|d|jj }|j}|jt,}n@t/|j5} t1j| }dddn #1swxYwY|j|} n|j} t9j|j} ||| \} } t?|j| }| |!|j"}|#d  |$}|%d rx|d kr |&d{VdS|'||||| | | \}}} } |r|%ds |&d{Vn|(d|d|)|d| *|d}|+|d{V}|(d|dn+#tX$rY|&d{VdSwxYw |&d{Vn#|&d{VwxYw)Nerror) rwr|r}r~rrrrrr log_level)targetT@rzhttp://localhostr)r$r%)rXr'z!exit)r"rr#r$r]r^rr*r(r)r]r )r. extra_bodyr')-rrrrwr|r}r~rrrrrrrrr9daemonstartrorur rrDEFAULT_EXAMPLESryaml safe_loadr(rr rrr#rOrrpr\rMr@closer5r?chat_completionto_json_stringrJKeyboardInterrupt)r* serve_args serve_commandthreadr rclientrr$rr(rr]r^r#rr"r.r. model_outputs r+r:zChatCommand._inner_runs_   'y'io"&)"=$(I$A!Y3!Y3$(I$A%)Y%CY^Y^!   J)44M=#4555F FM LLNNN ,s2TY5MM%)Y^{%B%B!! %&@&@ &@&@AAy   %'HHd()) -Q>!,, - - - - - - - - - - - - - - - 9 $$&&DD9D"2"B4CZ"["[*.*N*NtUl*m*m'<!T-DPTUUU &&t'9:: T***) %( %&__.. ((--I!W,,Bllnn$$$$$$$$$?PTOqOq'1!%&/%-.?)5!%PrPPLm-> )! 0E0Ej0Q0Q! (llnn$$$$$$$$+!KKJ G GHHH//->-M-M-O-O!&  0&/%<%\###\#& ] uE[]jEj?k    6PDPDPD! PD sDcN*+ PD , PDPD4jPD tDz+T1 2PDPDPDPDh'''W%W%W%W%W%r-r__main__z meta-llama/Llama-3.2-3b-Instructzhttp://localhost:8000)Cr8rrrrr<stringrargparserrcollections.abcr dataclassesrr threadingrtypingr rDhuggingface_hubr r transformersr r rtransformers.commandsrtransformers.commands.servingrrtransformers.utilsrrrr rich.consoler rich.liver rich.markdownrrrrset ascii_letters whitespaceALLOWED_KEY_CHARSdigitsALLOWED_VALUE_CHARSrCrZrAr-r[r#rkrrrarrrr9rir-r+res   ........))))))(((((((( LLLLLLLL =<<<<<FFFFFFFFDDDDDDDD8? !!JJJ'$$$$$$&&&&&&LLLC,v/@@AAc 6=(6+<'>?? &YYYYYYYYx V*V*V*V*V*V*V* V*ryz%z%z%z%z%,z%z%z%z  z =??D)KD&)@D& ;t  DHHJJJJJ r-