昺hddlmZmZmZmZmZmZmZmZddl m Z ddl m Z erddl mZmZddlmZGdde ZdS) ) TYPE_CHECKINGAnyDict GeneratorListMappingOptionalUnion)CallbackManagerForLLMRun)LLM)RESTfulChatModelHandleRESTfulGenerateModelHandle)LlamaCppGenerateConfigceZdZUdZeed<eeed< eeed< eeefed< ddeedeedeffd Z e defd Z e de eeffd Z dd ed eeed eededef dZ ddedd ed eededdeeddff dZxZS) Xinferencea`Xinference` large-scale model inference service. To use, you should have the xinference library installed: .. code-block:: bash pip install "xinference[all]" Check out: https://github.com/xorbitsai/inference To run, you need to start a Xinference supervisor on one server and Xinference workers on the other servers Example: To start a local instance of Xinference, run .. code-block:: bash $ xinference You can also deploy Xinference in a distributed cluster. Here are the steps: Starting the supervisor: .. code-block:: bash $ xinference-supervisor Starting the worker: .. code-block:: bash $ xinference-worker Then, launch a model using command line interface (CLI). Example: .. code-block:: bash $ xinference launch -n orca -s 3 -q q4_0 It will return a model UID. Then, you can use Xinference with LangChain. Example: .. code-block:: python from langchain_community.llms import Xinference llm = Xinference( server_url="http://0.0.0.0:9997", model_uid = {model_uid} # replace model_uid with the model UID return from launching the model ) llm( prompt="Q: where can we visit in the capital of France? A:", generate_config={"max_tokens": 1024, "stream": True}, ) To view all the supported builtin models, run: .. code-block:: bash $ xinference list --all client server_url model_uid model_kwargsNc  ddlm}n"#t$r}td|d}~wwxYw|pi}tjdi|||d|jt d|jt d|||_dS)Nr) RESTfulClientz`Could not import RESTfulClient from xinference. Please install it with `pip install xinference`.rrrzPlease provide server URLzPlease provide the model UID) xinference.clientr ImportErrorsuper__init__r ValueErrorrr)selfrrrre __class__s k/var/www/html/services/rag-service/venv/lib/python3.11/site-packages/langchain_community/llms/xinference.pyrzXinference.__init__Vs   7 7 7 7 7 7 7   2   $)r   (& ,    ? "899 9 > !;<< <#mJ// s )$)returncdS)zReturn type of llm. xinferencerrs r" _llm_typezXinference._llm_typevs |c<id|jid|jid|jiS)zGet the identifying parameters.rrrrr&s r"_identifying_paramszXinference._identifying_params{s< T_- DN+ t01  r(promptstop run_managerkwargsc ^|j|j}|di}i|j|}|r||d<|r:|dr%d}|||||D]}||z }|S|||} | ddd S) aqCall the xinference model and return the output. Args: prompt: The prompt to use for generation. stop: Optional list of stop words to use when generating. generate_config: Optional dictionary for the configuration used for generation. Returns: The generated string by the model. generate_configr,stream)modelr+r-r0r+r0choicesrtext)r get_modelrgetr_stream_generategenerate) rr+r,r-r.r3r0combined_text_outputtoken completions r"_callzXinference._calls$ %%dn554:JJ?PRT4U4UBT.B/B  +&*OF #  4228<< 4#% ..' / / . . %-$$' 'vWWJi(+F3 3r(r3)rr r0rc#rK|||}|D]}t|tr|dg}|rk|d}t|trN|dd} |d} |r|| |j| | VdS) a^ Args: prompt: The prompt to use for generation. model: The model used for generation. stop: Optional list of stop words to use when generating. generate_config: Optional dictionary for the configuration used for generation. Yields: A string token. r4r5rr6r2logprobs)r<verbose log_probsN)r: isinstancedictr8on_llm_new_tokenrA) rr3r+r-r0streaming_responsechunkr5choicer<rBs r"r9zXinference._stream_generates$#^^?,  ( $ $E%&& $))Ir22 $$QZF!&$//$ & 62 6 6$*JJz$:$: &'88&+T\Y9$  $ $r()NN)__name__ __module__ __qualname____doc__r__annotations__r strrrpropertyr'rr*rr r>r rr9 __classcell__)r!s@r"rr s@@D KKK &}#sCx.   :%)#'00SM0C=0 000000@3X WS#X%6   X %):> (4(4(4tCy!(467 (4  (4 (4(4(4(4\;?>B !$!$KL!$!$67 !$ "":; !$ 3d? # !$!$!$!$!$!$!$!$r(rN)typingrrrrrrr r langchain_core.callbacksr #langchain_core.language_models.llmsr rr rxinference.model.llm.corerrrr(r"rUsVVVVVVVVVVVVVVVVVVVV======333333ATTTTTTTT@@@@@@D$D$D$D$D$D$D$D$D$D$r(