ޘh) ddlmZmZddlmZddlmZmZmZer ddl m Z m Z m Z GddeZ dded ed ed efd Zd ede d efdZded eedfdZdS))c_floatsizeof)Enum) TYPE_CHECKINGOptionalUnionAutoFeatureExtractor AutoProcessor AutoTokenizerc.eZdZeZedefdZdS)ParameterFormatreturnc*t|jS)zf Number of byte required for this data type Returns: Integer > 0 )rvalue)selfs _/var/www/html/services/rag-service/venv/lib/python3.11/site-packages/transformers/onnx/utils.pysizezParameterFormat.sizesdj!!!N)__name__ __module__ __qualname__rFloatpropertyintrrrrrs> E "c"""X"""rr dimensionfixed_dimensionnum_token_to_addrc |dkr|}||z}|S)ze Args: dimension: fixed_dimension: num_token_to_add: Returns: rr)rrr s r compute_effective_axis_dimensionr"&s#A~~#  !!I rnum_parametersdtypec||jzS)a7 Compute the size taken by all the parameters in the given the storage format when serializing the model Args: num_parameters: Number of parameters to be saved dtype: The data format each parameter will be saved Returns: Size (in byte) taken to save all the parameters )r)r#r$s r"compute_serialized_parameters_sizer&9s EJ &&r model_name)r r r clddlm}m}m} |j|S#t t tf$rd\}} |j|}n#t tf$rYnwxYw |j|}n#t tf$rYnwxYw||t d|d||YdS||cYS|cYSwxYw)a> Gets a preprocessor (tokenizer, feature extractor or processor) that is available for `model_name`. Args: model_name (`str`): Name of the model for which a preprocessor are loaded. Returns: `Optional[Union[AutoTokenizer, AutoFeatureExtractor, AutoProcessor]]`: If a processor is found, it is returned. Otherwise, if a tokenizer or a feature extractor exists, it is returned. If both a tokenizer and a feature extractor exist, an error is raised. The function returns `None` if no preprocessor is found. r r )NNNz&Couldn't auto-detect preprocessor for z1. Found both a tokenizer and a feature extractor.)r r r from_pretrained ValueErrorOSErrorKeyError)r'r r r tokenizerfeature_extractors rget_preprocessorr0GsHFEEEEEEEEE%,},Z888  *%%%'1$ $ 5 5jAAII"    D   D 4 DZ P P  "    D   %6%Bvvvv  #4#<44  "   $ $ $ $)%sbB3A  B3 AB3AB3"A32B33BB3BB3(B3.B32B3N)r)ctypesrrenumrtypingrrrr)r r r rrr"r&strr0rrrr5s:#"""""""1111111111FEEEEEEEEEE " " " " "d " " "c]`il& 's '? 'WZ ' ' ' '&%&%%8p2q)r&%&%&%&%&%&%r