ޘh5ddlZddlmZmZddlmZddlmZddlm Z m Z m Z m Z m Z mZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZddlmZdd l m!Z!dd l"m#Z#dd l$m%Z%dd l&m'Z'dd l(m)Z)ddl*m+Z+ddl,m-Z-ddl.m/Z/ddl0m1Z1ddl2m3Z3ddl4m5Z5ddl6m7Z7ddl8m9Z9ddl:m;Z;ddlm?Z?ddl@mAZAddlBmCZCddlDmEZEddlFmGZGide%de)de+d e7d!e!d"e?d#eAd$e5d%e/d&e9d'e;d(e-d)e1d*eEd+e'd,eGd-eCe3e#e=d.ZHide de de d%ed ed!e d"ed#ed$ed'ed(ed)ed&ed*ed+e d,ed-eee ed.ZIejJeKZLGd/d0ZMGd1d2ZNd3eOfd4ZPd5eOfd6ZQd7ZRdS)8N)OptionalUnion) AutoConfig)logging) AqlmConfigAutoRoundConfig AwqConfigBitNetQuantConfigBitsAndBytesConfigCompressedTensorsConfig EetqConfigFbgemmFp8ConfigFineGrainedFP8Config FPQuantConfig GPTQConfig HiggsConfig HqqConfig Mxfp4ConfigQuantizationConfigMixinQuantizationMethod QuantoConfig QuarkConfig SpQRConfig TorchAoConfig VptqConfig) HfQuantizer)AqlmHfQuantizer)AutoRoundQuantizer) AwqQuantizer)BitNetHfQuantizer)Bnb4BitHfQuantizer)Bnb8BitHfQuantizer)CompressedTensorsHfQuantizer)EetqHfQuantizer)FbgemmFp8HfQuantizer)FineGrainedFP8HfQuantizer)FPQuantHfQuantizer)GptqHfQuantizer)HiggsHfQuantizer)HqqHfQuantizer)Mxfp4HfQuantizer)QuantoHfQuantizer)QuarkHfQuantizer)SpQRHfQuantizer)TorchAoHfQuantizer)VptqHfQuantizerawqbitsandbytes_4bitbitsandbytes_8bitgptqaqlmquantoquarkfp_quanteetqhiggshqqzcompressed-tensors fbgemm_fp8torchaobitnetvptqspqr)fp8z auto-roundmxfp4cDeZdZdZedefdZedZdS)AutoQuantizationConfigz The Auto-HF quantization config class that takes care of automatically dispatching to the correct quantization config given a quantization config stored in a dictionary. quantization_config_dictc |d}|dds|ddr*|ddrdnd}tj|z}n|td|tvr9td|d t t t|}||S) N quant_method load_in_8bitF load_in_4bit_4bit_8bitThe model's quantization config from the arguments has no `quant_method` attribute. Make sure that the model has been correctly quantizedUnknown quantization type, got  - supported types are: ) getrBITS_AND_BYTES ValueError AUTO_QUANTIZATION_CONFIG_MAPPINGlistAUTO_QUANTIZER_MAPPINGkeys from_dict)clsrGrIsuffix target_clss d/var/www/html/services/rag-service/venv/lib/python3.11/site-packages/transformers/quantizers/auto.pyrXz AutoQuantizationConfig.from_dictys/33NCC # ' ' > > BZB^B^_motBuBu  8 < <^U S S`WWY`F->>>[>( # #[ # # #r]rFceZdZdZedeeeffdZedZ edeeefde efdZ e dZ dS) AutoHfQuantizerz The Auto-HF quantizer class that takes care of automatically instantiating to the correct `HfQuantizer` given the `QuantizationConfig`. r_c bt|trt|}|j}|t jkr|jr|dz }n|dz }|tvr9td|dtt t|}||fi|S)NrMrLrOrP) isinstancerlrFrXrIrrRrJrVrSrUrW)rYr_rerIr[s r\ from_configzAutoHfQuantizer.from_configs )4 0 0 X"8"B"BCV"W"W *7  -< < <"/ (' ' 5 5 5:,::/446677::  ,L9 z-88888r]c Ptj|fi|}||S)N)rFrarq)rYrdrer_s r\razAutoHfQuantizer.from_pretraineds04DEbmmflmm2333r]quantization_config_from_argsc  |d}nd}t|trDt|trtj|}nt|}|D|jj|jjkr*td|jjd|jjdt|ttttttfri|g| }|D]\}}t||||dt!|dz }|dkr*t|tst%j|nt(||S) z handles situations where both quantization_config from args and quantization_config from model config are present. NzYou passed `quantization_config` or equivalent parameters to `from_pretrained` but the model you're loading already has a `quantization_config` attribute. The `quantization_config` from the model will be used.zThe model is quantized with z but you are passing a z| config. Please make sure to pass the same quantization config class to `from_pretrained` with different loading attributes.z"However, loading attributes (e.g. z]) will be overwritten with the one you passed to `from_pretrained`. The rest will be ignored.)rprlr rXrF __class__rgrSrr rr rget_loading_attributesitemssetattrrUrWwarningswarnloggerinfo)rYr_rs warning_msgloading_attr_dictattrvals r\merge_quantization_configsz*AutoHfQuantizer.merge_quantization_configss ) 4y K K )4 0 0 \7II \&5&?@S&T&T##&<&F&FGZ&[&[# * 5#-6:W:a:jjjF/B/L/UFFoLoVo_FFF  #YJacno   ~ .9 = T T V V .4466 8 8 c+T37777 ~EVE[E[E]E]@^@^~~~ ~K "  Z0C[%Q%Q  M+ & & & & KK $ $ $""r]c |dd}|dds|ddr*|ddrdnd}tj|z}n|td|tvrGt d|d tt d dSd S) NrIrJFrKrLrMrNrOrPz~. Hence, we will skip the quantization. To remove the warning, you can delete the quantization_config attribute in config.jsonT) rQrrRrSrTr|warningrUrVrW)rGrIrZs r\supports_quant_methodz%AutoHfQuantizer.supports_quant_methods/33NDII # ' ' > > BZB^B^_motBuBu  8 < <^U S S`WWY`F-.register_config_fn s\ 5 5 5DDDDEE E#677 JHII I36(0 r]r`)rrs` r\register_quantization_configr s$ r]namecfd}|S)zRegister a custom quantizer.ctvrtddt|tstd|t<|S)Nz Quantizer 'rz!Quantizer must extend HfQuantizer)rVrSrr)rYrs r\register_quantizer_fnz1register_quantizer..register_quantizer_fns[ ) ) )E4EEEFF F#{++ B@AA A'*t$ r]r`)rrs` r\register_quantizerrs$ ! r]cDt|d}|r!t|jsd}|s|Q|r&t|j||_n||_t|j|} nd} | | |||||| |}| |}| |}t| jdds | jj } t| d| |d<| |||fS)Nr_F) pre_quantized)dtypefrom_tf from_flax device_map weights_only dequantizevaluequant) hasattrrnrr_rrqvalidate_environment update_dtypeupdate_device_mapupdate_tp_planrbrI) configr_rrrrr user_agentr hf_quantizerrIs r\get_hf_quantizerr*s^F$9::M_BB6C]^^  +7  =)8)S)S*,?**F & &*=F &&22  &'3   ))!% *   ))%00!33J?? ,,V44|7uMM O';HL"),"N"NJw   22r])Srztypingrrmodels.auto.configuration_autorutilsrutils.quantization_configrr r r r r rrrrrrrrrrrrrrrbaserquantizer_aqlmrquantizer_auto_roundr quantizer_awqr!quantizer_bitnetr"quantizer_bnb_4bitr#quantizer_bnb_8bitr$quantizer_compressed_tensorsr%quantizer_eetqr&quantizer_fbgemm_fp8r'quantizer_finegrained_fp8r(quantizer_fp_quantr)quantizer_gptqr*quantizer_higgsr+ quantizer_hqqr,quantizer_mxfp4r-quantizer_quantor.quantizer_quarkr/quantizer_spqrr0quantizer_torchaor1quantizer_vptqr2rVrT get_loggerrgr|rFrnstrrrrr`r]r\rsM""""""""777777.++++++444444''''''//////222222222222FFFFFF++++++666666@@@@@@222222++++++------))))))------//////------++++++111111++++++ <++ O   O     " O  >6&!   O!" O#$ %$ ).$ 9$+$+$ J $  J $  J $ l$ [$ $ 9$1$/$ [$}$ $  J!$" J#$$ ! )$$$ .  H % %&#&#&#&#&#&#&#&#Rkkkkkkkk\      !S ! ! ! ! $3$3$3$3$3r]