ݘhcP dZddlZddlmZddlmZddlmZddlm Z m Z m Z m Z ddl mZmZmZmZe r ddlZddlmZe jeZgd gd gd d d gd gdgd d ddgd gd gd d d gd gd gd d d dZdddddddddddddddddddddddddZdZ d$defdZdZdZdZd Z d!Z!d"Z"d#Z#dS)%z;AWQ (Activation aware Weight Quantization) integration fileN)version)ACT2FN)PreTrainedModel)is_auto_awq_availableis_ipex_availableis_torch_availablelogging)AwqBackendPackingMethod AwqConfigAWQLinearVersionExllamaVersion)q_projk_projv_projo_proj) gate_projup_proj down_proj)input_layernormpost_attention_layernormnormF) attentionmlp layernorm use_alibi)w1w3w2g.A)rrrr rope_theta)mistralmixtralllamallavaactc_fc)r%layer_before_act dense_h_to_4hrfc_in gelu_impl) starcoder2RefinedWebModelfalconmptgptjgpt_neox gpt_bigcodebloomcddlm}|tvr|S|D]\}}t|d}t|d}||kr`t ||rPt |t|d}|j}tj|} ||| |j |<t||} |S)Nr)ScaledActivationr%r') awq.modules.actr4AWQ_SCALES_MAPPINGSnamed_childrenhasattrgetattr out_featurestorchones_modulesreplace_quantization_scales) model model_typer4namemoduleact_namelayer_before_act_namer'size scale_like_s e/var/www/html/services/rag-service/venv/lib/python3.11/site-packages/transformers/integrations/awq.pyr>r>Ms000000,,, ,,..<< f&z259 3J ?@R S 8  /D E E &u.A*.MN`.abb #0DD))J#3#3FJ#G#GEN4 ' ; ; Lreturnc |g}|j}tstd|tjkr|jt jkr ddlm }|}n|jt j kr ddl m }|}n|jt j kre|jdtjkr ddlm} | }n~|jdtjkr ddlm} | }nZtd |jd|jt jkr dd lm} | }ntd |jdd lm} | }|D]\} }g| t;|t<jr| |vrtAfd |Dsd|j!}|j"}||j#|j$|||j%du|j&j'|j(| <d}|j(| )dtUtW|,dkrt[||||\}}.d||fS)a Public method that recursively replaces the Linear layers of the given model with AWQ quantized layers. `accelerate` is needed to use this method. Returns the converted model and a boolean that indicates if the conversion has been successful or not. During the module replacement, we also infer the backend to use through the `quantization_config` object. Args: model (`torch.nn.Module`): The model to convert, can be any `torch.nn.Module` instance. quantization_config (`AwqConfig`): The quantization config object that contains the quantization parameters. modules_to_not_convert (`list`, *optional*): A list of modules to not convert. If a module name is in the list (e.g. `lm_head`), it will not be converted. current_key_name (`list`, *optional*): A list that contains the current key name. This is used for recursion and should not be passed by the user. has_been_replaced (`bool`, *optional*): A boolean that indicates if the conversion has been successful or not. This is used for recursion and should not be passed by the user. NzAWQ (either `autoawq` or `llmawq`) is not available. Please install it with `pip install autoawq` or check out the installation guide in https://github.com/mit-han-lab/llm-awqr) WQLinear_GEMM) WQLinear_GEMVr)WQLinear_Exllama)WQLinear_ExllamaV2Unrecognized Exllama version:  WQLinear_IPEXzUnrecognized AWQ version: )WQLinearc3FK|]}|dvVdS).N)join).0keycurrent_key_names rH z*replace_with_awq_linear..s5[[Sschh'7888[[[[[[rI)w_bit group_size in_featuresr:biasdevTF)modules_to_not_convertrYquantization_confighas_been_replaced)/backendr ValueErrorr AUTOAWQrr GEMMawq.modules.linear.gemmrLGEMVawq.modules.linear.gemvrMEXLLAMAexllama_configrONEawq.modules.linear.exllamarNTWOawq.modules.linear.exllamav2rOIPEXawq.modules.linear.gemm_ipexrRawq.quantize.qmodulerSr7append isinstancennLinearanyr]r:bitsr\r^weightdevicer=requires_grad_lenlistchildrenreplace_with_awq_linearpop)r?r`rarYrbrdrL target_clsrMrNrOrRrSrArBr]r:rGs ` rHrr^s8%!#!)G " "  ~   )111  &*:*? ? ? = = = = = =&JJ (,<,A A A = = = = = =&JJ (,<,D D D"1)<@RRRGGGGGG- $3I>.BTTTKKKKKK/  !qBUBdenBo!q!qrrr (,<,A A A B B B B B B&JJW:M:UWWXX X111111 ,,.. ! ! f  #! %%% fbi ( ( ;T9O-O-O[[[[DZ[[[[[ ;$0 %2 '1z-22= +!-D0 , (((t$%)!t$33E::: tFOO%%&& ' '! + +#:'=!1$7"3 $$$ A  R    # ##rIct|tstd|jj|j|j}|j|d<n|jjtvrkt|jj}|j d}|j }|j }t|d|}||d<||d<||d<|j|d<ntd |S) af Returns the fusing mapping given the quantization config and the model Args: model (`~PreTrainedModel`): The model to fuse - note this model should have been converted into AWQ format beforehand. quantization_config (`~transformers.quantization_config.AWQConfig`): The quantization configuration to use. z:The model should be an instance of `PreTrainedModel`, got N max_seq_lenTdecodernum_key_value_heads hidden_sizenum_attention_headsaFusing mapping not found either on the quantization config or the supported `AWQ_FUSED_MAPPINGS`. Please pass a `fused_mapping` argument in the `quantization_config` or raise an issue on transformers https://github.com/huggingface/transformers to add its support.)rur TypeError __class____name__modules_to_fusefuse_max_seq_lenconfigr@AWQ_FUSED_MAPPINGSget_text_configrrr9re)r?racurrent_fused_mappingrrrrs rHget_modules_to_fusers e_ - -qoUZUdUmooppp*6 3 C/B/Sm,,  $6 6 6 25<3J K--d-;;( $8%f.CEXYY0;m,7J347J34/B/Sm,, N    ! rIc t|trtj|}|j}t ||}t |dd}|tjkrddl m }ddl m }ddl m}ntdg |D]\ }|t# fd|Dr#t%|d |||jd krt)| |d ||nt*d t/||| |} | r.  d dt5 dkrg|D]R\ }t# fd Dr1t7|dr!t7|jdr d|j_S|S)aJ Optionally fuse some modules in the model to speedup inference. Args: model (`~PreTrainedModel`): The model to fuse - note this model should have been converted into AWQ format beforehand. quantization_config (`Union[AwqConfig, dict]`): The quantization configuration to use. r`Nr)QuantAttentionFused) QuantFusedMLP)FasterTransformerRMSNormz0Fusing is only supported for the AutoAWQ backendc3 K|]}|vV dSN)rWmodule_name_to_not_convertrAs rHrZz#fuse_awq_modules..s*oo:T-5oooooorIripexrz7The IPEX version AWQ does not support fuse mlp for now.rUc3 K|]}vV dSrr)rWfused_attention_parent_modulefused_attention_modules module_names rHrZz#fuse_awq_modules..(s:;X 66rIr_attn_implementationcustom)rudictr from_dictrdrr9r rfawq.modules.fused.attnrawq.modules.fused.mlprawq.modules.fused.normrre named_modulesrx_fuse_awq_layernormr _fuse_awq_mlploggerinfo_fuse_awq_attention_layersrtsplitr}r8rr) r?rardrr`rrrrBattention_has_been_fusedrrrAs @@@rHfuse_awq_modulesrs\%t,,G'12EFF!)G)%1DEEO$%8:RTXYY)111>>>>>>777777CCCCCCCKLLL ++--?? f ! -ooooXnooooo  OK8&BZ[[[  && 0 0 %u'=v} U U U U KKQ R R R$> 6?D2E$ $   $ ? # * *4::c??1+= > > >  "##a''#(#6#6#8#8 B B K\s B68,,BH^1_1_B9AFM6 LrIc|D]^}t||rLt||}||j|j|jj|j|<~_dS)a Fuse the LayerNorm layers into a target class using autoawq Args: fuse_module_names (`list[str]`): The list of module names to fuse module (`nn.Module`): The pytorch parent module that has layernorm modules to fuse target_cls (`~autoawq.FasterTransformerRMSNorm`): The `FasterTransformerRMSNorm` class as it only supports that class for now. N)r8r9rzvariance_epsilontor{r=)fuse_module_namesrBrr old_modules rHrr0s) 6; ' '  55J+5:!+,,b")** OK (rIc t|dkrdSt||drt||d}t||d}t||d}|jj}|jd} | j} t| } ||||| } | dd\} }| | }t||| |~~~dSdS)a Fuse the MLP layers into a target class using autoawq Args: model (`~PreTrainedModel`): The input pretrained model current_module_name (`str`): The current submodule name fuse_module_names (`list[str]`): The list of module names to fuse. For the MLP layers it has to be an array of length 3 that consists of the 3 MLP layers in the order (gate (dense layer post-attention) / up / down layers) module (`nn.Module`): The pytorch parent module that has layernorm modules to fuse target_cls (`~autoawq.QuantFusedMLP`): The `QuantFusedMLP` class as it only supports that class for now. rNrTrrU) r}r8r9qweightr{rr hidden_actrrsplit get_submodulesetattrr)r?current_module_namerrBrrrrprevious_devicerr activation_fn new_module parent_name child_nameparents rHrrGs $ ""v(+,,*F$5a$899 &"3A"677F$5a$899 #+2--d-;;& z* Z 9g}MM "5"<"boolrrrrrrrrrrIrHrs>= ,,,,,,YYYYYYYYYYYYLLL  H % %>==444LLL >==!!!LLL >==444LLL >==444LLL ): V<<$/JJAAi 8 8w 7 7?CC f== o F F  &  f$f$  f$f$f$f$R&!&!&!R===@.&*&*&*R\!\!\!~2     rI