ݘhdddlZddlZddlZddlZddlmZddlmZmZm Z m Z m Z ddl Z ddl mZddlmZddlmZddl mZddlmZmZmZmZmZdd lmZdd lmZmZm Z m!Z!dd l"m#Z#dd l$m%Z%dd l&m'Z'ddl(m)Z)ddl*m+Z+ddl,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2ddl3m4Z4m5Z5ddl6m7Z7m8Z8m9Z9ddl:m;Z;mZ>m?Z?m@Z@mAZAmBZBmCZCddlmDZDmEZEmFZFmGZGmHZHddlImJZJddlKmLZLmMZMmNZNmOZOmPZPmQZQmRZRmSZSmTZTmUZUmVZVmWZWmXZXmYZYmZZZm[Z[m\Z\m]Z]m^Z^m_Z_m`Z`maZambZbmcZcmdZdmeZeddlfmgZgmhZhmiZimjZjmkZkmlZlmmZmerddlnmoZoddlpmqZqddlrmsZse2jteuZve.rddlwmxZxmyZygdZzeGdde-Z{eGd d!e-Z|eGd"d#e-Z}eGd$d%e-Z~e{Ze{Ze{Ze|Ze|Ze|Ze}Ze}Ze~Ze~Ze eefZe eefZe eefZe eefZe eefZe e{e|fZe e}e~fZe eefZGd&d'eJZd(Zd/d*Zd+ee-d,ed-e-fd.ZdS)0N) dataclass) TYPE_CHECKINGAnyCallableOptionalUnion) file_exists)version)nn)Cache DynamicCacheEncoderDecoderCacheQuantizedCache StaticCache)PretrainedConfig)check_python_requirementsget_cached_module_fileget_class_in_moduleresolve_trust_remote_code)is_deepspeed_zero3_enabled)is_fsdp_managed_module)create_masks_for_generate)isin_mps_friendly)ExtensionsTrie) ModelOutputis_accelerate_availableis_hqq_availableis_optimum_quanto_availableis_torchdynamo_exportinglogging)DisjunctiveConstraintPhrasalConstraint) BeamScorerBeamSearchScorerConstrainedBeamSearchScorer) AssistantVocabTranslatorCacheAssistedCandidateGenerator-AssistedCandidateGeneratorDifferentTokenizersCandidateGeneratorEarlyExitCandidateGeneratorPromptLookupCandidateGenerator%UniversalSpeculativeDecodingGenerator_prepare_attention_mask_prepare_token_type_ids) ALL_STATIC_CACHE_IMPLEMENTATIONS'DEPRECATED_STATIC_CACHE_IMPLEMENTATIONSSTATIC_CACHE_IMPLEMENTATIONSGenerationConfigGenerationMode)ContinuousMixin)#EncoderNoRepeatNGramLogitsProcessor'EncoderRepetitionPenaltyLogitsProcessorEpsilonLogitsWarperEtaLogitsWarperExponentialDecayLengthPenaltyForcedBOSTokenLogitsProcessorForcedEOSTokenLogitsProcessorHammingDiversityLogitsProcessorInfNanRemoveLogitsProcessorLogitNormalizationLogitsProcessorListMinLengthLogitsProcessor!MinNewTokensLengthLogitsProcessorMinPLogitsWarperNoBadWordsLogitsProcessorNoRepeatNGramLogitsProcessor PrefixConstrainedLogitsProcessor RepetitionPenaltyLogitsProcessorSequenceBiasLogitsProcessor$SuppressTokensAtBeginLogitsProcessorSuppressTokensLogitsProcessorTemperatureLogitsWarperTopKLogitsWarperTopPLogitsWarperTypicalLogitsWarper.UnbatchedClassifierFreeGuidanceLogitsProcessor)ConfidenceCriteriaEosTokenCriteriaMaxLengthCriteriaMaxTimeCriteriaStoppingCriteriaStoppingCriteriaListStopStringCriteria)PreTrainedModel)PreTrainedTokenizerBase) BaseStreamer)AlignDevicesHookadd_hook_to_module)past_key_values cache_paramsstatememspast_buckets_statescHeZdZUdZejed<dZee ej ed<dZ ee ej ed<dZ ee e ej ed<dZ ee e ej ed<dZee e e ej ed<dS) GenerateDecoderOnlyOutputaw Outputs of decoder-only generation models, when using non-beam methods. Args: sequences (`torch.LongTensor` of shape `(batch_size, sequence_length)`): The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. scores (`tuple(torch.FloatTensor)` *optional*, returned when `output_scores=True`): Processed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size, config.vocab_size)`. logits (`tuple(torch.FloatTensor)` *optional*, returned when `output_logits=True`): Unprocessed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size, config.vocab_size)`. attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size, num_heads, generated_length, sequence_length)`. hidden_states (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size, generated_length, hidden_size)`. past_key_values (`tuple(tuple(torch.FloatTensor)))`, *optional*, returned when `use_cache=True`): Returns the model cache, used to speed up decoding. Different models have a different cache format, check the model's documentation. Usually, a [`~cache_utils.Cache`] instance. sequencesNscoreslogits attentions hidden_statesr])__name__ __module__ __qualname____doc__torch LongTensor__annotations__rertuple FloatTensorrfrgrhr]e/var/www/html/services/rag-service/venv/lib/python3.11/site-packages/transformers/generation/utils.pyrcrcs415FHU5,- .55515FHU5,- .555<@JuU%6789@@@?CM8E%(9":;<CCCHLOXeE%0A*B$CDELLLLLrsrcceZdZUdZejed<dZee ej ed<dZ ee ej ed<dZ ee ej ed<dZ ee ej ed<dZee e ej ed<dZee e ej ed <dZee e ej ed <dZee e e ej ed <dS) GenerateEncoderDecoderOutputaI Outputs of encoder-decoder generation models, when using non-beam methods. Args: sequences (`torch.LongTensor` of shape `(batch_size*num_return_sequences, sequence_length)`): The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. scores (`tuple(torch.FloatTensor)` *optional*, returned when `output_scores=True`): Processed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size, config.vocab_size)`. logits (`tuple(torch.FloatTensor)` *optional*, returned when `output_logits=True`): Unprocessed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size, config.vocab_size)`. encoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer of the decoder) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. encoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. decoder_attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size, num_heads, generated_length, sequence_length)`. cross_attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size, num_heads, generated_length, sequence_length)`. decoder_hidden_states (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size, generated_length, hidden_size)`. past_key_values (`tuple(tuple(torch.FloatTensor)))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Returns the model cache, used to speed up decoding. Different models have a different cache format, check the model's documentation. Usually, a [`~cache_utils.Cache`] instance. rdNrerfencoder_attentionsencoder_hidden_statesdecoder_attentionscross_attentionsdecoder_hidden_statesr])rirjrkrlrmrnrorerrprqrfrwrxryrzr{r]rrrsrtrvrvs4!!F15FHU5,- .55515FHU5,- .555=Au'8!9:AAA@D8E%*;$<=DDDDHuU->'?!@AHHHBFhuU5+<%=>?FFFGK8E%0A*B$CDKKKHLOXeE%0A*B$CDELLLLLrsrvceZdZUdZejed<dZeej ed<dZ ee ej ed<dZ ee ej ed<dZ eejed<dZee e ej ed<dZee e ej ed <dZee e e ej ed <dS) GenerateBeamDecoderOnlyOutputa Outputs of decoder-only generation models, when using beam methods. Args: sequences (`torch.LongTensor` of shape `(batch_size*num_return_sequences, sequence_length)`): The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. sequences_scores (`torch.FloatTensor` of shape `(batch_size*num_return_sequences)`, *optional*, returned when `output_scores=True`): Final beam scores of the generated `sequences`. scores (`tuple(torch.FloatTensor)` *optional*, returned when `output_scores=True`): Beam transition scores for each vocabulary token at each generation step. Beam transition scores consisting of log probabilities of tokens conditioned on log softmax of previously generated tokens in this beam. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size*num_beams, config.vocab_size)`. logits (`tuple(torch.FloatTensor)` *optional*, returned when `output_logits=True`): Unprocessed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size*num_beams, config.vocab_size)`. beam_indices (`torch.LongTensor`, *optional*, returned when `output_scores=True`): Beam indices of generated token id at each generation step. `torch.LongTensor` of shape `(batch_size*num_return_sequences, sequence_length)`. attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size*num_beams, num_heads, generated_length, sequence_length)`. hidden_states (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size*num_beams*num_return_sequences, generated_length, hidden_size)`. past_key_values (`tuple(tuple(torch.FloatTensor)))`, *optional*, returned when `use_cache=True`): Returns the model cache, used to speed up decoding. Different models have a different cache format, check the model's documentation. Usually, a [`~cache_utils.Cache`] instance. rdNsequences_scoresrerf beam_indicesrgrhr])rirjrkrlrmrnror~rrqrerprfrrgrhr]rrrsrtr}r}s@48hu0188815FHU5,- .55515FHU5,- .555/3L(5+,333<@JuU%6789@@@?CM8E%(9":;<CCCHLOXeE%0A*B$CDELLLLLrsr}c,eZdZUdZejed<dZeej ed<dZ ee ej ed<dZ ee ej ed<dZ eejed<dZee ej ed<dZee ej ed <dZee e ej ed <dZee e ej ed <dZee e ej ed <dZee e e ej ed <dS) GenerateBeamEncoderDecoderOutputa Outputs of encoder-decoder generation models, when using beam methods. Args: sequences (`torch.LongTensor` of shape `(batch_size*num_return_sequences, sequence_length)`): The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. sequences_scores (`torch.FloatTensor` of shape `(batch_size*num_return_sequences)`, *optional*, returned when `output_scores=True`): Final beam scores of the generated `sequences`. scores (`tuple(torch.FloatTensor)` *optional*, returned when `output_scores=True`): Beam transition scores for each vocabulary token at each generation step. Beam transition scores consisting of log probabilities of tokens conditioned on log softmax of previously generated tokens in this beam. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size*num_beams, config.vocab_size)`. logits (`tuple(torch.FloatTensor)` *optional*, returned when `output_logits=True`): Unprocessed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size*num_beams, config.vocab_size)`. beam_indices (`torch.LongTensor`, *optional*, returned when `output_scores=True`): Beam indices of generated token id at each generation step. `torch.LongTensor` of shape `(batch_size*num_return_sequences, sequence_length)`. encoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer of the decoder) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. encoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size*num_beams*num_return_sequences, sequence_length, hidden_size)`. decoder_attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size*num_beams*num_return_sequences, num_heads, generated_length, sequence_length)`. cross_attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size, num_heads, generated_length, sequence_length)`. decoder_hidden_states (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size*num_beams*num_return_sequences, generated_length, hidden_size)`. past_key_values (`tuple(tuple(torch.FloatTensor)))`, *optional*, returned when `use_cache=True`): Returns the model cache, used to speed up decoding. Different models have a different cache format, check the model's documentation. Usually, a [`~cache_utils.Cache`] instance. rdNr~rerfrrwrxryrzr{r])rirjrkrlrmrnror~rrqrerprfrrwrxryrzr{r]rrrsrtrr se((T48hu0188815FHU5,- .55515FHU5,- .555/3L(5+,333=Au'8!9:AAA@D8E%*;$<=DDDDHuU->'?!@AHHHBFhuU5+<%=>?FFFGK8E%0A*B$CDKKKHLOXeE%0A*B$CDELLLLLrsrc$eZdZdZ d~deeeejfdee de fdZ de j dee jd ee j dee je j ffd Zde j dee jd ee j dee je j ffd Z dde j d eed ee j dee jd ee j f dZ ddee jdee jdeeee jfdee jeeeee jfffdZ ddee jdee jdeeee jfde j fdZde jdedeeefde j fdZde jdeededeeeffdZ ddededeee jfde jdee jdee j eee jfff dZe dded e dee j dee j eeefffd!Z dd"e!deeefd e d#edeeeff d$Z"dede j de jd%d&d'e#d(d)d*d)dede$fd+Z% dded,eed-e j d.ee ee jge&efd'ee#deedeeeefd/ee jd0ee jde#fd1Z' dded2ee(d3ed)de(fd4Z)d5ee#e(fd6ee#e(fdee#e(ffd7Z* dd8e jd9ee jd:ee jd;e de jf d<Z+d=Z,deeeffd>Z-d?Z.d@Z/ ddeedAee dBedeeeffdCZ0dDZ1dEededFedefdGZ2e3de fdHZ4deded%d&dedIede f dJZ5de fdKZ6 d~dedLee deee jeffdMZ7dedede fdNZ8e j9 ddee jdeed'ee#d2ee(d.ee ee jge&efdOee d%ed&dPedQd/ee jd0ee jdAee dReeee fdee:e j ffdSZ;dTe dOe de jde fdUZ< dde j d3ed)de j fdVZ=de j d'e#d2e(dedOe dPedQdee>e j ffdWZ?edXe jde jfdYZ@edXe jdedZede jfd[ZAedXe jd:e jde jfd\ZBed]e jd^e jd_e jd`e jdaedbedceddee efdeeCfdfZDed]e jd`e jdge jddee effdhZEdie jdje jdke jdaedcedle dmedZednededee je je jffdoZFdpe jdqe jdre jdge jdZedee je je jff dsZGd8e jdqe jd_e jdpe jd:e jdre jd]e jd`e jdge jdte jdZedaedcedeeCddee efdee je je je jff duZHde j d'e#d2e(dedOe deeIe j ff dvZJde j dweKd'e#d2e(dedOe f dxZLde j dyeMd'e#d2e(dedOe deeIe j ffdzZNde j d{e$d'e#d2e(dedOe dPedQdee>e j ffd|ZOde j defd}ZPdS)GenerationMixinaO A class containing all functions for auto-regressive text generation, to be used as a mixin in model classes. Inheriting from this class causes the model to have special generation-related behavior, such as loading a `GenerationConfig` at initialization time or ensuring `generate`-related tests are run in `transformers` CI. A model class should inherit from `GenerationMixin` to enable calling methods like `generate`, or when it has defined a custom `generate` method that relies on `GenerationMixin`, directly or indirectly, which approximately shares the same interface to public methods like `generate`. Three examples: - `LlamaForCausalLM` should inherit from `GenerationMixin` to enable calling `generate` and other public methods in the mixin; - `BlipForQuestionAnswering` has a custom `generate` method that approximately shares the same interface as `GenerationMixin.generate` (it has a few extra arguments, and the same output). That function also calls `GenerationMixin.generate` indirectly, through an inner model. As such, `BlipForQuestionAnswering` should inherit from `GenerationMixin` to benefit from all generation-related automation in our codebase; - `BarkModel` has a custom `generate` method and one of its inner models calls `GenerationMixin.generate`. However, its `generate` does not share the same interface as `GenerationMixin.generate`. In this case, `BarkModel` should NOT inherit from `GenerationMixin`, as it breaks the `generate` interface. The class exposes [`~generation.GenerationMixin.generate`], which can be used for: - *greedy decoding* if `num_beams=1` and `do_sample=False` - *multinomial sampling* if `num_beams=1` and `do_sample=True` - *beam-search decoding* if `num_beams>1` and `do_sample=False` - *beam-search multinomial sampling* if `num_beams>1` and `do_sample=True` - *diverse beam-search decoding* if `num_beams>1` and `num_beam_groups>1` - *constrained beam-search decoding* if `constraints!=None` or `force_words_ids!=None` - *assisted decoding* if `assistant_model` or `prompt_lookup_num_tokens` is passed to `.generate()` To learn more about decoding strategies refer to the [text generation strategies guide](../generation_strategies). Npretrained_model_name_or_pathtrust_remote_codereturnc tj|}d}|r@tjtj|dsd}nt |dsd}|st d|dd|d}t |||| |t|fd d i|t|fd di|}td |}|S) at Loads and returns a custom generate function, given a model repo. Args: pretrained_model_name_or_path (`str` or `os.PathLike`): Can be either: - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co. - A path to a *directory* containing model weights saved using [`~PreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`. trust_remote_code (`bool`, *optional*): Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to `True` for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine. **kwargs: Additional keyword arguments for remote code loading. Raises: OSError: If `pretrained_model_name_or_path` does not contain a `custom_generate` subdirectory. Returns: A callable that can be used to generate text. Tzcustom_generate/generate.pyF`zw` does not contain a `custom_generate` subdirectory with a `generate.py` file, can't load the custom generate function.zThe repository `zS` contains custom generation code that will override the default `generate` method.)has_local_codehas_remote_code error_messagerequirements_filez custom_generate/requirements.txt module_filegenerate) ospathexistsjoinr OSErrorrrrr) selfrrkwargs is_local_codehas_custom_generate_folderrmodulecustom_generate_functions rtload_custom_generatez$GenerationMixin.load_custom_generate}sX:'DEE %)"  37>>"',,/LNk"l"lmm 3-2*<>[\\ 3-2*) O1OOO  -< - - -  "  )( --'      " )  =_ ci   ( )  7T X^  $7z6#J#J ''rs input_ids inputs_embedscache_positionchtr||||S|,|jddkr|dd|jd df}n\||d|jdkr|dd|jd df}n(|jd|jdkr |dd|f}||fS)a Generic cache-dependent input preparation The code is put in a separate function to allow granular unit testing as it needs a different implementation to be exportable. If we have cache: let's slice `input_ids` through `cache_position`, to keep only the unprocessed tokens - Exception 1: when passing input_embeds, input_ids may be missing entries - Exception 2: some generation methods do special slicing of input_ids, so we don't need to do it here - Exception 3: with synced GPUs cache_position may go out of bounds, but we only want dummy token in that case. - Exception 4: If input_embeds are passed then slice it through `cache_position`, to keep only the unprocessed tokens and generate the first token for each sequence. Later use the generated Input ids for continuation. The current implementation does not rely on ``self`` and could be a class method. It is left as a standard method to be easily rewritten. Nr"r)r ,_cache_dependant_input_preparation_exportingshape)rrrrs rt"_cache_dependant_input_preparationz2GenerationMixin._cache_dependant_input_preparations* $ % % oDDYP]_mnn n  $);q)@)@)!!!n.B1.E-E-G-G*GHMM  %r"ioa&888!!!!n&:1&=%=%?%?"?@II _Q >#7#: : :!!!!^"34Ii''rsc| |dd|f}n.branch_1s3$QQQ)=a)@(@(B(B%BCIIKKKrscZ|dd|jd dfSrrrrs rtbranch_2zNGenerationMixin._cache_dependant_input_preparation_exporting..branch_2s3 ^%9!%<$<$>$>!>?EEGGGrsc>|dd|fSNrrs rtbranch_3zNGenerationMixin._cache_dependant_input_preparation_exporting..branch_3s" N!2399;;;rsr"rcD|||fSrr)rrrrs rtzNGenerationMixin._cache_dependant_input_preparation_exporting..s' ??!))Frscj|tj|d|jdkfd||gfS)Nrr"cltj|jd|jdkd||gS)Nr"rc*|Srrrs rtrzrGenerationMixin._cache_dependant_input_preparation_exporting......syGXGXrsrmcondr)rrrs rtrz`GenerationMixin._cache_dependant_input_preparation_exporting....s? % $-OA$6.:Nq:Q$Q$,%X%X%.$? !"!"rsr)rrrrrs rtrzNGenerationMixin._cache_dependant_input_preparation_exporting.. sR% *2.)/!2DD$'7  Frsr)rrrrrrrs @@@rtrz:5(( $M98i''rsr]attention_maskc i}||d<|||d<||||\}}|jjrdnd}|jjsT|)t||jdkr d||<||d<nL|t j||<d|d<n#|t j||<|jjr|nd} |jjr|d dn|}|jjrd nd } |jjrd nd } || | | ttj |j jvrI|d dz } | |dkd| || <dD]} | | }|q|j| d|djdn||jd}|dd| df}|t j}||| <t'|t(r[|jrS|P|jdkrD|jjs|d|dj\}}}n||jdd\}}t/||j|}t3|dr|nd}t/|dd}||t/|dd}|s| d}| | } t/|dt6}||jt j||f|j|||| |}n/|||||j|||j|}|||| <| | |d <|D]\}}||vr|||<|dd|S)a Prepare the model inputs for generation. It includes operations like computing the 4D attention mask or slicing inputs given the existing cache. See the forward pass in the model documentation for expected arguments (different models might have different requirements for e.g. `past_key_values`). This function should work as is for most LLMs. rNr]decoder_input_idsrr"r) memory_formatdecoder_attention_maskrdecoder_position_ids position_idsrr)rtoken_type_idsrr get_decoder5_prepare_4d_causal_attention_mask_with_cache_positionrrdtype)config input_embedsrrr]rr)sequence_length target_lengthrr batch_sizerr]labels) rris_encoder_decoderlenrrrmcontiguous_formatpopgetsetinspect signatureforward parameterskeyslongcumsum masked_fill_ isinstancer is_compileablendimgetattrbase_model_prefixhasattrrremptyrget_max_cache_shapeitems)rrr]rrrr model_inputs input_ids_keyencoder_attention_maskattention_mask_keyposition_ids_keyrmodel_input_name model_inputcurrent_input_lengthrr_ base_modeldecodercausal_mask_creation_functionrkeyvalues rtprepare_inputs_for_generationz-GenerationMixin.prepare_inputs_for_generation!s$ )7 %&  &.=L* +'+'N'N=.(( $M9 04{/M^++S^ {- a(S-@-@MDWXYDZ-Z-Z.2 ]+0= _--/8ooELco.d.d ]+04 _--*3//H_/*`*`L '48;3Q![W[:>+:X lFJJ/ 6 6 6^l :>9Wm55]m59[5Sg11Yg  & +,,4 C(9$,(G(G(R(W(W(Y(Y$Z$ZZZ)..0077;;a?L  % %n&91 = = ='3F# $![ = =  **%566K&".(++O<<H%_5;A>>)-8>qA) #.aaa2F1F1G1G.G"HK"-"3"3%BY"3"Z"ZK1< -.  . .1 .1 *#q((;1 Tl?6S6_1=o1N1T. OQQ.:=.I.OPRQRPR.S+ O!t'=tDDJ29*m2T2T^j,,...Z^G,3SUY-- )-49L07TVZ11- -4!-!1!12B!C!C+//0@AA 07>Y[t0u0u-!>!>;!&j/-JRVR\!]!]!]#1#1$3!-#1 " " ""?!>"$3"1"E"E"G"G*#1);$3 " " "  %/=L+ , ! --CL) *!,,.. * *JC,&&$) S! 4(((rsinputs bos_token_id model_kwargsc $|jjr2t|dr"|jj|jkr |jjn|jfd|D}|d}||td|dd|dd ||}d krd |vr|d |d n|jjsd ttj |j j v}|std |jjd |||| |d <|d d c}n|td|d d c}||||}||fS)zT This function extracts the model-specific `inputs` for generation. encoderc,i|]\}}||k ||Srrr).0kv input_names rt z9GenerationMixin._prepare_model_inputs..s/bbbARSWaRaRa1RaRaRarsNz `inputs`: z` were passed alongside z0 which is not allowed. Make sure to either pass z or z=...rrzAYou passed `inputs_embeds` to `.generate()`, but the model class z doesn't have its forwarding implemented. See the GPT2 implementation for an example (https://github.com/huggingface/transformers/pull/21405), and feel free to open a PR with it!)rzMYou passed `inputs_embeds` and `input_ids` to `.generate()`. Please pick one.)rrrrmain_input_namerr ValueErrorrrrrrr __class__ri*_maybe_initialize_input_ids_for_generation)rrrr inputs_kwarghas_inputs_embeds_forwardingrs @rt_prepare_model_inputsz%GenerationMixin._prepare_model_inputssA K * .i(( . ,0DDD5JJ-Jbbbb););)=)=bbb $'' D99  #(:IVIIZII,2II8BIII  %!F  $ $L)H)HO,4  1111[3 T/>#%d&HIITYY[[CC0,4$x\`\j\sxxx-1,[,[L|-\-- [)&2/%BO" %$%tuuu%1/%BO" @@Wcddz<//rsc8||S|d}|jjrL|J|jdd}t j|t j|jdzSd}| D]+}t|t j r|j d}n,d|vr(t j|dft j|jS|td t j|dft j|j|zS) z3Initializes input ids for generation, if necessary.Nencoder_outputsrrdeviceir"rrzB`bos_token_id` has to be defined when no `input_ids` are provided.)rrrlast_hidden_statesizermonesrrvaluesrTensorrr)rrrrrrrrs rtrz:GenerationMixin._maybe_initialize_input_ids_for_generations$  M&**+<== ; ) Ro.I#5::<z\GenerationMixin._prepare_encoder_decoder_kwargs_for_generation...Is1II!x**1--IIIIIIrsr)rrr)irrelevant_prefixs @rtrzRGenerationMixin._prepare_encoder_decoder_kwargs_for_generation..FsT   %IIII7HIIIII e   rsrrc$i|] \}}|v || Srrrr)rr)rencoder_signatures rtrzRGenerationMixin._prepare_encoder_decoder_kwargs_for_generation..Ns2$3HeQY]nQnQn%QnQnQnrsoutput_attentionsoutput_hidden_states return_dictrrr) get_encoderrr r!r\r[rrrrrrr/r0r) rr rrr rencoder_kwargsencoder_accepts_wildcardr.r,s @@rt._prepare_encoder_decoder_kwargs_for_generationz>GenerationMixin._prepare_encoder_decoder_kwargs_for_generation3s""$$ 4 ) ) Sw ++ S26 //"7,w7P7P7P7P &'rsrdecoder_start_token_idrc|d|vr|d}n"d|vr|dkr|d}nd}||j}|jdkrH|jd|kr t d|d|jd|dd}n&t j|dft j| |z}||}nd |j j vs5|j j d kr&d |j jj vrn|j j d vrn|dddf|dddfkr_t j||gd }d|vrC|d}t jt j|ddddf|fd }||d<||fS)zGPrepares `decoder_input_ids` for generation with encoder-decoder modelsNrrr"rz1`decoder_start_token_id` expected to have length z but got rrdonutzvision-encoder-decoder)whisperdimr)rrrrrviewrmrrrrilowerr model_typerallitemcat ones_like)rrrrr6rrrs rt)_prepare_decoder_input_ids_for_generationz9GenerationMixin._prepare_decoder_input_ids_for_generation\sC  #(;|(K(K , 0 01D E E   L ( (-=-L-L , 0 0 = =   $  >[F ! &! + +%+A.*<< ~ ~~]s]yz{]|~~&<%@%@Q%G%G " " J?%*VLLLOee #  $ 6  /5577 7 7 K "&> > >7dkNaNlNrNrNtNtCtCt  [ #{ 2 2  1%)?1)EE J J L L Q Q S S P % +ACT*U[] ^ ^ ^ '<77)56N)O&)._%;<._expand_dict_for_generationsq% d d+++&s+7">##6 EE8+9*=*O*OP[ab*O*c*cN3'! !rsNrr:rzMIf `is_encoder_decoder` is True, make sure that `encoder_outputs` is defined.)rGrr)rDrrrrIs` rt_expand_inputs_for_generationz-GenerationMixin._expand_inputs_for_generations !  l* * " " " " "  !33KQ3GGI22<@@  k 122: !pqqq.I.I,WhJi.j.jL* +,&&rsoutputsnum_new_tokenscPtD]$}||vr|dvrd}n|}t||||<n%d|vr@|d}tj||dddfdgd|d<|sHd|vrC|d}tj|||jddfgd|d<nGd |vrC|d } tj| | | jddfgd|d <|d d r|d dd|z|d <nw|d } tj | ddz| d|zdz| j  | j } tj| | f|d <|S)N)rar`r]rrr:rrr"rr$Trr) ALL_CACHE_NAMESrrmrA unsqueezenew_onesrrrarangertor) rrKrrrLpossible_cache_name cache_namerrrpast_positions new_positionss rt#_update_model_kwargs_for_generationz3GenerationMixin._update_model_kwargs_for_generations2$3   "g--&*III!2JJ!4J+27!?16#^%<%RST>UWX=Y%Z%Z[ac222 -. (<77)56N)O&9>+-C-L-LNdNjklNmopMq-r-rs::: 56   K . . X-9:J-KBCC-PSa-aL) * *)--.>??N!Lr"Q&r(:^(Ka(OWeWkb&'' .3Y 7V-W-WL) *rsassistant_modelrXlogits_processortarget_tokenizerrYassistant_tokenizerc ^td|||fD} |jt||||||} n|j(t |j|j|j|j} n| r|jdurZtj |||j j |d} d|j_t!|||||||||  } n_|jdurt#|||||||| } n>t%d t'|jjt+||||||} | S) zU Returns the candidate generator to be used in `assisted_generation` c3K|]}|duV dSrrr)rrs rtr*z;GenerationMixin._get_candidate_generator..s&"s"sQ1D="s"s"s"s"s"srsN)rrXr rr rY)rnum_output_tokensmax_matching_ngram_size max_lengthT)rXassistant_prune_lm_head) rrXr rr rYrZr[atm_translatorF)rrXr rr rYrZr[z7Invalid value for `do_sample`: expected a boolean, got )r?assistant_early_exitr,prompt_lookup_num_tokensr-rr_r` do_sampler(get_translatorrget_text_config vocab_sizer repetition_penaltyr.r*rtyperir)) rr rr rXrYrZr[rdifferent_tokenizerscandidate_generatorrbs rt_get_candidate_generatorz(GenerationMixin._get_candidate_generators #"s"s?L\^q:r"s"s"sss  1 ="=# $"3)+!1 ###   7 C"@.@"3"L(9(Q,7 ###   "-  *d22!>!M$'K//11<$3,0 """HL1D&K'$3&7!-"/%5%5(;#1 ' ' '###,55&S'$3&7!-"/%5%5(; ' ' '##!zdSdSnNoNoNxzz#=# /"3)+!1 ### #"rsinput_ids_seq_lengthencoder_input_idsprefix_allowed_tokens_fnnegative_prompt_idsnegative_prompt_attention_maskc *t} |g}|j<|jdkr1| t|j||| |j|j(| t |j|j?|jdkr4| t|j|j |j |j g|j dkr\t|j dkr*| t|j | ntjd t"|j3|jdkr(| t'|j |j2|jd kr'| t+|j|jf|jd kr[t|j dkr)| t/|j|ntjd t"|j-| t3|j|j|jKt9|dd:|jd kr/| t;|j|j||jLt9|dd;|jd kr0| t?||j|j||0| tA||j |j z|j!'| tE|j!|j#/| tI|j%|j#||j&dur!| tO|j(.| tS|j(|j||j*)| tW|j*||j,@|} |dks|j!| n| dz} | t[|j,| ||.| |} |j/r|j dkrjta|jtbrt|jdz} n:ta|jtdj3r|jj d dz} nd} nd} |j42|j4dkr'| tk|j4|j64|j6d kr)| to|j6| |j84|j8dkr)| ts|j8| |j:)| tw|j:| |j<4|j<dkr)| t{|j<| |j>>d|j>cxkrdkr,nn)| t|j>| |j@?d|j@cxkrdkr-nn*| t|j@| ||jBJ| |jBC|jDEjF||jGdur!| t| S)z This class returns a [`LogitsProcessorList`] list object that contains all relevant [`LogitsProcessor`] instances used to modify the scores of the language model head. Nr")unconditional_idsunconditional_attention_maskr$ sequence_bias)diversity_penalty num_beamsnum_beam_groups?r )penaltyrozyPassing `encoder_repetition_penalty` requires some form of `input_ids` to be passed to `generate`, ignoring the argument.)r}rz{Passing `encoder_no_repeat_ngram_size` requires some form of `input_ids` to be passed to `generate`, ignoring the argument.rrT)top_kmin_tokens_to_keep)top_pr)min_pr)massr)epsilonr)rrr)IrAguidance_scaleappendrPr$rwrIryr>rzr{encoder_repetition_penaltyrrr8warningswarn UserWarningrirHno_repeat_ngram_sizerFencoder_no_repeat_ngram_sizer7 bad_words_idsrEr min_lengthrrBmin_new_tokensrCrGforced_bos_token_idr<forced_eos_token_idr=r`remove_invalid_valuesr? exponential_decay_length_penaltyr;suppress_tokensrKbegin_suppress_tokensrJ_merge_criteria_processor_listrerlistrmr  temperaturerLrrMrrNrrD typical_prOepsilon_cutoffr9 eta_cutoffr:watermarking_configconstruct_processorrrgrhrenormalize_logitsr@) rr rnrorprYrrrqrr processors begin_indexrs rt_get_logits_processorz%GenerationMixin._get_logits_processor;s")**  #!   + 7%4&91O/9      * 6   9HYHghhh i i i  . :?P?beh?h?h   /&7&I/9$5$E     8 D!<CC$*++q00!!; 1 L*; 9  / ;@Q@dhk@k@k   >GXGklll m m m  1 =BSBhklBlBl   :;L;abb c c c  : F!>BB$*++q00!!7)F) 9  * 6   )%3%7     ( 4)+>EEQ!,q00   (%0%7!     , 8)+>EEQ!0144   1(%4%7!     $ /   0,%/3D3TT     0 <   -%9     0 <   -%0%9!     2d : :   9;; < < <  = I   -%F%7(     , 8   -%5!     2 >.K)1,,0A0U0]  1_    4%;!   88EUVV   &- !*Q../A4HH+),->-P)Q)QTU)U&& 1 CU\RR+):)L)RST)UXY)Y&&)*&&%&"!,8=N=Z^a=a=a!!"9:K:W"X"XYYY &27H7NRS7S7S!!$+<+BWijjj!&27H7NQT7T7T!!$+<+BWijjj!&2!!$+<+BWijjj!*6;L;VY\;\;\!!'->-H]oppp!/;FWFf@l@l@l@lil@l@l@l@l@l!!' 1 @Ug !+7CBSB^1` at generate-time. normalize_logits (`bool`, *optional*, defaults to `False`): Whether to normalize the logits (which, for legacy reasons, may be unnormalized). Return: `torch.Tensor`: A `torch.Tensor` of shape `(batch_size*num_return_sequences, sequence_length)` containing the transition scores (logits) Examples: ```python >>> from transformers import GPT2Tokenizer, AutoModelForCausalLM >>> import numpy as np >>> tokenizer = GPT2Tokenizer.from_pretrained("gpt2") >>> model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2") >>> tokenizer.pad_token_id = tokenizer.eos_token_id >>> inputs = tokenizer(["Today is"], return_tensors="pt") >>> # Example 1: Print the scores for each token generated with Greedy Search >>> outputs = model.generate(**inputs, max_new_tokens=5, return_dict_in_generate=True, output_scores=True) >>> transition_scores = model.compute_transition_scores( ... outputs.sequences, outputs.scores, normalize_logits=True ... ) >>> # input_length is the length of the input prompt for decoder-only models, like the GPT family, and 1 for >>> # encoder-decoder models, like BART or T5. >>> input_length = 1 if model.config.is_encoder_decoder else inputs.input_ids.shape[1] >>> generated_tokens = outputs.sequences[:, input_length:] >>> for tok, score in zip(generated_tokens[0], transition_scores[0]): ... # | token | token string | log probability | probability ... print(f"| {tok:5d} | {tokenizer.decode(tok):8s} | {score.numpy():.3f} | {np.exp(score.numpy()):.2%}") | 262 | the | -1.414 | 24.33% | 1110 | day | -2.609 | 7.36% | 618 | when | -2.010 | 13.40% | 356 | we | -1.859 | 15.58% | 460 | can | -2.508 | 8.14% >>> # Example 2: Reconstruct the sequence scores from Beam Search >>> outputs = model.generate( ... **inputs, ... max_new_tokens=5, ... num_beams=4, ... num_return_sequences=4, ... return_dict_in_generate=True, ... output_scores=True, ... ) >>> transition_scores = model.compute_transition_scores( ... outputs.sequences, outputs.scores, outputs.beam_indices, normalize_logits=False ... ) >>> # If you sum the generated tokens' scores and apply the length penalty, you'll get the sequence scores. >>> # Tip 1: recomputing the scores is only guaranteed to match with `normalize_logits=False`. Depending on the >>> # use case, you might want to recompute it with `normalize_logits=True`. >>> # Tip 2: the output length does NOT include the input length >>> output_length = np.sum(transition_scores.numpy() < 0, axis=1) >>> length_penalty = model.generation_config.length_penalty >>> reconstructed_scores = transition_scores.sum(axis=1) / (output_length**length_penalty) >>> print(np.allclose(outputs.sequences_scores, reconstructed_scores)) True ```Nrrr"r:)rmrQrr<rRrexpandrstackreshape transposerrgrhr functional log_softmaxrsummaxrgather) rrdrerrbeam_indices_maskmax_beam_lengthbeam_sequence_indicescut_idxindicestransition_scoress rtcompute_transition_scoresz)GenerationMixin.compute_transition_scoresls h   <q (:;;@@QGGJJ9K[\\L'..r3v;;??LV$$,,S[["==GG1MM  :^^B (C(C(E(E(PRXR^_aRbccFX(44V4CCF^^B R(899F)1,055777<z7GenerationMixin._validate_assistant..s%"o"o"oDSW[nSnSn4SnSnSnrsc3pK|]0}tj|tj|kV1dSr)rr)rrrXrs rtr*z6GenerationMixin._validate_assistant..sRX\ T**go6Ld.S.SSrszThe main model and the assistant don't have compatible encoder-dependent input shapes. Ensure you load the assistant with the correct encoder-decoder class, e.g. `AutoModelForSpeechSeq2Seq` for Whisper.zc(see https://huggingface.co/docs/transformers/en/generation_strategies#universal-assisted-decoding)z`assistant_tokenizer` is not required when the main and assistant models use the same tokenizer. Please omit `assistant_tokenizer` from `generate()` .z~The main and assistant models have different tokenizers. Please provide `tokenizer` and `assistant_tokenizer` to `generate()` )rrdirr?rrgrh)rrXrr[ are_equal doc_referencers`` @rt_validate_assistantz#GenerationMixin._validate_assistantsy  " F ; ) /2H2[ "b"b"b "o"o"o"oC8N4O4O"o"o"o `sI  J r  ; & & ( ( 37M7]7]7_7_7j j j". }mz}}}/.  $7$? fVcfff%@$?rscf|jjrdD]}||dg}tt j|jj}d|vsd|vr.|tt j|jjz}|jjrt||j d}t|dd}||t|dd}|0tt j|jj}||z}t|dd}||t|dd}|:tt j|jj} |d| Dz}| D] \}} | ||vr| |!|rtd|d dS) zXValidates model kwargs for generation. Generate argument typos will also be caught here.)rNrrrrch|]}d|S)r"rr)rxs rt z9GenerationMixin._validate_model_kwargs..)sJJJ!~!~~JJJrsz8The following `model_kwargs` are not used by the model: zG (note: typos in the generate arguments will also show up in this list))rrrrrrrrrrrrrr) rrrunused_model_args model_argsrrencoder_model_argsrdecoder_model_argsrs rt_validate_model_kwargsz&GenerationMixin._validate_model_kwargss ; ) ,, , ,  d++++*4+MNNYZZ  z ! !^z%A%A #g/ ==HII IJ ; ) K t'=tDDJdIt44G:#9!*i>>"%():7?)K)K)V%W%W"00 dIt44G:#9!*i>>"%():7?)K)K)V%W%W"JJ7IJJJJ &,,.. . .JC S %:%:!((---  FK\FFF   rsc 6|r5|j.|jdkr#tjd|jdt||jkr.|jjrdnd}td|d|d |jd d }|r|d |jd z }|j>|j|jkr.tjd|jd|jd|zt|j H|j |z}||jkr5tjd|j d|d|jd|ztdSdSdS)z=Performs validation related to the resulting generated lengthNz0Using the model-agnostic default `max_length` (=zz) to control the generation length. We recommend setting `max_new_tokens` to control the maximum length of the generation.rrzInput length of z is z, but `max_length` is set to z}. This can lead to unexpected behavior. You should consider increasing `max_length` or, better yet, setting `max_new_tokens`.z Generation will stop at the defined maximum length. You should decrease the minimum length and/or increase the maximum length.z" Note that `max_length` is set to z, its default value.z-Unfeasible length constraints: `min_length` (z.) is larger than the maximum possible length (z).z1Unfeasible length constraints: `min_new_tokens` (z$), when added to the prompt length (z/), is larger than the maximum possible length () max_new_tokensr`rrrrrrrr)rr input_ids_lengthhas_default_max_lengthinput_ids_stringmin_length_error_suffixrs rt_validate_generated_lengthz*GenerationMixin._validate_generated_length5s  " &7&F&NSdSosuSuSu MCTC_     0; ; ;6:k6Te22Ze U#3UU9IUU%0UUU  +  "  #g5F5Qggg  #  ' 38I8TWhWs8s8s MR@Q@\RR1B1MRRRTkl     + 7*9t|dr.|p+|jj j|ddj dk}|r|j d||d } td i| |_|rZ|j d |ddj d|d } t|jtd i| |_n|j|jS) z Sets a cache for `generate`, that will persist across calls. A new cache will only be initialized a new `generate` call requires a larger cache or uses a different batch size. Returns the resulting cache object. rN offloaded_cacherr"T)r)rr offloading)rrr)rrrrrself_attention_cachermax_batch_sizercross_attention_cacherrgrrreset) rrrrrrequires_cross_attention_cache offload_cachecache_to_checkneed_new_cacheself_attention_cache_kwargscross_attention_cache_kwargss rt _get_cachezGenerationMixin._get_caches K * ]l.>.>?P.Q.QY].] '$';; 4 " " qA_pT[==eiepNh'' ' <(M9 <, : <+m;  * gdH.E.E r;4BlSdFefgFhFnopFqq   +55d5CC!.+++ ' &DD(CDDDK- l"k99$9GG%12C%DQ%G%Ma%P"/00, 2$+{?j?jMi?j?jkk K     {rscJj otfddDS)a Return `True` if current model can use a `DynamicCache` instance when initializing the `past_key_values`. This adds exception for some models like `Mamba` models which use their own caches and do not need to initialize the Cache in advance in order to save memory (because no back and forth `to_legacy_cache` and `from_legacy_cache` will be performed for mamba-based models). c3NK|]}|jvV dSr)rir=)rspecial_model_nameclss rtr*zBGenerationMixin._supports_default_dynamic_cache..PsJ, , " cl&8&8&:&: :, , , , , , rs)reformerminimaxxlnetlfm2) _is_statefulr?)r"s`rt_supports_default_dynamic_cachez/GenerationMixin._supports_default_dynamic_cacheGsL## , , , , ', , , ) )  rsmax_cache_lengthctfddD}|sdnd}jjp|ddu}||} | r|jt d|dt | trAr-|stj | ntj | ||<dS|j d urdSs,|j#tjd |jd tdS|1|j*t d |jd d|_||} | t&jt&jfvs |jdkri} n dji} |j|jt,vrw|jt.vr+t d|jdt0d|jt5|j|j|z||||<dS|jdkrΉjjsst d|j|jni} d| vrj| d<| dd} | dkrtAstCd| dkrtEstCdtGdd| i| ||<dS|jdkrtdi| ddi||<dSd|jvrtdi| ||<dSdS|s tdi| n#ttdi| tdi| ||<dS) z Prepares the cache for generation (if applicable), given `generate`'s parameterization. If a cache is instantiated, writes it to `model_kwargs`, under the name expected by the model. c3XK|]$}|jjvV%dSr)rrir=)r class_namers rtr*z@GenerationMixin._prepare_cache_for_generation..gs:ttPZjDN,C,I,I,K,KKttttttrs)mambafalconh1r]r^rNzMPassing both `cache_implementation` (used to initialize certain caches) and `zB` (a Cache object) is unsupported. Please use only one of the two.FzThis model does not support `Cache` instances, it only supports the legacy cache format (tuple of tuples). `cache_implementation` (set to z) will be ignored.zRAn assistant model is provided, using a dynamic cache instead of a cache of type='z'. dynamic_fullrzUsing `cache_implementation='z(' is deprecated. Please only use one of z9, and the layer structure will be inferred automatically.)rrrr quantizedzThis model does not support the quantized cache. If you want your model to support quantized cache, please open an issue and tag @zucchini-nlp.backendquantozYou need to install optimum-quanto in order to use KV cache quantization with optimum-quanto backend. Please install it via with `pip install optimum-quanto`HQQzYou need to install `HQQ` in order to use KV cache quantization with HQQ backend. Please install it via with `pip install hqq`rrTdynamicrr)$rrrrrrrrpr(rfrom_legacy_cacherr$rrrrrget_generation_moder5ASSISTED_GENERATIONCONTRASTIVE_SEARCHr1r2r3rrrznum_return_sequences cache_configrgrr ImportErrorrr)rr rrXrr)is_hybrid_cacherTruser_defined_cachegeneration_modedynamic_cache_kwargsr:r1s` rt_prepare_cache_for_generationz-GenerationMixin._prepare_cache_for_generationZstttt^sttttt.=Q&&>  K * ]l.>.>?P.Q.QY].] '*--j99  ) 5A TdnTTT,e44 9]9]9_9_ :SL23EFFF,>?QRRZ( F  &% / / F3355  5A BSBh  F  &+<+Q+]   ?%:???   6:  2 ,??PP  BNDef f f 5GG#% $,dk#:  1 = 59YYY$9=ddd''~8I8^~~&B~~~,0??):)O"#4#>@Q@fggjtt"2!- ,;,, Z((( #7;FF;19]9]9_9_$M BSA_Ak0==qs <//-1[-H-H-J-JL*&**9h??h&&/J/L/L&%S%%.>.@.@%%H,:+Z+Z'+Z\+Z+Z Z((("7;FF+7+`+`:N+`+`[_+`+`+` Z(((/DDD+7+O+O:N+O+O Z(((ED6u 443444()M)M8L)M)M|OsOs^rOsOstt  $ $ $rscdttj|jjvS)z Return True if the current model supports the keyword argument `logits_to_keep` in forward() to save memory. Checking it in this way allows to avoid using a new model attribute. logits_to_keep)rrrrrr)rs rt_supports_logits_to_keepz(GenerationMixin._supports_logits_to_keeps3  3w'8'F'F'Q'V'V'X'X#Y#YYYrskwargs_has_attention_maskc,d fd }||j|}||j|}||j|}||j|}jjr||n|}| |jdkr|d}|F|D||st d|d}t d|djjr|td|At||  r||st d |Jtj|s|dk rt d |d ||_||_||_||_dS)a Prepares the special tokens for generation, overwriting the generation config with their processed versions converted to tensor. Note that `generation_config` is changed in place and stops being serializable after this method is called. That is no problem if called within `generate` (`generation_config` is a local copy that doesn't leave the function). However, if called outside `generate`, consider creating a copy of `generation_config` first. Nc||S||nj}t|tjr||Stj||tjS)Nrr)rrrmr rRtensorr)tokenrrs rt_tensor_or_nonez@GenerationMixin._prepare_special_tokens.._tensor_or_nones\} %1VVt{F%.. (xx'''<fEJGGG Grsr~rzThe attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.z)Setting `pad_token_id` to `eos_token_id`:z for open-end generation.z\`decoder_start_token_id` or `bos_token_id` has to be defined for encoder-decoder generation.r zThe attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.z;`eos_token_id` should consist of positive integers, but is zq. Your generation will not stop until the maximum length is reached. Depending on other flags, it may even crash.r)rrrr6rrrrOrrrrrrrmis_floating_point_bos_token_tensorrr_decoder_start_token_tensor) rr rDrrJbos_token_tensoreos_token_tensorpad_token_tensordecoder_start_token_tensors ` rt_prepare_special_tokensz'GenerationMixin._prepare_special_tokenssU  H H H H H H+?+<+IRXYYY*?+<+IRXYYY*?+<+IRXYYY%4_5F5]fl%m%m%m" ; ) .H.T**Zj '  ',<,AQ,F,F/99!<<   #(8(D(4=V4q 02  NNrGWrrr s s s ; ) .H.Pn   (!+;K[\\\``bb ))4=V4##C  '  #$4 5 5 (:JQ:N9S9S9U9U ( NNrN^rrr   /?+.>+.>+8R555rsc.|jrdS|jjdkp!t|jduo |jj}t |dto |dj }|o|}t|dd||j j z}t|drMt|j}d|vot!|dk}|| z}d |v}|| z}|j|st"d |S) zp Determines whether to trigger auto-compilation of the model's forward pass at generation time. FcudaNr] hf_quantizerrcpur"diskzsYou have set `compile_config`, but we are unable to meet the criteria for compilation. Compilation will be skipped.)disable_compilerrjboolcompile_config_compile_all_devicesrrr rrrUrrrrrrr) rrr valid_hardwareusing_compilable_cache can_compileall_model_deviceshas_cpu_offloadhas_disk_offloads rt_valid_auto_compile_criteriaz,GenerationMixin._valid_auto_compile_criteria3s[  , 5)V3 t  ,D 8 r=N=]=r8 8  |''(9::E B B u|TeGfGu %?)?  4 . . : 4,; ;K 4 ) ) 0 #D$6$=$=$?$? @ @ #'88WSAR=S=SVW=WO . .K &)::  // /K  + 7 7   #    rs synced_gpusstreamerrZcustom_generatec +| dd}| tt| tr_hd++fdtD}|| |j| fd|i| }|d?d|i|S| dd}| dd}|j| fi| \}|| | ||||4tst|otjdk}||n t}||n t!}d t#t%j|jjv}d |v}|d ddu}||j|\}}}|jd }|j}||| |jjsjjc|dkr]tA|jd krEtCj"|dddfjkd krtF$d|jjs |dkrd_%|s|r|r|&|||d <n5|r3|dkr-tA|d jd krtOd|jjrd |vr|(|||}|jjr(|)|||j*|j\}}n|dkr|n|d}|j+d?|tYj-j.|jjd|\}}j/r|0||}|'|1|2|jd}| dduoj3du}| dduoj4du}|5||||||6r d|vrd|d<|7||j3dz }|jd|kr"|dkr|jjs||jdz }|8||||9|} |j-dkrtOd|jj:|jj:krBtwj<d|jj:d|jj:d|jj:dtz|>|||||j|| |  }!|j?d?||d | }"j%|d!<t| tr| ||f|!|"||d"|}#nK| tjBkrj.dkrtOd#j.d$|dkrtOd%|d!stOd&jCd'vrtOd(|jDrtOd)|jEjF|G|||||||*}$|jH|f|$|!|"||d+|}#n}| tjIkrBtFJd,|stOd-tjL||fd.|d/| S| tjMkrMtFJd0|stFJd1tjL||fd2|d/| S| tjNtjOfvr|jP|f|!|"||d"|}#n| tjQtjRfvr|jS|f|!|"|d3|}#no| tjTkrjtFJd4t|j-|jjVjWj.jXj35}%|jY||%f|!|"|d3|}#n| tjZkrtFJd6g}&j[j[}&j\ifd7}'tj\trtAj\d kr |'j\D]}(t|(d trt|(trtA|(d kr |'td8|(Dr |'td9|(Dr |'t|(})ndt|(trtA|(d kr |'td:|(Dr |'t|(})|&a|)!t|&|j-|jjVjWj.j3;}*|jc|f|*|!|"|d<|}#jddurCt|#d=r3t|#jgd>|#jgh|#_g|#S)@a Generates sequences of token ids for models with a language modeling head. Most generation-controlling parameters are set in `generation_config` which, if not passed, will be set to the model's default generation configuration. You can override any `generation_config` by passing the corresponding parameters to generate(), e.g. `.generate(inputs, num_beams=4, do_sample=True)`. For an overview of generation strategies and code examples, check out the [following guide](../generation_strategies). Parameters: inputs (`torch.Tensor` of varying shape depending on the modality, *optional*): The sequence used as a prompt for the generation or as model inputs to the encoder. If `None` the method initializes it with `bos_token_id` and a batch size of 1. For decoder-only models `inputs` should be in the format of `input_ids`. For encoder-decoder models *inputs* can represent any of `input_ids`, `input_values`, `input_features`, or `pixel_values`. generation_config ([`~generation.GenerationConfig`], *optional*): The generation configuration to be used as base parametrization for the generation call. `**kwargs` passed to generate matching the attributes of `generation_config` will override them. If `generation_config` is not provided, the default will be used, which has the following loading priority: 1) from the `generation_config.json` model file, if it exists; 2) from the model configuration. Please note that unspecified parameters will inherit [`~generation.GenerationConfig`]'s default values, whose documentation should be checked to parameterize generation. logits_processor (`LogitsProcessorList`, *optional*): Custom logits processors that complement the default logits processors built from arguments and generation config. If a logit processor is passed that is already created with the arguments or a generation config an error is thrown. This feature is intended for advanced users. stopping_criteria (`StoppingCriteriaList`, *optional*): Custom stopping criteria that complements the default stopping criteria built from arguments and a generation config. If a stopping criteria is passed that is already created with the arguments or a generation config an error is thrown. If your stopping criteria depends on the `scores` input, make sure you pass `return_dict_in_generate=True, output_scores=True` to `generate`. This feature is intended for advanced users. prefix_allowed_tokens_fn (`Callable[[int, torch.Tensor], list[int]]`, *optional*): If provided, this function constraints the beam search to allowed tokens only at each step. If not provided no constraint is applied. This function takes 2 arguments: the batch ID `batch_id` and `input_ids`. It has to return a list with the allowed tokens for the next generation step conditioned on the batch ID `batch_id` and the previously generated tokens `inputs_ids`. This argument is useful for constrained generation conditioned on the prefix, as described in [Autoregressive Entity Retrieval](https://huggingface.co/papers/2010.00904). synced_gpus (`bool`, *optional*): Whether to continue running the while loop until max_length. Unless overridden, this flag will be set to `True` if using `FullyShardedDataParallel` or DeepSpeed ZeRO Stage 3 with multiple GPUs to avoid deadlocking if one GPU finishes generating before other GPUs. Otherwise, defaults to `False`. assistant_model (`PreTrainedModel`, *optional*): An assistant model that can be used to accelerate generation. The assistant model must have the exact same tokenizer. The acceleration is achieved when forecasting candidate tokens with the assistant model is much faster than running generation with the model you're calling generate from. As such, the assistant model should be much smaller. streamer (`BaseStreamer`, *optional*): Streamer object that will be used to stream the generated sequences. Generated tokens are passed through `streamer.put(token_ids)` and the streamer is responsible for any further processing. negative_prompt_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): The negative prompt needed for some processors such as CFG. The batch size must match the input batch size. This is an experimental feature, subject to breaking API changes in future versions. negative_prompt_attention_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Attention_mask for `negative_prompt_ids`. use_model_defaults (`bool`, *optional*): When it is `True`, unset parameters in `generation_config` will be set to the model-specific default generation configuration (`model.generation_config`), as opposed to the global defaults (`GenerationConfig()`). If unset, models saved starting from `v4.50` will consider this flag to be `True`. custom_generate (`str` or `Callable`, *optional*): One of the following: - `str` (Hugging Face Hub repository name): runs the custom `generate` function defined at `custom_generate/generate.py` in that repository instead of the standard `generate` method. The repository fully replaces the generation logic, and the return type may differ. - `str` (local repository path): same as above but from a local path, `trust_remote_code` not required. - `Callable`: `generate` will perform the usual input preparation steps, then call the provided callable to run the decoding loop. For more information, see [the docs](../../generation_strategies#custom-generation-methods). kwargs (`dict[str, Any]`, *optional*): Ad hoc parametrization of `generation_config` and/or additional model-specific kwargs that will be forwarded to the `forward` function of the model. If the model is an encoder-decoder model, encoder specific kwargs should not be prefixed and decoder specific kwargs should be prefixed with *decoder_*. Return: [`~utils.ModelOutput`] or `torch.LongTensor`: A [`~utils.ModelOutput`] (if `return_dict_in_generate=True` or when `config.return_dict_in_generate=True`) or a `torch.LongTensor`. If the model is *not* an encoder-decoder model (`model.config.is_encoder_decoder=False`), the possible [`~utils.ModelOutput`] types are: - [`~generation.GenerateDecoderOnlyOutput`], - [`~generation.GenerateBeamDecoderOnlyOutput`] If the model is an encoder-decoder model (`model.config.is_encoder_decoder=True`), the possible [`~utils.ModelOutput`] types are: - [`~generation.GenerateEncoderDecoderOutput`], - [`~generation.GenerateBeamEncoderDecoderOutput`] rN>rrrerglobal_keys_to_excludec$i|] \}}|v || Srrrr)rrrrgs rtrz,GenerationMixin.generate..s+!u!u!ueSV^tStSt#uStStStrsmodelrr[r"rrrr~r rzA decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.rTrz1`attention_mask` passed to `generate` must be 2D.)rrrr6r)rrDrr`r)r rrrr rrBzZ`streamer` cannot be used with beam search (yet!). Make sure that `num_beams` is set to 1.z~You are calling .generate() with the `input_ids` being on a device type different than your model's device. `input_ids` is on z, whereas the model is on z. You may experience unexpected behaviors or slower generation. Please make sure that you have put `input_ids` to the correct device by calling for example input_ids = input_ids.to('z ') before running `.generate()`.) r rnrorprYrrrqrr)r rrr$)rYrr rcrdzFnum_return_sequences has to be 1 when doing assisted generate, but is rz6assisted generate is only supported for batch_size = 1z+assisted generate requires `use_cache=True`)staticrsliding_windowz=assisted generate is not supported with Static cache classes`zCassisted generation is not supported with stateful models, such as )r rr rXrYrZr[r)rlrYrr rcrdzDoLa generation was moved to a `custom_generate` repo: https://hf.co/transformers-community/dola. To prevent loss of backward compatibility, add `custom_generate='transformers-community/dola'` to your `generate` call before v4.62.0.zDoLa generation requires `trust_remote_code=True` in your `generate` call, since it loads https://hf.co/transformers-community/dola.ztransformers-community/dola)rer raContrastive search was moved to a `custom_generate` repo: https://hf.co/transformers-community/contrastive-search. To prevent loss of backward compatibility, add `custom_generate='transformers-community/contrastive-search'` to your `generate` call before v4.62.0.zContrastive search requires `trust_remote_code=True` in your `generate` call, since it loads https://hf.co/transformers-community/contrastive-search.z)transformers-community/contrastive-search)rYrr rczGroup Beam Search is scheduled to be moved to a `custom_generate` repository in v4.55.0. To prevent loss of backward compatibility, add `trust_remote_code=True` to your `generate` call.)rrzrlength_penaltydo_early_stoppingnum_beam_hyps_to_keepr{r`zConstrained Beam Search is scheduled to be moved to a `custom_generate` repository in v4.55.0. To prevent loss of backward compatibility, add `trust_remote_code=True` to your `generate` call.c4tdjd)Nzo`force_words_ids` has to either be a `list[list[list[int]]]` or `list[list[int]]` of positive integers, but is r)rforce_words_idsr srt typeerrorz+GenerationMixin.generate..typeerror% s3$]8I8Y]]]rsc3BK|]}t|t VdSr)rrr token_idss rtr*z+GenerationMixin.generate..5 s/YY9:i#>#>>YYYYYYrsc3HK|]}td|DVdS)c3NK|] }t|t p|dkV!dSrNrrrtoken_ids rtr*z5GenerationMixin.generate...8 s9jjT\Z#%>%>!>!N(Q,jjjjjjrsNr+rts rtr*z+GenerationMixin.generate..7 sN ) jj`ijjjjjrsc3NK|] }t|t p|dkV!dSrxryrzs rtr*z+GenerationMixin.generate..A s9hhS[Jx$=$= = MAhhhhhhrs) constraintsrrzrrlrmrnr`)constrained_beam_scorerrYrr rcr]to_legacy_cacherr)irrstrlocalsrrrrrrrrrdistget_world_sizerArVrrrrrrrrrrrrRrrrrrmrrrr$rrr5rCrMrJrrzr9 token_healing heal_tokensputrVr`rrrCrr@r6rjrrrrrrr5r7rr'rrirm_assisted_decodingDOLA_GENERATIONrrrr8SAMPLE GREEDY_SEARCH_sample BEAM_SAMPLE BEAM_SEARCH _beam_searchGROUP_BEAM_SEARCHr&rlearly_stoppingr{_group_beam_searchCONSTRAINED_BEAM_SEARCHr}rprrr#r$rr'_constrained_beam_searchreturn_legacy_cacherrr]r),rrr rYrrprcrXrdrqrrrrerrgenerate_argumentsrrr[raccepts_attention_maskrequires_attention_maskrDr rrrrrrrr)r>prepared_logits_processorprepared_stopping_criteriaresultrl beam_scorerfinal_constraintsrrword_ids constraintr~rgs, ` @rtrzGenerationMixin.generate\sf#JJ':DAA  &:os+K+K &&&& ""v!u!u!uvxx~~?O?O!u!u!u   % %f - - -'@t'@((3D(HN(( $,+MM$M:LMM MJJ{D11 $jj)>EE*I$*I 1+ + 5;+ + '< ##L$5$5$7$7888   )=PQQQ  577W;QRV;W;Wv]a]p]r]ruv]vK/?/K++QdQfQf1B1N--ThTjTj!1S9J4<9X9X9c9h9h9j9j5k5k!k"3<"G$0$4$45Et$L$LTX$X!9=8R8R %2L9 9 5 '#(+ % $$%68QZ`$aaa{- "3?NN +,,11ImAAArE26G6YYZZ]^^^l{- /2Bo2U2U*.  '( V-D VI_ V-1-X-X0,..L) * *' V;..3|DT7U7[3\3\_`3`3` !TUUU ; ) .?|.S.SNN|-=?PL ; ) l&*&T&T%!1)'8'T$+ 'U'' #I||*:[)H)H lN^N^_jNkNkI#E$"D# -79J9_``#{=# #  # #  <  * ?((I>>I   LL ) ) )%?1-!'L!9!9T!A!nFWFbjnFn!'L!9!9T!A!nFWFbjnFn ::/#9#9-'- ;    ( ( * * //?|/S/S-.L) * ''(9;KMcddd -7!;   "&6 6 6 O33K24  3A 6 6  ** |_jJZ   ,??PP  %6%@1%D%Dl  ; y/4 4 4 M*@I@P@U**+***UYT_Td ***    %)$>$>/!1+%=- '% 3+I%? % % !&AT%@& /CT`i& & ms& & " %6$? [! ox 0 0Q $_ ";"<"3'!    FF B B B 599 H/DHHHA~~ !YZZZ , P !NOOO 59___ !`aaa  !sZ^ZhZqss #'"?"?"3#+ /!1!*$7)#@ # # -T, $7!:"<"3'!    FF > > >   :    %  J#+!>"3"3     A A A   :    % ##X #+!L"3"3    !68T U U U!T\!:"<"3'! FF!;^=W X X X&T&!:"<"3'   FF @ @ @   s    +%+5$+0?"3"B&7&L 1 A,7   K-T,";"<"3' FF F F F   s   !#  ,8$5$A! 0<##4#DdKK ,<==BBIKKK 1 A99H!(1+t44A)(D99(S]]a=O=O%IKKKYYPXYYYYY(%IKKK-5(&IKKK%:8%D%D )(D99(S]]a=O=O%IKKKhh_ghhhhh(%IKKK%6x%@%@ %,,Z8888'B-%+5$+0?"3"B&7&L,7 ' ' ' #3T2(?!:"<"3' F  1T 9 9 122 :.0ABBN%+%;%K%K%M%MF " rsthis_peer_finishedc|rZtj|rdnd|}tj|tjj|dkrdSn|rdSdS)z Returns whether there are still unfinished sequences in the device. The existence of unfinished sequences is fed through `this_peer_finished`. ZeRO stage 3-friendly. rxr|r~)opFT)rmrHr all_reduceReduceOpSUMr@)rrrcrthis_peer_finished_flags rt_has_unfinished_sequencesz)GenerationMixin._has_unfinished_sequencesf s}  ',l:L3U33RU^d&e&e&e # O3 8I J J J J&++--44u5  5trsctdjj}}t}t d|}d|dD}|dd j|j }tj ||k||} | d kr|S|ddd f }d d fd |D} t!t#|| D] \} \} } || } tj| |kr< fd|| D}t+|dkrs|| fxxdz cc<||| dd } | d krt+| | |kdkr||d <||d ||| <|S)a Generates sequences of token ids for models with a language modeling head. Parameters: input_ids (`torch.LongTensor`): The sequence used as a prompt for the generation. tokenizer (`PreTrainedTokenizerBase`, *optional*): The tokenizer used to decode the input ids. Return: `torch.LongTensor` where each sequence has its tail token replaced with its appropriate extension. Nzs When generating with token healing, you must pass the model's tokenizer to the `tokenizer` argument of `generate`.r")rrc6g|]}|Srr)strip)rr(s rtrz/GenerationMixin.heal_tokens.. s bbb17799bbbrsT)skip_special_tokenspt)return_tensorspaddingrr c3jK|]-}|dV.dS)rN)decodereplace)rt space_tokrs rtr*z.GenerationMixin.heal_tokens.. sASSQY%%a((00i@@SSSSSSrsc>i|]}|fdS)g$@)convert_tokens_to_ids)ralt_tokrs rtrz/GenerationMixin.heal_tokens.. s9FM0099;Trs)prefixr|rvrq)rrrr get_vocabr4 batch_decoderrRrrmwherenumeltolistconvert_ids_to_tokensr enumeratezipr?r@ extensionsrrrrO)rrrrr vocab_trier promptstail_ids tail_toks batch_idxtail_idtail_tok batch_idsseq_bias trimmed_idsrs ` @rtrzGenerationMixin.heal_tokensx s  *  &/%;Y=Sl #I$7$7$9$9:: ,ALYYYcbi&<&| r|d dnd}| r|d dnd}|j dd\}}d}tj |tj|j }|||j|}|j}|||}|rrd t&jd <|jjd kr9|j2|jjr&t0d d|j_||j}|j|j||fi|}d}nd}||||jrB|j|fi|}|| rd| ini|| rd| ini|r|di|ddi}d}n |di|ddi}| |||jj }|r|r|j!dddddf"dtj#|j} ||| }!| ri| r||!fz }| r|| fz }| r6||jj r|j$fn|j%fz }|jj r ||j&fz }| r||jj r|j'fn|j(fz }|rKtRj*+|!d}"tj,|"d-d}#ntj.|!d}#|r|#|z|d|z zz}#tj/||#dddfgd}|'|0|#1||||z}|2dk}|dz }~||||jB||3| r_|jj r+ti||||||||| d Stk|||||| dS|S)a Generates sequences of token ids for models with a language modeling head using **multinomial sampling** and can be used for text-decoder, text-to-text, speech-to-text, and vision-to-text models. Parameters: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): The sequence used as a prompt for the generation. logits_processor (`LogitsProcessorList`): An instance of [`LogitsProcessorList`]. List of instances of class derived from [`LogitsProcessor`] used to modify the prediction scores of the language modeling head applied at each generation step. stopping_criteria (`StoppingCriteriaList`): An instance of [`StoppingCriteriaList`]. List of instances of class derived from [`StoppingCriteria`] used to tell if the generation loop should stop. generation_config ([`~generation.GenerationConfig`]): The generation configuration to be used as parametrization of the decoding method. synced_gpus (`bool`): Whether to continue running the while loop until max_length (needed to avoid deadlocking with `FullyShardedDataParallel` and DeepSpeed ZeRO Stage 3). streamer (`BaseStreamer`, *optional*): Streamer object that will be used to stream the generated sequences. Generated tokens are passed through `streamer.put(token_ids)` and the streamer is responsible for any further processing. model_kwargs: Additional model specific kwargs will be forwarded to the `forward` function of the model. If model is an encoder-decoder model the kwargs should include `encoder_outputs`. Return: [`~generation.GenerateDecoderOnlyOutput`], [`~generation.GenerateEncoderDecoderOutput`] or `torch.LongTensor`: A `torch.LongTensor` containing the generated tokens (default behaviour) or a [`~generation.GenerateDecoderOnlyOutput`] if `model.config.is_encoder_decoder=False` and `return_dict_in_generate=True` or a [`~generation.GenerateEncoderDecoderOutput`] if `model.config.is_encoder_decoder=True`. c36K|]}t|dVdS)rN)r)rrs rtr*z*GenerationMixin._sample.. s,'l'lh.(I(I'l'l'l'l'l'lrsrrNrrgrhr Fr0TOKENIZERS_PARALLELISMflash_attention_2zWhen using Flash Attention 2 and a static cache, you cannot use the option `CompileConfig(fullgraph=True)` as FA2 introduces graph breaks. We overrode the option with `fullgraph=False`.Tr~r/r0r1rrrrrr:r" num_samplesrr] rdrerfrwrxryrzr{r]rdrerfrgrhr])6rr/r0 output_scores output_logitsreturn_dict_in_generaterrerrrrrmrrrr__call__rbrenviron_attn_implementationrZ fullgraphrrget_compiled_callprefill_chunk_size_prefill_chunkingrrrrWrfrRfloat32ryrgrzr{rhr rsoftmax multinomialsqueezeargmaxrArrVrendrvrc)$rrrYrr rcrdrrr/r0rrrhas_eos_stopping_criteriarere raw_logitsryrzr{rwrxrcur_lenrunfinished_sequences model_forwardcompile_forward is_prefillrrKnext_token_logitsnext_token_scoresprobs next_tokenss$ rtrzGenerationMixin._sample s_V): -?0E)7 )7 "3"K$''l'lZk'l'l'l$l$l!%/ 0LML3P PRRD $;\@Q\RRX\"9Z>OZ22VZ'> bCW b^b # t{'E Vg!q.?!@!D!D\!R!R!Rmq H\f ./33ODDDbf " (obqb1 G"$z*EJyO_```77AQS_``  ;;LJ[\\  U36BJ/ 0{/3FFF$3?DUDdDn?''fBG%4> 223D3STTM  / ;141)=N__R^__LJJJ,,-?U^Ue,ffI =4=iXX<XXL   L] e!46G H Hce f f f   Rf n!79M N Nln o o o J$@@@@4@@@" '-II,IIDIII CC#';#ADL  1 !(qqq"aaax 8 ; ;U]clcs ; t t !1 0 [batch_size * num_beams, ...]rr"r Nrrrmr)rHrs rt_flatten_beam_dimz!GenerationMixin._flatten_beam_dim sBV\""}VeAhq&9%:U122Y%FGGGrsrzcnt|j}tj|||g|ddzS)z=[batch_size * num_beams, ...] -> [batch_size, num_beams, ...]r"Nr)rHrrzrs rt_unflatten_beam_dimz#GenerationMixin._unflatten_beam_dim s7V\""}Vj)% Use a heuristic that assumes the best score comes from the current length minus the decoder prompt length. -> See detailed discussion: https://github.com/huggingface/transformers/pull/20901#issuecomment-1369845565 2. `early_stopping == "never"`: -> Estimate the best score using either `max_length` or `cur_len`, depending on the sign of `length_penalty`. -> A positive length penalty favors longer sequences, so we use `max_length` in that case. NOTE: the canonical beam search implementation can be replicated with `early_stopping="never"` and `length_penalty=0.0`, which are NOT the default flags. The default behavior was empirically found to produce better sequences (prior to 2022), and changing it is BC breaking. neverrxNr"T)r;keepdimrer)rmrrr) rrrrrr`rrrlbest_hypothetical_lengthbest_possible_running_scoreworst_finished_scores rt_check_early_stop_heuristicz+GenerationMixin._check_early_stop_heuristic sJ W $ $#)=)='14F'F $ $'.1C'C $&9!!!RaR%&@D\^lDl&m#${+;UY{XYcg=h=h=hij=kmstt2UY '*> >BPT6 6 6   rs!next_token_hits_stopping_criteriactj|}tj||duz}tj|}||z|zS)zv Beam Search stopping condition -- halts the generation loop if any of these conditions becomes False T)rmrr?)rrrrimprovement_possibleexists_open_beamvalid_continuationss rt%_beam_search_has_unfinished_sequencesz5GenerationMixin._beam_search_has_unfinished_sequences s] %y)LMM#Y'788Nd!$($6$67KMf$g$g!!%!3!34EG`!a!a*,19qqq!!!W}-|JxGGGLLRQRSSV__ !:\!IH^!!!!QQQ2D(D"DE57PPPrsrrrc||tjdzz}tj||d}|||}|||} |||} || | fS)z Given the top-K continuations, their scores, and whether they hit a stopping criteria, select the best non-finished beams to continue beam search in the next iteration. rrr")rRrmrrr) rrrrrrztopk_running_log_probsnext_topk_indicesrrrs rt%_get_running_beams_for_next_iterationz5GenerationMixin._get_running_beams_for_next_iteration- s"02S2V2VW\Wd2e2ehn2n!n!J'=KKKAN ../EGXYY"001GIZ[[#112KM^__ "57KKKrstop_num_beam_maskc| | dddfz}|| dz| z |zz }tj|dd|duz}||tjdzz }||tjdzz }||dzz }tj||fd}tj||fd}tj||fd}tj||fd}tj|| d}|||}|||}|||}|||}||||fS) z Updates the finished beams if (and only if) there are new completed sequences that have a higher score than the current finished sequences. Nr"rT)axiskeepdimsrr:r)rmr?rRrrArr)rrdrrrrrrrrr#rzrrrlrdid_top_num_beams_just_finishedbeams_in_batch_are_fullmerged_sequences merged_scoresmerged_beam_indicesmerged_is_sent_finishedtopk_merged_indicess rt_update_finished_beamsz&GenerationMixin._update_finished_beamsC s2+LN_`dfgfgfg`gNh*h'(GaK:L,LQ_+_`"'),<2PT"U"U"UYgkoYo"p144U]CCfLL??CCEMRRU[[[ ;;vEE !9i1G%HaPPP ;"?QGGG #i7P(QWXYYY"')-=?^,_ef"g"g"g#j)DDDQG&&'79LMM ((8KLL ))*=?RSS --.EGZ[[+|5EEErsc |j}|j}|j} |j} |j} |j} |j} |j}|j}|j }|j }|j }|j }|j dd\}}||z}|jjdkr |jj}nH|jjdkr|j}n|jj}|}d}| |j dnd}t-dd|z|z}t/jt/j|t.jt/j||z t.jfd |j}|||j|}|j}|rtAd | r| rd nd}| r| rd nd}| r| rd nd} | r| rd nd}!| r| rd nd}"| r| rd nd}#| rJ|jj!r>| r|d "d nd}$| r|d "dnd}%| |p|dnd}&t/j#|||f|&t.j$|j}'|%||||'ddddd|f<|'&'}(t/j||ft.j(|j})d|)ddddf<t/j#||fdt.j(|j}*t/j||ft.j|j}+t/j|dft.j|j},t/j||ft.j|j}-t/j#||||z fdt.j)|j}.|.&'} |*|||jrc|+|'ddddd|f}/|j,|/fi|}0|0-| rd| ini|0-| rd| ini|d i|0ddi}1|.|1||jj!}|r|r|1j/dddddfdt.j0|j}2tbj23|2d }3||/|3}3| r| r||2'fz }| r| r||3'fz }| r6|!|jj!r|1j4fn|1j5fz }!|jj!r |"|1j6fz }"| r|#|jj!r|1j7fn|1j8fz }#~1|%|3||}3|3|)dddddfz}3t/j9|3|||zf}3|:|3|'|.||||||| \}4}5}6||+|5ddddd|dzf|}-|%|-||}-|;|4|5|6|-|\}'})}.|<|(|5|*|4| |6|,|+|-||||||\}(}*} }+|"ddk|+|.d||z f}7t{|dr |>|d|7|d<n|d?|7|dz}|@|,|)|*|+||||| },|A|,|+|-| }|*|||jc|+|(ddd|ddf}(|+|*ddd|f}*|+| ddd|ddf} | dzBd }8||8z}9|(ddd|9f}(| ddd|8f} | rg| sd}*|jj!r-t|(|*||| |$|%|!|"|#|"d! St|(|*||| |!|#|"d"S|(S)#a Generates sequences of token ids for models with a language modeling head using **beam search decoding** and can be used for text-decoder, text-to-text, speech-to-text, and vision-to-text models. If it's the first time you're diving into Beam Search, we recommend you read the following blog post: https://huggingface.co/blog/how-to-generate (especially the beam search section). You can recompute the sequence scores from the individual scores using the `compute_transition_scores` function (https://huggingface.co/docs/transformers/main_classes/text_generation#transformers.GenerationMixin.compute_transition_scores) Parameters: input_ids (`torch.LongTensor` of shape `(batch_size*num_beams, sequence_length)`): The sequence used as a prompt for the generation. logits_processor (`LogitsProcessorList`): An instance of [`LogitsProcessorList`]. List of instances of class derived from [`LogitsProcessor`] used to modify the prediction scores of the language modeling head applied at each generation step. stopping_criteria (`StoppingCriteriaList`: An instance of [`StoppingCriteriaList`]. List of instances of class derived from [`StoppingCriteria`] used to tell if the generation loop should stop. generation_config ([`~generation.GenerationConfig`]): The generation configuration to be used as parametrization of the decoding method. synced_gpus (`bool`): Whether to continue running the while loop until max_length (needed to avoid deadlocking with `FullyShardedDataParallel` and DeepSpeed ZeRO Stage 3). model_kwargs: Additional model specific kwargs will be forwarded to the `forward` function of the model. If model is an encoder-decoder model the kwargs should include `encoder_outputs`. Return: [`generation.GenerateBeamDecoderOnlyOutput`], [`~generation.GenerateBeamEncoderDecoderOutput`] or `torch.LongTensor`: A `torch.LongTensor` containing the generated tokens (default behaviour) or a [`~generation.GenerateBeamDecoderOnlyOutput`] if `model.config.is_encoder_decoder=False` and `return_dict_in_generate=True` or a [`~generation.GenerateBeamEncoderDecoderOutput`] if `model.config.is_encoder_decoder=True`. Nr MoshiDepthDecoderImageGPTForCausalImageModelingFrr"rr:z`low_memory=True` is not supported after the beam search refactor. Please check the discussion in #35802 *after the PR got merged*, and add a comment there if your questions are not yet answered.rrrrgrhr) fill_valuerrrrr~r/r0r1Trr) rrrrrrerrzrhr)rrrrrz)rdrrrrrrrrr#rzrrrlrr]._reorder_cache) rrrrrr`rrrl rdr~rerfrrwrxryrzr{r]rdr~rerfrrgrhr])Errr/r0rrrrerrlr`rzr9rrriraudio_vocab_sizeget_output_embeddings out_featuresrgrhrrmrArrYzerosrRrr low_memoryrrrfullr rdetachrfloatint32rrrrrWrfrr rrryrgrzr{rhrrr"r.rr3 reorder_cacherr rrr}):rrrYrr rcrrrr/r0rrrrerrlr`rzr9batch_size_unflattenedrrrhrr n_eos_tokensrr# sequential all_scoresrrryrzr{rwrxoutput_fill_valuerrdrrrrrrflat_running_sequencesr model_outputsrf log_probsrrrbeam_idxmax_generated_length output_lengths: rtrzGenerationMixin._beam_searchz s8 \): (: -?0E)7 )7 "3"K%/ *9*9&1 %/ 0E*3/"1"*='+y8 > "&9 9 95JJ ^ $(H H H3355BJJ4466AJ$" 1=0H|)!,,a Aq</009< !I Z5: 6 6 6 ]U^E^glgq8r8r8r s    "Y     77AQS_`` '1  t  4P PRRD 3P PRRD 5R-Rrrd $;\@Q\RRX\"9Z>OZ22VZ'> bCW b^b # t{'E Vg!q.?!@!D!D\!R!R!Rmq H\f ./33ODDDbf "@L?WL;LO]_!J J /(+#     -1,D,DYPZ\e,f,f!!!QQQ.)%,,..4466 $k:y*A]f]mnnn%)AAAqrrE"j*i!8TQVQ\eneuvvv !; I'>ejYbYijjj/4j*aPUPZclcs.t.t.t+-2K  #5:i>N- - - ) %z J$8 9bPUP[dmdt   ,2244::<< ,,-?U^Ue,ffL %)%;%;TeeXdeeL   L] e!46G H Hce f f f   Rf n!79M N Nln o o o DBB<BBTBBBM CC#';#ADL  1 #)!!!R(366D ^g^n6ooF 11&b1AAI(()?KKI'  46<<>>"33J*7}79??#4#4"66J$N&;999;;+68& {5N(]-K,MM(');9<<>>+9;)00J RRI!$7111d $CCI i*i*>T1UVVIQUPmPm&/"3%9#5#+#%%Qn Q Q MN24M1B0A&&'=aaaMgPQkM>Q'RSS11 -150H0H1:}11 - LPKuKu-'=*C2S# LvLL H 24HFJE`E`#'='-)*C4W!12S"3##5--FaFF BI{L2B. 1488D112FsGVhLhGh2ijj4!122L6:6I6I,WhJiks6t6tL!233 !23AA(KKKkG262R2R4W$7'!1%#5--3S 3 3 /&*%O%O3 1 &&" O,,-?U^Ue,ffL `**9QQQ8M9M8Mqqq5P+QRR ,,[S?S>SUVUVUV;V.WXX ".!1 7 7 9 9>>1>EEIIKK*-AA aaa-/0 #AAA'<(<'<$<= "   #" {- 7'%0%%!-'9*?'9%5*?$0$4$45F$G$G    5'%0%%!-1"7$0$4$45F$G$G     rsrc 456789:|j}|j} |j} |j} |j} |j} |j}|j9|j}9|z:t|j |z5|j }|j \}}| ||j |}|r| r5:fdt|D8nd895z|krtd95zd|d|r| rdnd}|r| rdnd}|r| rdnd}|r| rdnd}|r| rdnd}|rJ|jjr>| r|ddnd}| r|dd nd}t'j59fd t&j| }d |dddd:f<|59zf}d }|j d}||||j rt'j59z|j| }t'j59zt&j| }|j|fi|}|| rd| ini|| rd| ini|di|ddi} || ||jj}|r|r|dz}| r&t'j| jdddddf}!| r.| jdddddf d|j }"t|D]66:z}#tC|#:z9}$|$|#z }%g}&t5D]24|&"49fdt|#|$D3||&}'| j|&dddf t&j#|j }(tHj%&|(d})|)j d}*||'|)|6}+|+||&'dz})|)(|+})| r|+|!|&<|)5|%|*z})| | j d nd },t'j)|)tUdd|,z|%zddd\})}-t'j+|-|*d}.|-|*z}-8tY8dnd}/|-|'|)|-|.|| |/6| }0|0d||&<|0d }1|0d!7|rB| r@t]678fd"tt8d D86<|'7||&<t'j/|'7ddf|1'dgd}'|'dddf||&<9t'j+7|%dz|#z7|%zz||&<|ri| r||!fz }| r||"fz }| r6||jjr| j0fn| j1fz }|jjr || j2fz }| r||jjr| j3fn| j4fz }t'j/||'dgd}~ |d#dKtk|d$r |6|d#||d#<n|d#7||dz}|j8sts|||rd}||||j 8tY8dnd}2|:|||-|.|| |j;|2|% }3|r| sd|3d&<|jjr?ty|3d'|3d&|||3d(||||||d#) St{|3d'|3d&|||3d(|||d#*S|3d'S)+a Generates sequences of token ids for models with a language modeling head using **diverse beam search decoding** and can be used for text-decoder, text-to-text, speech-to-text, and vision-to-text models. Parameters: input_ids (`torch.LongTensor` of shape `(batch_size*num_beams, sequence_length)`): The sequence used as a prompt for the generation. beam_scorer (`BeamScorer`): An derived instance of [`BeamScorer`] that defines how beam hypotheses are constructed, stored and sorted during generation. For more information, the documentation of [`BeamScorer`] should be read. logits_processor (`LogitsProcessorList`): An instance of [`LogitsProcessorList`]. List of instances of class derived from [`LogitsProcessor`] used to modify the prediction scores of the language modeling head applied at each generation step. stopping_criteria (`StoppingCriteriaList`): An instance of [`StoppingCriteriaList`]. List of instances of class derived from [`StoppingCriteria`] used to tell if the generation loop should stop. generation_config ([`~generation.GenerationConfig`]): The generation configuration to be used as parametrization of the decoding method. synced_gpus (`bool`): Whether to continue running the while loop until max_length (needed to avoid deadlocking with `FullyShardedDataParallel` and DeepSpeed ZeRO Stage 3). model_kwargs: Additional model specific kwargs that will be forwarded to the `forward` function of the model. If model is an encoder-decoder model the kwargs should include `encoder_outputs`. Return: [`~generation.GenerateBeamDecoderOnlyOutput`], [`~generation.GenerateBeamEncoderDecoderOutput`] or `torch.LongTensor`: A `torch.LongTensor` containing the generated tokens (default behaviour) or a [`~generation.GenerateBeamDecoderOnlyOutput`] if `model.config.is_encoder_decoder=False` and `return_dict_in_generate=True` or a [`~generation.GenerateBeamEncoderDecoderOutput`] if `model.config.is_encoder_decoder=True`. c bg|]+}tdtzD,S)c3K|]}dVdSrrNrrrrs rtr*z@GenerationMixin._group_beam_search...s"!P!P"!P!P!P!P!P!Prs)rprange)rrr num_sub_beamss rtrz6GenerationMixin._group_beam_search..s>qqqUVE!P!PeMJ4N.O.O!P!P!PPPqqqrsN)Batch dimension of `input_ids` should be , but is rrrrrgrhrrrFr"r~r/r0r1Trr)rrc g|] }z|z Srrrr)ridxrrzs rtrz6GenerationMixin._group_beam_search..Rs#fffY.4fffrsr:)current_tokensbeam_group_idxr r;largestsortedfloor) rounding_mode)rrr group_indexrnext_beam_scoresnext_beam_tokensnext_beam_indicesc3TK|]"}||fzV#dSrrr)rirWrHrs rtr*z5GenerationMixin._group_beam_search..sM99WX ^4Xa[AXa[NR999999rsr]r3rrr`rrsequence_scoresrdrr4r5)>rrr/r0rrrrzr{r _beam_hypsrrrrPrrrrrmr;r=r<rr9rrrrrW zeros_likerfrRrextendrr rrrO expand_asrrdivrprocessrprAryrgrzr{rhrr3r?is_doner?finalizer`rr});rrrrYrr rcrrrr/r0rrrr{rbatch_beam_sizerrerryrzr{rwrxrrrrVreordering_indicesrrKprocessed_scoreraw_logit_scoregroup_start_idx group_end_idx group_sizebatch_group_indicesgroup_input_idsrrrhnext_token_scores_processedrAr next_indicesprocess_beam_indices beam_outputsbeam_next_tokensfinal_beam_indicessequence_outputsrrrWrHrrzrQs; @@@@@@@rtrz"GenerationMixin._group_beam_search s V): (: -?0E)7 )7 "3"K) %5!_4 /00OC !#,? 77AQS_`` " } qqqqqZ_`oZpZpqqqLLL z !_ 4 4oI OZ22VZ'> bCW b^b # t{'E Vg!q.?!@!D!D\!R!R!Rmq H\f ./33ODDDbf " j*i!8$ekZ`aaa *+ AAA&'!&& Y(>'@AA "&_Q/,,-?U^Ue,ff[ *"[i)?y_efffN"'Z)-C5:^d!e!e!e >4=iXX<XXL   L] e!46G H Hce f f f   Rf n!79M N Nln o o od<<\<!!!R(3K"L"L b#*.B":"="=4PYP`"="a"a"'"8"8O O "0="@ #Om$CY O O *_< ')#!&z!2!2I'..fffffoWd@e@efff#,,?"@ %,N3FAAA3M$N$Q$Q- 0@%R%%!%'M$=$=%2%>%%!/4R8 .>.>#%6~ft///+%@+NaBbBlBlmoBpBp$p!$5$?$?@[$\$\! W;VO$78%6$:$::zT^G^$_$_!9E8P|1!44VW 16%s1a,.>'?'?*'LRS]ajn222.!; %yjPWXXX )J6 AM@Xs<'<'<'<^b$*22#% !-!-!5 .'9 3   4@@R3S /0#/0B#C '(;<*}38999999\abefrstfubvbv\w\w99944L02A1J -."')_Xqqq[-IK[KeKefhKiKi,jpr"s"s"s6Eaaae6L23  (Jg V V VV%&*,.##677'  100F 5?"44J$H&9=9Wr355^e^p]r&{5H(W-E,GG(');96688%35)  9n.F.Fr.J.J"KQSTTTI  1488D4!122V6:6I6I$%679K77L!233!!23AABTUUUkG" *c*;*;Iv*N*N&O&O *%)"w,,-?U^Ue,ff[ *z7C6NSr222TX&//    %%(3+10    # 1  ;6: !23{- 7.{;%56G%H!%!1.!A'9*?'9%5*?$0$4$45F$G$G    5.{;%56G%H!%!1.!A1"7$0$4$45F$G$G    $K0 0rsr~c *+|j}|j} |j} |j} |j} |j} |j}t|j}|j }|j dd\}}| ||j |}||z|krtd||zd|d|r| rdnd}|r| rdnd}|r(| r&tdt|Dnd+|r| rdnd}|r| rdnd}|r| rdnd}|rJ|jjr>| r|dd nd}| r|dd nd}t'j||ft&j|j }d |ddd df<|||zf}d}|j d }||||j rR|j|fi|}|| rd| ini|| rd| ini|di|ddi}||||jj}|r|r|d z}|jdddddfdt&j|j }t<j |d} ||| }!|!|dddf!|!z} | "}"|ri| r|| fz }| r||fz }| r6||jjr|j#fn|j$fz }|jjr ||j%fz }| r||jjr|j&fn|j'fz }| j d}#| |||#z} | | j dnd}$t'j(| tSdd |$z|zd dd\} }%|%|#z *}&|%|#z}%|+|| |%|&|"|| +| }'|'d}|'d}(|'d*t'j,|*ddf|(-dgd}~|ddKt]|dr |/|d*|d<n|d0*|r8| r6t*+fd tt+D+|d z}|j1ste|||rd}||||j R|3|||%|&|| |j4+|! })|r| sd|)d"<|jjr?tk|)d#|)d"|||)d$||||||d% Stm|)d#|)d"|||)d$|||d&S|)d#S)'a| Generates sequences of token ids for models with a language modeling head using **constrained beam search decoding** and can be used for text-decoder, text-to-text, speech-to-text, and vision-to-text models. Parameters: input_ids (`torch.LongTensor` of shape `(batch_size*num_beams, sequence_length)`): The sequence used as a prompt for the generation. constrained_beam_scorer (`ConstrainedBeamSearchScorer`): A derived instance of [`BeamScorer`] that defines how beam hypotheses are constructed, stored and sorted during generation, while satisfying a list of positive constraints. For more information, the documentation of [`ConstrainedBeamSearchScorer`] should be read. logits_processor (`LogitsProcessorList`): An instance of [`LogitsProcessorList`]. List of instances of class derived from [`LogitsProcessor`] used to modify the prediction scores of the language modeling head applied at each generation step. stopping_criteria (`StoppingCriteriaList`): An instance of [`StoppingCriteriaList`]. List of instances of class derived from [`StoppingCriteria`] used to tell if the generation loop should stop. generation_config ([`~generation.GenerationConfig`]): The generation configuration to be used as parametrization of the decoding method. synced_gpus (`bool`): Whether to continue running the while loop until max_length (needed to avoid deadlocking with `FullyShardedDataParallel` and DeepSpeed ZeRO Stage 3). model_kwargs: Additional model specific kwargs will be forwarded to the `forward` function of the model. If model is an encoder-decoder model the kwargs should include `encoder_outputs`. Return: [`~generation.GenerateBeamDecoderOnlyOutput`], [`~generation.GenerateBeamEncoderDecoderOutput`] or `torch.LongTensor`: A `torch.LongTensor` containing the generated tokens (default behaviour) or a [`~generation.GenerateBeamDecoderOnlyOutput`] if `model.config.is_encoder_decoder=False` and `return_dict_in_generate=True` or a [`~generation.GenerateBeamEncoderDecoderOutput`] if `model.config.is_encoder_decoder=True`. Nr rRrSrrrc3K|]}dVdSrNrrrOs rtr*z;GenerationMixin._constrained_beam_search..4s"55"555555rsrrgrhrrr"Fr~r/r0r1Trrrr:rrX)rrrrr^r_r`r]r3c3HK|]}||fzVdSrrr)rrbrHrs rtr*z;GenerationMixin._constrained_beam_search..s8$r$rTU\(1+%>(1+%O$r$r$r$r$r$rrsrcrdrdrr4r5)7rrr/r0rrrrrerzrrrrrprPrrrrmr9r=r<rrrrWrfrRrr rrrhrryrgrzr{rhrrrrjrArOrr3r?rkr?rlr`rr}),rrr~rYrr rcrrrr/r0rrrrrzrmrrerryrzr{rwrxrrrrrKrrrvscores_for_all_vocabrhrArrwryrzr|rHrs, @@rtrz(GenerationMixin._constrained_beam_searchsX): (: -?0E)7 )7 "3"K0;<< +5 #,?2A2#6 77AQS_`` z !_ 4 4oI OZ22VZ'> bCW b^b # t{'E Vg!q.?!@!D!D\!R!R!Rmq H\f ./33ODDDbf " k:y"9U^Uefff ! AAAqrrE!&& Y(>'@AA "&_Q/,,-?U^Ue,ffj *=4=iXX<XXL   L] e!46G H Hce f f f   Rf n!79M N Nln o o od<<\<R>\>\+??! $5#:#:#<#< '  3022F 7#4"66J$H&9=9Wr355^e^p]r&{5H(W-E,GG(');96688%35)+04J 1 6 6z9zCY Z Z 5A4L<-a00RSL-2Z!3q!l*:#;#;i#GQX\ei... * {(*4::<? #$78H 9Xqqq[#9;K;U;UVX;Y;Y"Z`bcccI  1488D4!122L6:6I6I,WhJiks6t6tL!233 !23AA(KKK& s= s$$r$r$r$r$rY^_bco_p_pYqYq$r$r$rrr kG&. *#6G6G SY6Z6Z2[2[ *%)"U,,-?U^Ue,ffj *X3;;    %%(3%1<    # 1  ;6: !23{- 7.{;%56G%H!%!1.!A'9*?'9%5*?$0$4$45F$G$G    5.{;%56G%H!%!1.!A1"7$0$4$45F$G$G    $K0 0rsrlc *+|j} |j} |j} |j} |j} |j}|r| rdnd}|r| rdnd}|r| rdnd}|r| rdnd}|r| rdnd}|rJ|jjr>| r|ddnd}| r|ddnd}|j dd\}}tj |tj |j }|||j |}d}d }||||j r|j d }||\}}||j }|||j }|j d |j d z }||d}t%j|}t'||j d |jj}t)||j d }d |vrItj|d tj|||z|j tj fd|d <|j|fi|} d| vr|d z| d<| | rd| ini| | rd| ini|di| }!|!jdd| d z dftj|j **+t9|dkrGt;|d zD]4}"||ddd||"zf*dd|"ddf*dd|"ddf<5| r|t=|||*|\}#}$n| rW*d}%tj |%dddddfd !d dddf}&n*"d}&|dd|df}'|'|&ddddfk#dd k$}$|r |$|kr|$d z}$|&ddd|$d zf}#tj||#fd}|'|%|#&|j d }(|!j'(|(d z |)|*|$|*|!||jj|$d z}|r|r|r|$d z})| r+|tW*fdt;|)Dz }| r+|tW+fdt;|)Dz }|r|(n|)})| rc|jjr1tY||!j-||)}tY||!j.||)d }n&|!j/dtY||!j/||)d }| r;|jjrtY||!j0||)}ntY||!j1||)}||||z}|2dk}d}||||j ||3ti|dr+|j5j6j7dkr|j8|j5j6_8|r_|jjr+ts|||||||||d Stu||||||dS|S)a Generates sequences of token ids for models with a language modeling head using **greedy decoding** or **sample** (depending on `do_sample`), assisted by candidate sequences. Assisted generation is an example of a candidate decoding strategy. Can be used for text-decoder, text-to-text, speech-to-text, and vision-to-text models. Parameters: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): The sequence used as a prompt for the generation. candidate_generator (`CandidateGenerator`): A derived instance of [`CandidateGenerator`] that defines how candidate sequences are generated. For more information, the documentation of [`CandidateGenerator`] should be read. logits_processor (`LogitsProcessorList`): An instance of [`LogitsProcessorList`]. List of instances of class derived from [`LogitsProcessor`] used to modify the prediction scores of the language modeling head applied at each generation step. stopping_criteria (`StoppingCriteriaList`): An instance of [`StoppingCriteriaList`]. List of instances of class derived from [`StoppingCriteria`] used to tell if the generation loop should stop. generation_config ([`~generation.GenerationConfig`]): The generation configuration to be used as parametrization of the decoding method. synced_gpus (`bool`): Whether to continue running the while loop until max_length (needed to avoid deadlocking with `FullyShardedDataParallel` and DeepSpeed ZeRO Stage 3). streamer (`BaseStreamer`, *optional*): Streamer object that will be used to stream the generated sequences. Generated tokens are passed through `streamer.put(token_ids)` and the streamer is responsible for any further processing. model_kwargs: Additional model specific keyword arguments will be forwarded to the `forward` function of the model. If model is an encoder-decoder model the kwargs should include `encoder_outputs`. Return: [`~generation.GenerateDecoderOnlyOutput`], [`~generation.GenerateEncoderDecoderOutput`] or `torch.LongTensor`: A `torch.LongTensor` containing the generated tokens (default behaviour) or a [`~generation.GenerateDecoderOnlyOutput`] if `model.config.is_encoder_decoder=False` and `return_dict_in_generate=True` or a [`~generation.GenerateEncoderDecoderOutput`] if `model.config.is_encoder_decoder=True`. rrNrrgrhr rFTr~r"rrGrr:rBr/r0rr)rrLc36K|]}dd|ddfVdSrrr)rrb new_logitss rtr*z5GenerationMixin._assisted_decoding..s6#[#[AJqqq!QQQw$7#[#[#[#[#[#[rsc36K|]}dd|ddfVdSrrr)rrbrs rtr*z5GenerationMixin._assisted_decoding..s7'f'fq(9!!!Q'(B'f'f'f'f'f'frs)is_decoder_attentionrX heuristicr]rr);rer/r0rrrrrrrrmrrrrrget_candidatesrRrr/r0rArQrrrfrrrrP_speculative_samplingrrrrrrrrVr]cropupdate_candidate_strategyrWrp_split_model_outputsrzryrgr{rhrrrrXr num_assistant_tokens_schedulenum_assistant_tokensrvrc),rrrlrYrr rcrdrrer/r0rrrrerryrzr{rwrxrrrris_first_iterationcandidate_input_idscandidate_logitscandidate_lengthis_done_candidatecandidate_kwargsrrKrb valid_tokens n_matchesrselected_tokenscandidate_new_tokens new_cur_lennewly_added_lengthrrs, @@rtrz"GenerationMixin._assisted_decodings7 b&/ -?0E)7 )7 "3"K0LML3P PRRD $;\@Q\RRX\"9Z>OZ22VZ'> bCW b^b # t{'E Vg!q.?!@!D!D\!R!R!Rmq H\f ./33ODDDbf " (obqb1 G$z*EJyO_```77AQS_`` "!,,-?U^Ue,ffX 'oa(G5H4V4VW`4a4a 1 !1"5"8"8"E"E +#3#6#6t{#C#C 28;ioa>PP  1 12Et L L  $y66 6 "5";A"> @^    77GI\IbcdIeff #3335:Y()9: Wg8H.HQZQainisttt 666 !12>4=>QffUeffL<//1AA1E -.   L] e!46G H Hce f f f   Rf n!79M N Nln o o od**\**G!,<+A(  A(A(A(A(F (# (  12 (!!12 ( u %"22 3 ( ( ( (D=(#=(  12=(!!12 =( u %"22 3 =(=(=(=(D,0595959 II#I"%I!!12 I   12 I !!12 IIIIZ*./3:> A0A0&A0u|,A0tC$567 A0 u|Xc]Del1B,CC D A0A0A0A0J*./3:> ``&`u|,`tC$567 `   ````@"|","38n "   """"H'|'#3- ' , ' c3h ''''^*. 9/9/9/9/3 ,- 9/ !& 9/ & 9/ uc5<&7!88 99/9/9/9/v#(04 ' ' '  'E,- ' uc3h/ 0 ' ' '\ 'L$) ///38n/! /  / c3h ////bN#+N##N#| N# + N# . N#4N#7N#N# N#N#N#N#f/3.2W[:> $156:AEcc+c'smc!+ c #+8S%,4Gc4R+S"T c ##67 c ctCH~.c&el3c)1(>c ccccR:> %%+%$$89%56 %  %%%%N#/1EEF#.0DDE# "$88 9 ####R04!& x!x!<x!el#x!u|, x!  x!  x!x!x!x!t////b***X6!6!6!rcga/a/!)*:!;a/QYZ^Q_a/rva/ % &a/a/a/a/F:,s,,TW,jo,,,,\    [ $~+~~+ ~  ~  ~ ~~~~@Z$ZZZZ5959 PSPS+PS$,D>PSu|S012 PSPSPSPSd''Rb'gk''''RU]__*.8<:><@W[&*7;-16:AE-1:>GG&G$$45G##67 G $$89 G #+8S%,4Gc4R+S"T Gd^G""34G>*G&el3G)1(>G%TNG"%X "67G ~u// 0GGG_GRDt]b]inr&]aSS)S6>?X6YS  SSSSjB#B.B0 B , B  B>*B $e&66 7BBBBHH%,H5<HHH\H JELJcJcJV[VbJJJ\J el%,5<\&, -2\, "\, \,  , ,  ,  ,  , dCi(, , , , \, \M-2\M,M,1<MdCi( MMM\M,4Q$|4Q!<4Q$l 4Q  4Q  4Q4Q4Q4Q4Q4Q u|U\5<7 84Q4Q4Q4QlL L!& L$)< L ,1< L  L u|U\5<7 8LLLL,3F<3F!& 3F\ 3F  3F l 3F$)<3F.3\3F ,3F,1<3F!<3F3F3F 3F3F dCi(!3F" u|U\5<E F#3F3F3F3FnN#N.N0 N , N  N !5#33 4NNNN` e1#e1 e1. e1 0 e1 , e1e1e1e1e1N n1#n1"=n1. n1 0 n1 , n1n1 !5#33 4n1n1n1n1`H#H0H. H 0 H , HH>*H $e&66 7HHHHT*5+;*P`******rsrc|dd| df}|d}|ddtj||fdd}|d}|ddtj||fdd} | |z } tj| } | | k} | ddk} |r| |kr| dz} |ddd| dzf}n|jd}|dd| ddf}| |krP|dd| ddf}tj||z d}| |n|}tj |dddddf}| dkr%tj |ddd| f|fd}n|}|| fS)a Applies sampling as in the speculative decoding paper (https://huggingface.co/papers/2211.17192, algorithm 1). Returns the selected tokens, as well as the number of candidate matches. NOTE: Unless otherwise stated, the variable names match those in the paper. Nrr:rr")rr) rrmrQr rand_likerrrclampdiv_rrA)rrrrrnew_candidate_input_idsqq_ir(p_iprobability_ratior_i is_acceptedrrgamma p_n_plus_1 q_n_plus_1p_primers rtrrsA2!!!6F5F5G5G2GH   R ((A AAAu|,--/F F G O OPQST U UCr""A AAAu|,--/F F G O OPQST U UCc  /+ , ,C**K,&&2&..27799IY*::: Q .qqq/IM//AB !&q)qqq)QQQ' u  111i?+Jk: #:CCCG LL ' ' ' ' G  g1 5 5 5 = =a @ @qqq I q== 9&=aaa)m&La%PVXYYYLLL  ""rsFc4t|dkr;d}|D]&}|r|n |jd}||dd|d|ffz }'||fz }|dz }||z}t|D]9}d}|D],}|r||zn |jd}||d||dzd|ffz }-||fz }:|S)z Given the (decoder/cross attentions)/(decoder hidden states) for multiple generated tokens, splits it into a tuple where each member corresponds to a single generated token. rrrr.Nr")rrrP) rK new_outputsr added_lenr new_tuplelayer last_dim_sizerbs rtrrPs 7||q   A AE';PGGRM %XgX~ ~ =>@ @III<1 W 9       B BE+?TGaKKU[QS_M %QQY >?A AIII< NrsrFrrcstdtdtfdDstddfdjD}di|S)z Stack a list of ModelOutput objects (or its subclasses) along the batch_size dimension. The function infers the specific ModelOutput subclass from the list provided. zInput list is empty.rc38K|]}t|VdSr)r)robjmodel_output_clss rtr*z&stack_model_outputs..vs.JJSz#/00JJJJJJrsz4All elements in the list should be of the same type.c tdDrdStdtjrtjdStdt rtddt r;t fdt tdDSt fdt tdDStdttfrtj Stdtd)z5 Reverse of `_split` function above. c3K|]}|duV dSrrr)rdatas rtr*z7stack_model_outputs.._concat..~s&--tt|------rsNrr:c 3K|]FtfdttddDVGdS)c3`K|]'tjfdDdV(dS)c,g|]}|Srrrr)rrrbjs rtrzLstack_model_outputs.._concat....s!$A$A$ADT!WQZ$A$A$Arsrr:NrmrA)rrrrbs @rtr*zAstack_model_outputs.._concat...sKjja%)$A$A$A$A$AD$A$A$AqIIIjjjjjjrsrN)rprPrrrbrs @rtr*z7stack_model_outputs.._concat..swjjjjjSXY\]abc]def]gYhYhSiSijjjjjrsc3^K|]&tjfdDdV'dS)c g|] }| Srrrr)rrrbs rtrzBstack_model_outputs.._concat...s'A'A'ADQ'A'A'Arsrr:Nrrs @rtr*z7stack_model_outputs.._concat..sHggaUY'A'A'A'AD'A'A'AqIIIggggggrszUnexpected attribute type: ) rrrmr rArprPrrr=rH TypeErrorrj)rs`rt_concatz$stack_model_outputs.._concatzs] ----- - - 4 d1gu| , , K9Tq))) ) Q ' ' K$q'!*e,, h"3tAw<<00 ggggSXY\]abc]dYeYeSfSfgggggg Q#u . . K<%% %I$tAw--IIJJ JrscBi|]fdDS)c0g|]}t|Srr)r)r model_outputrs rtrz2stack_model_outputs...s#OOOGL!,,OOOrsrr)rrrrFs @rtrz'stack_model_outputs..sK  77OOOOOOO P Prsrr)rrjr?__dataclass_fields__)rFrconcatenated_datarrs` @@rtstack_model_outputsrjs 1/000M!,-- JJJJMJJJ J JQOPPPKKK0!6   0 0/ 0 00rs)F)rrrr dataclassesrtypingrrrrrrmtorch.distributed distributedrhuggingface_hubr packagingr r cache_utilsr rrrrconfiguration_utilsrdynamic_module_utilsrrrrintegrations.deepspeedrintegrations.fsdpr masking_utilsr pytorch_utilsrtokenization_utilsrutilsrrrrr r!beam_constraintsr#r$ beam_searchr%r&r'rlr(r)r*r+r,r-r.r/r0r1r2r3r4r5continuous_batchingr6logits_processr7r8r9r:r;r<r=r>r?r@rArBrCrDrErFrGrHrIrJrKrLrMrNrOrPrrQrRrSrTrUrVrWmodeling_utilsrXtokenization_utils_baserY streamersrZ get_loggerriraccelerate.hooksr[r\rNrcrvr}rGreedySearchDecoderOnlyOutput"ContrastiveSearchDecoderOnlyOutputSampleDecoderOnlyOutput%ContrastiveSearchEncoderDecoderOutput GreedySearchEncoderDecoderOutputSampleEncoderDecoderOutputBeamSearchDecoderOnlyOutputBeamSampleDecoderOnlyOutputBeamSearchEncoderDecoderOutputBeamSampleEncoderDecoderOutputGreedySearchOutput SampleOutputBeamSearchOutputBeamSampleOutputContrastiveSearchOutputrrrrrrrrrrrsrtrsW   !!!!!!@@@@@@@@@@@@@@ ''''''322222 @?????666666555555------//////GFFFFFFFRRRRRRRRRR                      1000008(000000AAAAAA''''''  H % %FEEEEEEEE  M M M M M M M  MF ,M,M,M,M,M;,M,M ,M^ (M(M(M(M(MK(M(M (MV 5M5M5M5M5M{5M5M 5Mt!:%>"3(D%#? 9;;!A!A;=ZZ[/1HHI 79TTU79TTU EGi ij79UUV8:ZZ[,.@@Aw>w>w>w>w>ow>w>w>t}5#5#5#p4.1tK'8.1BR.1Wb.1.1.1.1.1.1rs