o wiv@sdZddlmZddlmZmZmZmZddlZddl m m Z ddl Zddlm Z ddlmZddlmZmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZmZm Z ddl!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.m/Z/e'rddl0m1Z1ddl2m3Z3e(4e5Z6ee%ddGdddeZ7ee%ddGdddeZ8    dJdd Z9gfd!d"Z:Gd#d$d$e j;ZGd'd(d(e j?Z@Gd)d*d*ej j?ZAd+d,ZBdKd-d.ZCGd/d0d0e j?ZD 1dLd2e j?d3ejEd4ejEd5ejEd6eejEd7eFd8eFfd9d:ZGGd;d<dd>eZIGd?d@d@eZJe%GdAdBdBe ZKGdCdDdDee$ZLe%GdEdFdFeKZMGdGdHdHeKeZNgdIZOdS)MzPyTorch Idefics model.) dataclass)AnyCallableOptionalUnionN)nn)ACT2FN)Cache DynamicCache)GenerationMixin)AttentionMaskConverter)FlashAttentionKwargs)GradientCheckpointingLayer) ModelOutput)ALL_ATTENTION_FUNCTIONSPretrainedConfigPreTrainedModel)Unpack) LossKwargsauto_docstringcan_return_tupleis_torch_flex_attn_availablelogging) IdeficsConfig)IdeficsPerceiverResampler)IdeficsVisionEmbeddingsIdeficsVisionTransformer) BlockMask)make_flex_block_causal_maskz{ Base class for Idefics model's outputs that may also contain a past key/values (to speed up sequential decoding). ) custom_introc@seZdZUdZdZeejed<dZ ee e ejed<dZ ee ejed<dZ ee ejed<dZ ee ejed<dS)IdeficsBaseModelOutputWithPasta last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. If `past_key_values` is used only the last hidden-state of the sequences of shape `(batch_size, 1, hidden_size)` is output. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and optionally if `config.is_encoder_decoder=True` 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if `config.is_encoder_decoder=True` in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`tuple(torch.FloatTensor)`, *optional*): Tuple of `torch.FloatTensor` (one for the output of the image embeddings, `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder, and optionally by the perceiver Nlast_hidden_statepast_key_values hidden_states attentionsimage_hidden_states)__name__ __module__ __qualname____doc__r#rtorch FloatTensor__annotations__r$tupler%r&r'r0r0i/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/transformers/models/idefics/modeling_idefics.pyr"6s r"zS Base class for Idefics causal language model (or autoregressive) outputs. c@seZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejed<dZee ejed<dZee ejed<dS) IdeficsCausalLMOutputWithPasta loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`tuple(torch.FloatTensor)`, *optional*): Tuple of `torch.FloatTensor` (one for the output of the image embeddings, `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder, and optionally by the perceiver Nlosslogitsr$r%r&r')r(r)r*r+r3rr,r-r.r4r$listr%r/r&r'r0r0r0r1r2Zs r2FcKsJt|jdddd|d|j}|d|}|dd|d<|dd|d<|dd|d<|dd|d<d|vrM|d}|d||d<|durY|d||d <|dduri|dd||d<|ddur}|dd||d<||fS|ddur|dd||d<||fS|ddur|dd||d<||fS) Nrr pixel_valuesimage_encoder_embeddingsperceiver_embeddingsimage_attention_masktoken_type_idsattention_mask) r,arangeshapeviewrepeattodevice index_selectget) input_ids expand_sizeis_encoder_decoderr<encoder_outputs model_kwargsexpanded_return_idxr;r0r0r1expand_inputs_for_generation{s:,      rKcsftjtjtjdfdd|D}|D]|r+tfdd|Dr+dqdq|S)N) LayerNormLinear Embeddingcsg|]}|qSr0r0).0m)mappingr0r1 z freeze_model..c3s|]}t|VqdSN) isinstance)rOt)moduler0r1 szfreeze_model..TF)rrLrMrNmodulesanyrequires_grad_)modelmodule_exceptionsmodule_exceptions_mappedr0)rQrWr1 freeze_models   r_csNeZdZdZ    d deeddffdd Zdd Zdefd d Z Z S) IdeficsDecoupledEmbeddinga Implements a decoupling of parameters to allow freezing (or not) a subset of the embeddings. In practise, the regular `weight` can be trained or frozen (i.e. `partially_freeze=True`), and if `num_additional_embeddings` > 0, then it will create `num_additional_embeddings` additional parameters that are always trained. If `num_additional_embeddings=0`, then the module defaults back to the regular behavior of `nn.Embedding`. FNpartially_freezereturnc  s|dur||krtd|d|tjd|||||d|||_||_||_||_|r5|jd|jdkrGt j |j|||d|_ dSdS) a) Args: num_embeddings (`int`): Size of the dictionary of embeddings num_additional_embeddings (`int`): Number of additional embeddings. Only useful when you `partially_freeze=True`. embedding_dim (`int`): The size of each embedding vector partially_freeze: (`bool`, *optional*, defaults to `False`): If `True`, the regular `weight` will be frozen. `additional_weight` is never frozen. padding_idx (`int`, *optional*): The padding index (needs to be less than num_embeddings) Note: there are a lot of other parameters to initialize a standard `nn.Embedding` such as `padding_idx`, `max_norm` or `norm_type`. We are not supporting these. Nz/padding_idx must be within num_embeddings. Got z and )num_embeddings embedding_dimrBdtype padding_idxFr)rcrdrBrer0) ValueErrorsuper__init__rcrfnum_additional_embeddingsraweightr[rrNadditional_embedding) selfrcrjrdrarBrerfkwargs __class__r0r1ris2   z"IdeficsDecoupledEmbedding.__init__cCsj|jdkr t||jS|}t||jk}||}|||j}d||<t||j}|||<|S)a we have 2 embeddings, with different indices - one pretrained self.weight and another self.additional_embedding.weight that is being trained. in order to make a lookup of the input ids, we: 1. find out the indices of the entries belonging to the 2nd embedding 2. extract those values while subtracting the size of the first embedding (num_embeddings), since the 2nd embedding starts from 0 and not num_embeddings 3. perform the 2nd embedding lookup 4. now we handle the 1st embedding, we overwrite indices belonging to the 2nd embedding with a padding index 5. perform the 1st embedding lookup 6. now we overwrite the values in the 1st embedding lookup with the values of the 2nd embedding lookup note: for the 1st embedding lookup we could have looked up only the low indices and not do the padding, but then we have to create a new tensor and populate it with 2 tensors that are spread out across various indices - i.e. not a simple concat - I haven't benchmarked the complex case if it's any faster, given that seqlens are usually relatively short it's probably not faster or if faster not by much - but might be a good idea to measure. r) rjF embeddingrkcloner,wherercrl)rmrEadditional_vocab_indicesinput_ids_additional_vocabadditional_embeddings full_vectorr0r0r1forwards z!IdeficsDecoupledEmbedding.forwardcCs$d|jd|jd|jd|jS)Nznum_embeddings=z, num_additional_embeddings=z, embedding_dim=, partially_freeze=)rcrjrdrarmr0r0r1 extra_reprs$z$IdeficsDecoupledEmbedding.extra_repr)FNNN) r(r)r*r+rboolrirystrr| __classcell__r0r0ror1r`s  5'r`c sjeZdZdZ     ddedededed ed df fd d Zd ejd ejfddZ d e fddZ Z S)IdeficsDecoupledLineara Implements a decoupling of parameters to allow freezing (or not) a subset of the parameters. In practise, the regular `weight` can be trained or frozen (i.e. `partially_freeze=True`), and if `out_additional_features` > 0, then it will create `out_additional_features * in_features` additional parameters that are always trained. If `out_additional_features=0`, then the module defaults back to the regular behavior of `nn.Linear`. rTN in_features out_featuresout_additional_featuresbiasrarbcsrt|||||||_||_||_||_|r&|jd|r&|jd|dkr7t j |||||d|_ dSdS)aG out_additional_features: int. Number of additional trainable dimensions. Only makes sense when `partially_freeze=True`. partially_freeze: bool. If True, the regular `weight` will be frozen and extra parameters (if any) will be trainable. If False, default to the regular behavior of nn.Linear. Fr)rrrrBreN) rhrirrarrrkr[rrrM additional_fc)rmrrrrrarBreror0r1ri)s$   zIdeficsDecoupledLinear.__init__inputcCs:t||j|j}|jdkr||}t||fd}|S)Nrr6)rqlinearrkrrrr,cat)rmroutputadditional_featuresr0r0r1ryMs   zIdeficsDecoupledLinear.forwardc Cs0d|jd|jd|jd|jdud|j S)z=Overwriting `nn.Linear.extra_repr` to include new parameters.z in_features=z, out_features=z, out_additional_features=z, bias=Nrzrrrrrar{r0r0r1r|Vs0z!IdeficsDecoupledLinear.extra_repr)rTTNN) r(r)r*r+intr}rir,Tensorryr~r|rr0r0ror1r s,  $ rcs.eZdZdfdd ZddZddZZS) IdeficsRMSNormư>cs&ttt||_||_dS)z= IdeficsRMSNorm is equivalent to T5LayerNorm N)rhrir Parameterr,onesrkvariance_epsilon)rm hidden_sizeepsror0r1ri]s  zIdeficsRMSNorm.__init__cCs\|tjdjddd}|t||j}|jjtj tj fvr)||jj}|j|S)Nr6T)keepdim) rAr,float32powmeanrsqrtrrkrefloat16bfloat16)rmr%variancer0r0r1ryes  zIdeficsRMSNorm.forwardcCst|jjd|jS)Nz, eps=)r/rkr>rr{r0r0r1r|oszIdeficsRMSNorm.extra_repr)r)r(r)r*riryr|rr0r0ror1r\s rcs0eZdZd fdd ZddZd dd ZZS) IdeficsEmbedding'Ncszt||_||_||_d|jtjd|jdtjdj|tj d|j}|j d|dd|j ||j j td dS) N?rrrerBreinv_freqF persistentseq_lenrBre)rhridimmax_position_embeddingsbaser,r=int64rAfloatregister_buffer_set_cos_sin_cacherrBget_default_dtype)rmrrrrBrror0r1rius & zIdeficsEmbedding.__init__cCs|||_tj|j|tjd|j}td||j}tj||fdd}|jd| |dd|jd| |dddS) Nrzi,j->ijr6r cos_cachedFr sin_cached) max_seq_len_cachedr,r=rtype_asreinsumrrcosrAsin)rmrrBrerVfreqsembr0r0r1rs z#IdeficsEmbedding._set_cos_sin_cachecCsN||jkr|j||j|jd|jd|j|jd|jd|j|jdfS)Nrr)rrrBrerrAr)rmxrr0r0r1rys zIdeficsEmbedding.forward)rrNrT)r(r)r*rirryrr0r0ror1rts rcCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..Nr6rr)r>r,r)rx1x2r0r0r1 rotate_halfsrcCsL|||}|||}||t||}||t||}||fS)anApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`): The position indices of the tokens corresponding to the query and key tensors. For example, this can be used to pass offsetted position ids when working with a KV-cache. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer)qkrr position_ids unsqueeze_dimq_embedk_embedr0r0r1apply_rotary_pos_embs rcs2eZdZdededeffdd ZddZZS) IdeficsMLPrintermediate_size hidden_actcsNttj||dd|_tj||dd|_tj||dd|_t||_dS)NFr) rhrirrM gate_proj down_projup_projr act_fn)rmrrrror0r1ris zIdeficsMLP.__init__cCs ||||||SrT)rrrr)rmrr0r0r1ry zIdeficsMLP.forward)r(r)r*rr~riryrr0r0ror1rs rrWquerykeyvaluer<scalingdropoutc Ks|t||dd|}|dur||}tjj|dtjd|j}tjj |||j d}t||} | dd } | |fS)Nr6)rreptrainingrr) r,matmul transposer functionalsoftmaxrrArerr contiguous) rWrrrr<rrrn attn_weights attn_outputr0r0r1eager_attention_forwards  rcseZdZdZ     ddedededed ed ed eeffd d Z de j dedefddZ       dde j dee j dee j dee j deee j dededee j dee j ee j eee j ffddZZS) IdeficsAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrFNr num_headsris_cross_attentionconfigqk_layer_norms layer_idxc st||_||_||_|||_||_d|_|jd|_||_ |dur1t d|j j d|j||jkrEtd|jd|d||_ttjdsRtd |jrt|jd s^|jn|jj}tj|j||jd d |_tj|||jd d |_tj|||jd d |_n'tj|j||jd d |_tj|j||jd d |_tj|j||jd d |_tj||j|d d |_t|j|_||_|jrt|j|jd |_t|j|jd |_ dSdS)NTgzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.z?hidden_size must be divisible by num_heads (got `hidden_size`: z and `num_heads`: z).scaled_dot_product_attentionz)this model requires pytorch 2.0 or higher embed_dimFrr)!rhrirrrhead_dimr is_causalrrlogger warning_oncerpr(rgrhasattrrr vision_configrrMq_projk_projv_projo_projr rotary_embrr rms_norm_eps q_layer_norm k_layer_norm) rmrrrrrrr kv_input_dimror0r1ris|       zIdeficsAttention.__init__tensorrbszcCs ||||j|jddS)Nrr)r?rrrr)rmrrrr0r0r1_shape=rzIdeficsAttention._shaper%key_value_statesr<rpast_key_valueoutput_attentions use_cachecache_positionrbc Ks|jp|du} |\} } } ||| | |j|jdd}| sD||| | |j|jdd}||| | |j|jdd}n)|\} }} ||| ||j|jdd}||| ||j|jdd}|j d}|dur|||d7}| s|j |t || d\}}t |||||\}}|durd|i}| |||j|\}}|jr||}||}t}|jjdkr|jjdkr|rtd nt|jj}||||||f|jsd n|j|jd | \}}|| | d }||}|rd}|||fS) Nrrrr)rreagersdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.r)rrr6)rsizerr?rrrrrr>rmaxrupdaterrrrrr_attn_implementationrrrrrrreshaperr)rmr%rr<rrrrrrnrrq_len_ query_states key_states value_stateskv_len kv_seq_lenrr cache_kwargsattention_interfacerrr0r0r1ry@s\ ""$"          zIdeficsAttention.forward)rFNFNNNNNFFN)r(r)r*r+rrr}rrrir,rr LongTensorr/ryrr0r0ror1rsbQ   rcseZdZddedeeffdd Z      ddejdeejd eej d ee ejd ee d ee d eej de ej ee ej ej fffddZ ZS)IdeficsDecoderLayerNrrcsrt|j|_t|j|j|j||d|_t|j|j|j d|_ t |j|j d|_ t |j|j d|_|j|_dS)N)rrrrrrrrr)rhrirrnum_attention_headsr self_attnrrrmlprrinput_layernormpost_attention_layernormrmrrror0r1ris"  zIdeficsDecoderLayer.__init__Fr%r<rrrrrrbc Ks|} ||}|jd|||||||d|\}} } tjj||j|jd}| |}|} ||}||}tjj||j|jd}| |}|f} |rN| | f7} |rU| | f7} | S)a] Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states )r%r<rrrrrrNr0)rrrrrrrr) rmr%r<rrrrrrnresidualself_attn_weightspresent_key_valueoutputsr0r0r1rys4       zIdeficsDecoderLayer.forwardrT)NNNFFN)r(r)r*rrrrir,rrr/r}r-ryrr0r0ror1rs2  rcseZdZddedeeffdd Z       ddejdeejd eejd eejd eejd ee d ee dee ejde ej ee ej ej fffddZ Z S)IdeficsGatedCrossAttentionLayerNrrc s~t|j|_t|j|jd|j||j|d|_t|j|j |j d|_ t |j|j d|_t |j|j d|_|j|_t|_t|_|jdkr|jdkrgttdd|j|_ttdd|j|_n|jdkrttd|_ttd|_ntd |jd |jd kr|jdkrttdd|j|_ttdd|j|_n|jdkrttd|_ttd|_njtd |jd |jd vr$|jdkrttjd |jdd|jfd|_ttjd |jdd|jfd|_n3|jdkrttjd |jdd|_ttjd |jdd|_ntd |jd t d|jdt!|dr9t!|ds=tddS)NT)rrrrrrrrrzerosvectorrrz Unknown value for `alpha_type` ()r>normalrandomgaussianr)rstdrzAlpha initialization scheme z not yet implemented!alpha_cross_attn alpha_densez+Alpha parameters not initialized correctly!)"rhrirrrrr cross_attnrrrrrrrrrrTanhact_cross_attn act_densealpha_initializer alpha_typerr,r!r(r)rgrr$alphas_initializer_rangeNotImplementedErrorrrror0r1risn             z(IdeficsGatedCrossAttentionLayer.__init__Fr%r<r'r:cross_attention_gaterrrrbc Ks|durtd|durtd|durtd|} ||}|jd ||||d| \}} } tjj||j|jd}| |dkdddddfd}| | |j |}|} | |}| |}tjj||j|jd}| ||j|}|f} |r| | f7} |r| | f7} | S) a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. image_attention_mask (`torch.FloatTensor`, *optional*): image attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. cross_attention_gate (`torch.FloatTensor`, *optional*): gate of size `(batch, seq_len)` used to zero-out cross-attention output for tokens attending no images. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states Nzt`image_hidden_states` is required for Idefics cross attention module which are visual features to be conditioned on.z`cross_attention_gate` is required for Idefics cross attention module to zero-out the cross-attention hidden_states attending to no images.zMPast key value states are not implemented for Idefics cross attention module.)r%rr<rrrrr0)rgr1rr*rrrrr masked_fillr,r(rrr-r))rmr%r<r'r:r2rrrrnrrrrr0r0r1ry"sD  "    z'IdeficsGatedCrossAttentionLayer.forwardrTr)r(r)r*rrrrir,rr}r/r-ryrr0r0ror1r s8E  r c@s<eZdZeZdZdZddgZdZdZ dZ dZ dZ ddZ dS) IdeficsPreTrainedModelr\Trr FcCs|jj}t|tjtjfr%|jjjd|d|j dur#|j j dSdSt|tj rF|jjjd|d|j durD|jj|j  dSdSt|tj r[|jjd|j j dSt|tri|jjddSt|trv|jjdSt|tr|jjdkr|jj |jj dS|jjdkr|jjd|jjddS|jjdvr|jjjd|jjd|jjjd|jjddSdSt|tr|jjdSdS)Nr)rr'rr!r>r$r%r&)rinitializer_rangerUrrMConv2drkdatanormal_rzero_rNrfrLfill_rrclass_embeddingr r.r(r)r0rlatents)rmrWr'r0r0r1 _init_weights|s@            z$IdeficsPreTrainedModel._init_weightsN)r(r)r*r config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_cache_class_supports_flash_attn_2_supports_static_cache_supports_attention_backendr=r0r0r0r1r4ps r4c@s eZdZdS)KwargsForCausalLMN)r(r)r*r0r0r0r1rGs rGc'seZdZdZdeffdd Zd.ddZgfdd Zgfd d Zd d Z ddZ e e               d/de ejde ejde ejde eejde ejde ejde ejde ejde ejde ede ede ede ede ede ejd eed!eeeff"d"d#Z d0deejd$fd%ejdejdedef d&d'Zedejd(ed)ed*ejdejd+ef d,d-ZZ S)1 IdeficsModelz Transformer decoder consisting of `config.num_hidden_layers` layers. Each layer is a [`IdeficsDecoderLayer`] Args: config: IdeficsConfig rcs t|_j|_j|_tjjjj |jd|_ j j |_ j |_ t j |_jrEj}tj j|j|j|j|j|_tfddtjD|_j|_j|j}tfddt|D|_d|_t jj!d|_"|#|$dS)N)rcrjrdrarfcg|]}t|dqS)r)rrOirr0r1rRz)IdeficsModel.__init__..crIrJ)r rKrMr0r1rRrNFr)%rhrir pad_token_idrf vocab_sizer`additional_vocab_sizerfreeze_text_layers embed_tokensr image_sizer vision_model use_resamplerperceiver_configrrresampler_depthresampler_n_headsresampler_head_dimresampler_n_latentsperceiver_resamplerr ModuleListrangenum_hidden_layerslayerscross_layer_intervalgated_cross_attn_layersgradient_checkpointingrrnorm post_initfreeze_relevant_params)rmrrWnum_cross_layersrorMr1risH     zIdeficsModel.__init__NcCs>|dur|j}|jr||j|jrt|j|jddSdSN)r])rrRfreeze_text_module_exceptionsfreeze_vision_layersr_rUfreeze_vision_module_exceptions)rmrr0r0r1rfs z#IdeficsModel.freeze_relevant_paramscCs"|j|jfD]}t||dqdSrh)r`rdr_)rmr]rWr0r0r1rRszIdeficsModel.freeze_text_layerscCst|j|ddSrh)r_rU)rmr]r0r0r1rjsz!IdeficsModel.freeze_vision_layerscC|jSrTrSr{r0r0r1get_input_embeddingsz!IdeficsModel.get_input_embeddingscC ||_dSrTrmrmrr0r0r1set_input_embeddings z!IdeficsModel.set_input_embeddingsFrEr<rr$ inputs_embedsr7r8r9r:rroutput_hidden_statesinterpolate_pos_encoding return_dictrrnrbc+ Ks|dur|jn|j}| dur| n|jj} | dur| n|jj} | dur$| n|jj} |dur.|n|jj}|du|duAr>td|jrM|jrM| rMt dd} |durV| |}d}| rst |t ssd}|durit}n t|}t d|j\}}}|dur|nd}||}|durtj|||jd|jd }|dur|dur|d d}||dkd|dd| df}n |dur|d}td d |||fDd krtd|dur |j|j|d}|jdd \}}|j||g|jd dR}|j|| dj}n|dur(|\}}}}|j|j|d}|||||}|jj rN|durC|!|}|d|d }}n|\}}}}|}n|dur_|d|d }}ntd|||||}| d}| d } | "ddd|} | ||||} |dur|\}}}||f}| durtj#||d } |$| } nd} | dkj%d dj|jdj&dd|} |durtj#||ftj'|jd}|(||||| }|}!| rdnd}"| rdnd}#d}$t)|j*D]Z\}%}&| r|"|!f7}"|%|j+dkr |j,|%|j+}'|'|!||f| | | | dd|}(|(d}!|&|!f|||| | |d|})|)d}!| r?|)| r.rz_Exactly 1 of pixel_values, image_encoder_embeddings or perceiver_embeddings has to be not-None.)rerB)r7rvzBIf `perceiver_embeddings` are passed, use_resampler should be Truerrrr0)r:r2rrr)r<rrrrr)r#r$r%r&r')0rBrrruruse_return_dictrgrcrrrrSrUr r from_legacy_cacher>get_seq_lengthr,r=longcumsum masked_fill_rsumrArerr?rUr#rrVr\r@rinvert_attention_maskrZsqueezer}_update_causal_mask enumerater`rarbrdto_legacy_cacher")+rmrEr<rr$rtr7r8r9r:rrrurvrwrrnrBreturn_legacy_cache batch_size seq_lengthr past_key_values_lengthseq_length_with_past num_imagesr' image_seq_lenimage_hidden_size text_seq_lenimage_batch_sizeimage_sequence_lengthimage_hidden_shaper2r%all_hidden_statesall_self_attnsnext_decoder_cacheidx decoder_layercross_attn_blockr layer_outputs next_cacher0r0r1rys     $          "        zIdeficsModel.forwardr input_tensorc Cs:|jjdkr|dur|dkr|SdS|jjdkr&t|tjr$t|}|S|dur.|nd}|dur7|jnd}|jjdkrO|sO|sOt j ||||j drOdS|j }|j d} |r^|} nt|tjri|j d n|| d} |j|| | |||j dd } |jjdkr|dur|jjd vr|st|j} t | | } | S) Nflash_attention_2rflex_attentionrFr)rtr is_trainingrr6)sequence_length target_lengthrerr)cudaxpunpu)rrrZrUr,rr r{is_compileabler _ignore_causal_mask_sdparrer>get_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionrBtypefinfomin_unmask_unattended) rmr<rrr$rpast_seen_tokensusing_compilable_cachererr causal_mask min_dtyper0r0r1rsT           z IdeficsModel._update_causal_maskrrrerc KsD|dur|dkr|}|St|j}tj||f|||jd}|dkr+tj|dd}|tj||jd|ddk9}|ddddddf |ddd}|dur| }|j d} |ddddddd| f|ddddddf |j} | dk} |ddddddd| f | ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N) fill_valuererBr)diagonalrxr6r)rr,rrfullrBtriur=rexpandrsr>rAr3) r<rrrerrrnrr mask_length padding_maskr0r0r1r s,  $ 6  zBIdeficsModel._prepare_4d_causal_attention_mask_with_cache_positionrT)NNNNNNNNNNNNFNNF)!r(r)r*r+rrirfrRrjrnrrrrrr,rrr5r-r}rrrr/r"ryr r staticmethodrrerrr0r0ror1rHs 0        ]  DrHc)seZdZddgZd2fdd ZddZdd Zd d Zd d ZddZ ddZ ddZ e e                d3deejdeejdeejdeeejdeejdeejdeejdeejdeejdeejdeed eed!eed"eed#eed$eejd%eed&eeeff$d'd(Z         d4fd)d* Z d5d+ed,eeefd-ed&eeefffd.d/ Z e!d0d1Z"Z#S)6IdeficsForVisionText2Textzmodel.embed_tokens.weightzlm_head.weightNcs>t|t||_t|j|j|jd|jd|_ | dS)NFr) rhrirHr\rrrPrQfreeze_lm_headlm_headre)rmrrUror0r1riIs   z"IdeficsForVisionText2Text.__init__cCs|jjSrTr\rSr{r0r0r1rnXsz.IdeficsForVisionText2Text.get_input_embeddingscCs ||j_dSrTrrqr0r0r1rr[s z.IdeficsForVisionText2Text.set_input_embeddingscCrlrTrr{r0r0r1get_output_embeddings^roz/IdeficsForVisionText2Text.get_output_embeddingscCrprTr)rmnew_embeddingsr0r0r1set_output_embeddingsarsz/IdeficsForVisionText2Text.set_output_embeddingscCrprTr\)rmdecoderr0r0r1 set_decoderdrsz%IdeficsForVisionText2Text.set_decodercCrlrTrr{r0r0r1 get_decodergroz%IdeficsForVisionText2Text.get_decodercCs|}|}t|jddr&|j|_|jdkr&|j|jks J|jj|j_t |drDt |drF|j |_ t |drHt |drJ|j|_dSdSdSdSdS) z Overwrite `transformers.modeling_utils.PreTrainedModel.tie_weights` to handle the case of IdeficsDecoupledLinear and IdeficsDecoupledEmbedding. tie_word_embeddingsTrrrcrrjN) rrngetattrrrkrjrrlrrrcr)rmoutput_embeddingsinput_embeddingsr0r0r1 tie_weightsjs    z%IdeficsForVisionText2Text.tie_weightsFrEr<rr$rtr7r8r9r:labelsrrrurvrwrrnrbcKs| dur| n|jj} | dur| n|jj} |dur|n|jj}|jd||||||||| | | | |d|d|}|d}||}d}| durS|jd|| |jjd|}t|||j |j |j |j dS)aC image_encoder_embeddings (`torch.FloatTensor`, *optional*): The output of the image encoder. perceiver_embeddings (`torch.FloatTensor`, *optional*): The output of the perceiver resampler. image_attention_mask (`torch.LongTensor`, *optional*): The attention mask for the image encoder. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoProcessor, IdeficsForVisionText2Text >>> model = IdeficsForVisionText2Text.from_pretrained("HuggingFaceM4/idefics-9b") >>> processor = AutoProcessor.from_pretrained("HuggingFaceM4/idefics-9b") >>> dogs_image_url_1 = "https://huggingface.co/datasets/hf-internal-testing/fixtures_nlvr2/raw/main/image1.jpeg" >>> dogs_image_url_2 = "https://huggingface.co/datasets/hf-internal-testing/fixtures_nlvr2/raw/main/image2.jpeg" >>> prompts = [ ... [ ... "User:", ... dogs_image_url_1, ... "Describe this image.\nAssistant: An image of two dogs.\n", ... "User:", ... dogs_image_url_2, ... "Describe this image.\nAssistant:", ... ] ... ] >>> inputs = processor(prompts, return_tensors="pt") >>> generate_ids = model.generate(**inputs, max_new_tokens=6) >>> processor.batch_decode(generate_ids, skip_special_tokens=True) ```NT)rEr<rr$rtr7r8r9r:rrrurvrwrr)r4rrP)r3r4r$r%r&r'r0) rrruryr\r loss_functionrPr2r$r%r&r')rmrEr<rr$rtr7r8r9r:rrrrurvrwrrnrr%r4r3r0r0r1rysH< z!IdeficsForVisionText2Text.forwardc  si} |dur|jjr|| d<n || d<n|| d<| dd| d<tj|f|||||| | d| | } | durP|durP| djd}| dd| df| d <| S) Nr9r8r7rvF)r$r<rtrrrr:rErr:)rrVpoprhprepare_inputs_for_generationr>)rmrEr<rrtr$rr7r'r:rrn images_kwargs model_inputsrror0r1rs4     z7IdeficsForVisionText2Text.prepare_inputs_for_generationrrIrGc s~tj|||fi|}d|vr8|d}|dddddfd}|ddr-||d<n tj||gdd|d<|j|d<|S)Nr:r6rrTrr')rh#_update_model_kwargs_for_generationrrDr,rr')rmrrIrGrnr: last_maskror0r1rs   z=IdeficsForVisionText2Text._update_model_kwargs_for_generationcs.d}|D]}|tfdd|Df7}q|S)Nr0c3s|] }|dVqdS)rN)rC)rO past_statebeam_idxr0r1rX0sz;IdeficsForVisionText2Text._reorder_cache..)r/)pastrreordered_past layer_pastr0rr1_reorder_cache,sz(IdeficsForVisionText2Text._reorder_cacherT)NNNNNNNNNNNNNFNN) NNNNNNNNNr)$r(r)r*_tied_weights_keysrirnrrrrrrrrrrr,rrr5r-r}rrGrr/r2ryrrdictr~rrrrrr0r0ror1rFs       g1  r)rrHr4)rFNN)r)r)Pr+ dataclassesrtypingrrrrr,torch.nn.functionalrrrqtorch.utils.checkpoint activationsr cache_utilsr r generationr modeling_attn_mask_utilsr modeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrmodeling_utilsrrrprocessing_utilsrutilsrrrrrconfiguration_ideficsr perceiverrvisionrr!torch.nn.attention.flex_attentionrintegrations.flex_attentionr get_loggerr(rr"r2rKr_rNr`rMrModulerrrrrrrrrrr r4rGrHr__all__r0r0r0r1s               -i<'  &Q-' o