o iD@sdZddlZddlmZddlmZmZmZmZddl Z ddl m Z ddl m Z ddl mZdd lmZdd lmZdd lmZmZmZmZdd lmZmZdd lmZddlmZmZm Z ddl!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'ddl(m)Z)m*Z*ddl+m,Z,m-Z-m.Z.ddl/m0Z0m1Z1m2Z2e&3e4Z5ee$ddGddde"Z6Gddde j7Z8 dJde j7de j9de j9de j9d ee j9d!e:d"e:fd#d$Z;Gd%d&d&e j7Ze$Gd+d,d,eZ?Gd-d.d.e j7Z@Gd/d0d0e?ZAGd1d2d2e j7ZBGd3d4d4e j7ZCGd5d6d6e j7ZDGd7d8d8e j7ZEGd9d:d:e j7ZFGd;d<dd>e j7ZHGd?d@d@e j7ZIGdAdBdBe?ZJe$dCdGdDdEdEe?ZKe$dFdGdGdHdHe?eZLgdIZMdS)KzPyTorch InstructBLIP model.N) dataclass)AnyCallableOptionalUnion)nn)ACT2FN)GenerationMixin)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsBaseModelOutputWithPooling,BaseModelOutputWithPoolingAndCrossAttentions)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging torch_int)OutputRecordercheck_model_inputs) AutoModelAutoModelForCausalLMAutoModelForSeq2SeqLM)InstructBlipConfigInstructBlipQFormerConfigInstructBlipVisionConfigzQ Class defining the outputs of [`InstructBlipForConditionalGeneration`]. ) custom_introc@seZdZUdZdZeeeje d<dZ eeeje d<dZ eeje d<dZ eeeje d<dZ eeeje d<deefd d ZdS) /InstructBlipForConditionalGenerationModelOutputa loss (`torch.FloatTensor`, *optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Language modeling loss from the language model. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head of the language model. vision_outputs (`BaseModelOutputWithPooling`): Outputs of the vision encoder. qformer_outputs (`BaseModelOutputWithPoolingAndCrossAttentions`): Outputs of the Q-Former (Querying Transformer). language_model_outputs (`CausalLMOutputWithPast` or `Seq2SeqLMOutput`): Outputs of the language model. Nlosslogitsvision_outputsqformer_outputslanguage_model_outputsreturncstfddDS)Nc3s.|]}|dvr |nt|VqdS)r+r,r-N)getattrto_tuple).0kselfs/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/transformers/models/instructblip/modeling_instructblip.py Js  zKInstructBlipForConditionalGenerationModelOutput.to_tuple..)tuplekeysr4r6r4r7r1Is z8InstructBlipForConditionalGenerationModelOutput.to_tuple)__name__ __module__ __qualname____doc__r)rr9torch FloatTensor__annotations__r*r+r,r-rr1r6r6r6r7r(.s  r(cs\eZdZdeffdd Zdejdededejfdd Zdd ej d e dejfd dZ Z S)InstructBlipVisionEmbeddingsconfigcst||_|j|_|j|_|j|_tt dd|j|_ tj d|j|j|jd|_ |j|jd|_|jd|_tt d|j|j|_dS)Nr#r) in_channels out_channels kernel_sizestrider)super__init__rC hidden_size embed_dim image_size patch_sizer Parameterr?randnclass_embeddingConv2dpatch_embedding num_patches num_positionsposition_embeddingr5rC __class__r6r7rITs  z%InstructBlipVisionEmbeddings.__init__ embeddingsheightwidthr.c Cs|jdd}|jjdd}tjs||kr||kr|jS|jddddf}|jddddf}|jd}||j} ||j} t|d} |d| | |}|dddd}t j j || | fdd d }|dddd dd|}tj ||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 r#Ng?rrrbicubicF)sizemode align_cornersdim)shaperUr?jit is_tracingrMrreshapepermuter functional interpolateviewcat) r5rYrZr[rSrTclass_pos_embedpatch_pos_embedrb new_height new_widthsqrt_num_positionsr6r6r7interpolate_pos_encodingfs(     z5InstructBlipVisionEmbeddings.interpolate_pos_encodingF pixel_valuesrqc Cs|j\}}}}|jjj}||j|d}|ddd}|j|dd|} t j | |gdd} |r<| | ||} n|j } | | ddd| dddf|} | S)Ndtyperr#r\ra)rcrRweightrttoflatten transposerPexpandr?rkrqrUr^) r5rrrq batch_size_rZr[ target_dtype patch_embeds class_embedsrYrUr6r6r7forwards *z$InstructBlipVisionEmbeddings.forwardF) r;r<r=r&rIr?Tensorintrqr@boolr __classcell__r6r6rWr7rBSs$(rBmodulequerykeyvalueattention_maskscalingdropoutc Kspt||dd|}|dur||}tjj|dd}tjj|||jd}t||} | dd} | |fS)Nr\ra)ptrainingr#r) r?matmulrxrrhsoftmaxrr contiguous) rrrrrrrkwargs attn_weights attn_outputr6r6r7eager_attention_forwards  rc sveZdZdZfddZdejdedefddZ dd ejd e ejd e eje eje e ejffd dZ Z S)InstructBlipAttentionz=Multi-headed attention from 'Attention Is All You Need' papercst||_|j|_|j|_|j|j|_|j|j|jkr-td|jd|jd|jd|_ d|_ |j |_ t j |jd|jdd|_|jr]t t|j}t t|j}nd}d}|duryt|tj|dd|f}t ||j_t |j|j|_dS) Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gFr)bias) requires_grad)rHrIrCrJrKnum_attention_heads num_headshead_dim ValueErrorscale is_causalattention_dropoutrLinearqkvqkv_biasrNr?zerosrk zeros_liker projection)r5rCq_biasv_biasrrWr6r7rIs0   zInstructBlipAttention.__init__tensorseq_lenbszcCs ||||j|jddS)Nr#r)rjrrrxr)r5rrrr6r6r7_shapes zInstructBlipAttention._shapeN hidden_states head_maskr.cKs|\}}}||}|||d|j||jddddd}|d|d|d}} } t} |jjdkrrIr?rrrrr9rrr6r6rWr7rs rc2eZdZfddZdejdejfddZZS)InstructBlipMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSr) rHrIrCr hidden_act activation_fnrrrJintermediate_sizefc1fc2rVrWr6r7rIs  zInstructBlipMLP.__init__rr.cCs"||}||}||}|Sr)rrrr5rr6r6r7r s   zInstructBlipMLP.forwardr;r<r=rIr?rrrr6r6rWr7rs rc sJeZdZdeffdd Zedejdejdee dej fdd Z Z S) InstructBlipEncoderLayerrCcsRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSNeps) rHrIrJrKr self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2rVrWr6r7rIs   z!InstructBlipEncoderLayer.__init__rrrr.cKsT|}||}|jd||d|\}}||}|}||}||}||}|S)N)rrr6)rrrr)r5rrrresidualr{r6r6r7rs    z InstructBlipEncoderLayer.forward) r;r<r=r$rIrr?rrrr@rrr6r6rWr7rsrc@sBeZdZUeed<dZdZdZdZdZ dZ dZ gdZ ddZ dS)InstructBlipPreTrainedModelrCblipT)InstructBlipQFormerEmbeddingsr%InstructBlipQFormerMultiHeadAttentionInstructBlipQFormerSelfOutputcCs|jj}t|tjtjfr%|jjjd|d|j dur#|j j dSdSt|tj r6|jjjd|ddSt|tj rK|j j |jj ddSt|trftjj|jd|dtjj|jd|ddSt|ttfru|jj dSdS)zInitialize the weightsr)meanstdN?)rCinitializer_range isinstancerrrQrudatanormal_rzero_ Embeddingrfill_rBinit trunc_normal_rUrP$InstructBlipForConditionalGenerationInstructBlipModel query_tokens)r5rfactorr6r6r7 _init_weightsHs"     z)InstructBlipPreTrainedModel._init_weightsN)r;r<r=r$rAbase_model_prefixsupports_gradient_checkpointing_supports_attention_backend_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_no_split_modulesrr6r6r6r7r5s  rc sVeZdZdZdeffdd Ze d deej de e de e effd d ZZS) InstructBlipEncodera Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`InstructBlipEncoderLayer`]. Args: config (`InstructBlipConfig`): The corresponding vision configuration for the `InstructBlipEncoder`. rCc:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr6)r)r2r{rCr6r7 jsz0InstructBlipEncoder.__init__..F) rHrIrCr ModuleListrangenum_hidden_layerslayersgradient_checkpointingrVrWrr7rIgs   zInstructBlipEncoder.__init__Nrrr.cKs.|}|jD] }||fd|i|}qt|dS)Nrlast_hidden_state)rr )r5 inputs_embedsrrr encoder_layerr6r6r7rms  zInstructBlipEncoder.forwardr)r;r<r=r>r$rIrrr?rrrrr9r rrr6r6rWr7r]s  rc seZdZUdZeed<eedZdeffdd Z e dde  dde e jd ed eed eeeffd d ZddZZS)InstructBlipVisionModelrrrC)r attentionscsJt|||_|j}t||_t||_tj ||j d|_ | dSr) rHrIrCrJrBrYrencoderrrrpost_layernorm post_init)r5rCrKrWr6r7rIs    z InstructBlipVisionModel.__init__F)tie_last_hidden_statesNrqrr.cKsn|durtd|j||d}|jdd|i|}|j}||}|dddddf}||}t||dS)Nz You have to specify pixel_values)rqrrr pooler_outputr6)rrYrrrr)r5rrrqrrencoder_outputsr pooled_outputr6r6r7rs   zInstructBlipVisionModel.forwardcC|jSr)rYr4r6r6r7get_input_embeddingsz,InstructBlipVisionModel.get_input_embeddings)NF)r;r<r=main_input_namer&rArr_can_record_outputsrIrrrr?r@rrrrr9rrrrr6r6rWr7rs*   rcsbeZdZdfdd ZddZddZdd Zd d Zd d Z    dde e fddZ Z S)rFcs"t||_|j|jdkrt|dstd|j|jf|j|_t|j|j|_|j|j|_ t |j|j |_ |rQt |j |j |_t |j |j |_nt |j|j |_t |j|j |_t |j|_t|dd|_|jdks{|jdkr|j|_t d|jd |j|_d |_dS) Nrembedding_sizezLThe hidden size (%d) is not a multiple of the number of attention heads (%d)position_embedding_typeabsolute relative_keyrelative_key_queryrr#F)rHrIrCrJrhasattrrrattention_head_size all_head_sizerrrencoder_hidden_sizerrDropoutattention_probs_dropout_probrr0rmax_position_embeddingsrdistance_embeddingsave_attentionr5rCis_cross_attentionrWr6r7rIs.   z.InstructBlipQFormerMultiHeadAttention.__init__cC ||_dSrattn_gradients)r5rr6r6r7save_attn_gradients z9InstructBlipQFormerMultiHeadAttention.save_attn_gradientscCrrrr4r6r6r7get_attn_gradientsrz8InstructBlipQFormerMultiHeadAttention.get_attn_gradientscCrr attention_map)r5rr6r6r7save_attention_maprz8InstructBlipQFormerMultiHeadAttention.save_attention_mapcCrrrr4r6r6r7get_attention_maprz7InstructBlipQFormerMultiHeadAttention.get_attention_mapcCs6|dd|j|jf}|j|}|ddddS)Nr\rrr#r)r^rr rjrg)r5x new_x_shaper6r6r7transpose_for_scoress z:InstructBlipQFormerMultiHeadAttention.transpose_for_scoresNrcKs|du}|r|||}|||} |}n|||}|||} ||} || } t| |dd} |jdksG|jdkr|d} tj | tj |j d dd}tj | tj |j d dd}||}| ||jd}|j| jd}|jdkrtd| |}| |} n|jdkrtd| |}td ||}| ||} | t|j} | j}|dur| |} tjdd | |}|r|jr||||j||}|dur||}t|| }|d d dd }|dd|jf}|j |}||fS)Nr\rr r r#rtdevicerszbhld,lrd->bhlrzbhrd,lrd->bhlrrarrr)r#rrrr?rrxrr^arangelongr%rjrrrvrteinsummathsqrtr rSoftmaxrr register_hookrrrgrr)r5rrrencoder_hidden_statesencoder_attention_maskrr key_layer value_layermixed_query_layer query_layerattention_scores seq_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_scores_dtypeattention_probsattention_probs_dropped context_layernew_context_layer_shaper6r6r7rsP              z-InstructBlipQFormerMultiHeadAttention.forwardrNNNN) r;r<r=rIrrrr r#rrrrr6r6rWr7rsrc8eZdZfddZdejdejdejfddZZS)rcsBtt|j|j|_tj|j|jd|_t|j |_ dSr) rHrIrrrJdenserrrhidden_dropout_probrrVrWr6r7rI0 z&InstructBlipQFormerSelfOutput.__init__r input_tensorr.cC&||}||}|||}|SrrCrrr5rrFr6r6r7r6  z%InstructBlipQFormerSelfOutput.forwardrr6r6rWr7r/ $rcsveZdZdfdd ZddZ    ddejdeejd eejd eejd eejd e e d ejfddZ Z S)InstructBlipQFormerAttentionFcs,tt|||_t||_t|_dSr)rHrIr attentionroutputset pruned_headsrrWr6r7rI?s    z%InstructBlipQFormerAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrr#ra)lenrrMrr rPrrrrrNrCrunion)r5headsindexr6r6r7 prune_headsEs z(InstructBlipQFormerAttention.prune_headsNrrrr-r.rr.c Ks0|jd|||||d|\}}|||} | S)N)rrrr-r.r6)rMrN) r5rrrr-r.rrr{attention_outputr6r6r7rWs  z$InstructBlipQFormerAttention.forwardrrA) r;r<r=rIrUr?rrr@rrrrr6r6rWr7rL>s,rLcr)InstructBlipQFormerIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) rHrIrrrJrrCrrstrr intermediate_act_fnrVrWr6r7rIns   z(InstructBlipQFormerIntermediate.__init__rr.cCs||}||}|Sr)rCrYrr6r6r7rvs  z'InstructBlipQFormerIntermediate.forwardrr6r6rWr7rWms rWcrB)InstructBlipQFormerOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) rHrIrrrrJrCrrrrDrrVrWr6r7rI~rEz"InstructBlipQFormerOutput.__init__rrFr.cCrGrrHrIr6r6r7rrJz!InstructBlipQFormerOutput.forwardrr6r6rWr7rZ}rKrZcsJeZdZfddZ     d deefddZdd Zd d ZZ S) InstructBlipQFormerLayercs~t|j|_d|_t||_||_||jdkr&t|dd|_d|_ nd|_ t ||_ t ||_ t ||_t ||_dS)Nr#rT)rF)rHrIchunk_size_feed_forward seq_len_dimrLrM layer_idxcross_attention_frequencycrossattentionhas_cross_attentionrW intermediaterZrNintermediate_query output_query)r5rCr^rWr6r7rIs     z!InstructBlipQFormerLayer.__init__Nrrc Ks|j|f||d|}|dkrk|ddd|ddf} |jr6|dur(td|j| f||||d|} t|j|j|j| } |jd|krit|j |j|j|dd|dddf | j } t j | | gdd} | St|j |j|j|} | S)N)rrrz>encoder_hidden_states must be given for cross-attention layers)rrr-r.r#ra)rMrarr`rfeed_forward_chunk_queryr\r]rcfeed_forward_chunkrvr%r?rk) r5rrrr-r. query_lengthrrVquery_attention_output layer_outputlayer_output_textr6r6r7rs^  z InstructBlipQFormerLayer.forwardcC||}|||}|Sr)rbrNr5rVintermediate_outputrir6r6r7rf  z+InstructBlipQFormerLayer.feed_forward_chunkcCrkr)rcrdrlr6r6r7rernz1InstructBlipQFormerLayer.feed_forward_chunk_queryNNNNr) r;r<r=rIrrrrfrerr6r6rWr7r[s  8r[cs>eZdZfddZe     ddeefddZZS) InstructBlipQFormerEncodercr)Ncsg|]}t|qSr6)r[)r2r^rr6r7rz7InstructBlipQFormerEncoder.__init__..F) rHrIrCrrrrlayerrrVrWrr7rIs  z#InstructBlipQFormerEncoder.__init__Nrrc KsVt|jjD]}|j|} |dur||nd} | ||| |f||d|}qt|dS)N)r.rgr)rrCrrrr) r5rrrr-r.rgri layer_modulelayer_head_maskr6r6r7rs"  z"InstructBlipQFormerEncoder.forwardro) r;r<r=rIrrrrrr6r6rWr7rps rpcs2eZdZdZfddZ    dddZZS) rz;Construct the embeddings from word and position embeddings.csttj|j|j|jd|_t|j|j|_ tj |j|j d|_ t |j|_|jdt|jdddt|dd|_||_dS) N) padding_idxr position_ids)r#r\F) persistentrr )rHrIrr vocab_sizerJ pad_token_idword_embeddingsrposition_embeddingsrr layernormrrDrregister_bufferr?r&ryr0rrCrVrWr6r7rIs  z&InstructBlipQFormerEmbeddings.__init__NrcCs|dur |d}nd}|dur |jdd|||f}|durI||}|jdkr;|||j}||}|durHtj ||fdd}n|}||j j j }| |}| |}|S)Nr#rr ra)r^rwcloner{rr|rvr%r?rkr}rurtr)r5 input_idsrw query_embedspast_key_values_lengthr4rYr|r6r6r7rs$    z%InstructBlipQFormerEmbeddings.forward)NNNr)r;r<r=r>rIrrr6r6rWr7r s rcseZdZdZdZdZdZdZee e dddge e dddgdZ de ffd d Z d d Zd dZddZ d"dejdeedejdedejf ddZee      d#dejdeejdeejdeejdeejdeejdeejdeedeeeje ffd d!Z!Z"S)$InstructBlipQFormerModelz Querying Transformer (Q-Former), used in InstructBLIP. Slightly modified from BLIP-2 as it also takes the instruction as input. Fr#z .attention)rT layer_namez.crossattention)rrcross_attentionsrCcs2t|||_t||_t||_|dSr)rHrIrCrrYrprrrVrWr6r7rISs    z!InstructBlipQFormerModel.__init__cCs|jjSrrYr{r4r6r6r7r]sz-InstructBlipQFormerModel.get_input_embeddingscCs ||j_dSrrr5rr6r6r7set_input_embeddings`s z-InstructBlipQFormerModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrrMrU)r5heads_to_prunerrrSr6r6r7 _prune_headscsz%InstructBlipQFormerModel._prune_headsr input_shaper% has_queryr.cCs|dkr|dddddddf}n|dkr(|ddddddf}n td|d|jd|j|jd}d|d }|S) a> Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (`torch.Tensor`): Mask with ones indicating tokens to attend to, zeros for tokens to ignore. input_shape (`tuple[int]`): The shape of the input to the model. device: (`torch.device`): The device of the input to the model. Returns: `torch.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`. rNrz!Wrong shape for input_ids (shape z) or attention_mask (shape )rsrg)rbrrcrvrt)r5rrr%rextended_attention_maskr6r6r7get_extended_attention_maskks   z4InstructBlipQFormerModel.get_extended_attention_maskNrrwrrr-r.rc sf|dur |dur td|dur|jdnd} j|||d} | dd} | \} } | j}|dur;tj| | f|d}|| |}|durt|t rU|d\}}}n|\}}}||f}t|t rofdd |D}n|durtj||d} |}n |}nd} |j j }j| f||||| d |}|j}|dddddf}t||d S) a$ query_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Hidden states to be used in the attention computation. If cross-attention, will be used for the query (i.e., key and value will use the encoder_hidden_states). Nz7You have to specify query_embeds when input_ids is Noner#r)rrwrr\)r%csg|]}|qSr6)invert_attention_mask)r2maskr4r6r7rrqz4InstructBlipQFormerModel.forward..)rrr-r.rgr)rrcrYr^r%r?onesrrlistr get_head_maskrCrrrr)r5rrrwrrr-r.rrgembedding_outputrrzr4r%rencoder_batch_sizeencoder_sequence_lengthr{encoder_hidden_shapeencoder_extended_attention_maskrsequence_outputrr6r4r7rsX     z InstructBlipQFormerModel.forwardr)NNNNNN)#r;r<r=r>rrrrr[rrrr%rIrrrr?rr9rr%rrrr LongTensorrr@rrrrrrr6r6rWr7r>sn      +  rz[ InstructBLIP base Model consisting of language model, qformer and vision encoder. cseZdZdZdgZdeffdd ZddZdd Zd d Z d d Z de j de j fddZee       dde j de j dee j dee j dee j dee j dee j dee jdedeedeeeffddZZS)rrrrrCcst|t|j|_ttd|j |j j |_ t |j |_t|j j |jj |_t|j|_|jjdur@|j|jj|jjdurN|j|jj|dSNr#)rHrIr vision_config vision_modelrrNr?rnum_query_tokensqformer_configrJrrqformerr text_configlanguage_projectionr from_configlanguage_modelrextend_keep_in_fp32_modulesrrVrWr6r7rIs      zInstructBlipModel.__init__cC |jSrrrr4r6r6r7rrz&InstructBlipModel.get_input_embeddingscC|j|dSrrrrr6r6r7r z&InstructBlipModel.set_input_embeddingscC,|jjs|jj|jj_|jj|jj_dSdSrrCuse_decoder_only_language_modelrsharedr embed_tokensdecoderr4r6r6r7 _tie_weights zInstructBlipModel._tie_weightscCP|j}t|dkrd|vrtjdkrtdt|jdr&d|jj _ dSdSz Some pre-processing hacks to make the model `accelerate` compatible. Check https://github.com/huggingface/transformers/pull/21707 for more details. r#raThe `language_model` is not in the `hf_device_map` dictionary and you are running your script in a multi-GPU environment. this may lead to unexpected behavior when using `accelerate`. Please pass a `device_map` that contains `language_model` to remove this warning. Please refer to https://github.com/huggingface/blog/blob/main/accelerate-large-models.md for more details on creating a `device_map` for large models._hf_hookTN hf_device_maprQr?cuda device_countloggerwarningr rrio_same_devicer5rr6r6r7_preprocess_accelerate" z(InstructBlipModel._preprocess_acceleraterrcC`|dur||tj|jjtj|jdk}|d}n||jjk}|d | |j}|SzZ Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`. Nr$r\ rr?rrCimage_token_idr'r%all unsqueeze expand_asrvr5rrspecial_image_maskr6r6r7get_placeholder_mask%  z&InstructBlipModel.get_placeholder_maskNFqformer_input_idsqformer_attention_maskrdecoder_input_idsdecoder_attention_maskrqrr.c Ks||jd || d| } | d} tj| ddtj| jd} |j| jddd}tj|ddtj| jd}|durCt |}tj ||gdd}|j d |||| | d| }|dddd|dddf}|dur|j |}|durt |}||}||j|j}|j||d }|||}|jjr|j d ||d | }n |j d ||||d | }t| ||d S)a qformer_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the Q-Former. Input tokens can optionally be provided to serve as text prompt, which the Q-Former model will encode. Indices can be obtained using [`InstructBlipProcessor`]. See [`InstructBlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) qformer_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Only relevant in case an encoder-decoder language model (like T5) is used. )rrrqrNr\r$r#ra)rrrr-r.rrr)rrrrr/r6)rr?rr^r'r%rryrc ones_likerkrrrrrvrtrmasked_scatterrCrr()r5rrrrrrrrrrqrr+ image_embedsimage_attention_maskrquery_attention_mask query_outputs query_outputlanguage_model_inputsroutputsr6r6r7r4sf(   $   zInstructBlipModel.forward)NNNNNNF)r;r<r=rrr$rIrrrrr?rr@rrrrrrrr rr9r(rrr6r6rWr7rsR    ra InstructBLIP Model for generating text given an image and an optional text prompt. The model consists of a vision encoder, Querying Transformer (Q-Former) and a language model. One can optionally pass `input_ids` to the model, which serve as a text prompt, to make the language model continue the prompt. Otherwise, the language model starts generating text from the [BOS] (beginning-of-sequence) token. cseZdZUeed<dZdZdgZdeffdd ZddZ d d Z d d Z d e j fddZddZddZddZddZ   d-dejdejdeejdeedeef ddZd ejd!ejfd"d#Zee        d.dejdejdeejd eejd$eejd%eejd&eejd!eejd'eejded(eed eee ffd)d*Z!e"      d/dejdeejdeejd eejd$eejd!eejded ejfd+d,Z#Z$S)0rrCrrTrcst|t|j|_tt d|j |j j |_ t|j |_t|j j |jj |_|jr7t|j}nt|j}|jdurI|j|j|jdurU|j|j||_|dSr)rHrIr _from_configrrrrNr?rrrrJrrrrrrrr!rr"rrrrr)r5rCrrWr6r7rIs     z-InstructBlipForConditionalGeneration.__init__cCrrrr4r6r6r7rrz9InstructBlipForConditionalGeneration.get_input_embeddingscCrrrrr6r6r7rrz9InstructBlipForConditionalGeneration.set_input_embeddingscCrr)rset_output_embeddings)r5new_embeddingsr6r6r7rrz:InstructBlipForConditionalGeneration.set_output_embeddingsr.cCrr)rget_output_embeddingsr4r6r6r7rrz:InstructBlipForConditionalGeneration.get_output_embeddingscCrr)r get_encoderr4r6r6r7rrz0InstructBlipForConditionalGeneration.get_encodercCrr)r get_decoderr4r6r6r7rrz0InstructBlipForConditionalGeneration.get_decodercCrrrr4r6r6r7rrz1InstructBlipForConditionalGeneration._tie_weightscCrrrrr6r6r7rrz;InstructBlipForConditionalGeneration._preprocess_accelerateNFrrrq return_dictcCs|j||dd}|d}tj|ddtj|jd}|j|jddd} tj| ddtj|jd} |dur@t |}tj | |gdd}|j ||| ||dd } | dddd| dddf} | | } |rr| || fS| S) a$ Encodes images into continuous embeddings that can be forwarded to the language model. Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input images. T)rrrqrrNr\r$r#ra)rrrr-r.r) rr?rr^r'r%rryrcrrkrr)r5rrrrrqrr+rrrrrrrr6r6r7get_image_featuress2   $  z7InstructBlipForConditionalGeneration.get_image_featuresrrcCrrrrr6r6r7rrz9InstructBlipForConditionalGeneration.get_placeholder_maskrrrlabelsrc Ks|j|||| dd\} } }|dur||}|dur t|}| |j|j} |j||d}||| }|j j r^|j d ||d| }|d}d}| dur]|j d || |j j jd| }nd| d<|j d ||||| d | }|j}|j}t||| ||d S) aD qformer_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the Q-Former. Input tokens can optionally be provided to serve as text prompt, which the Q-Former model will encode. Indices can be obtained using [`InstructBlipProcessor`]. See [`InstructBlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) qformer_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Only relevant in case an encoder-decoder language model (like T5) is used. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import InstructBlipProcessor, InstructBlipForConditionalGeneration >>> import torch >>> from PIL import Image >>> import requests >>> model = InstructBlipForConditionalGeneration.from_pretrained("Salesforce/instructblip-vicuna-7b") >>> processor = InstructBlipProcessor.from_pretrained("Salesforce/instructblip-vicuna-7b") >>> device = "cuda" if torch.cuda.is_available() else "cpu" >>> model.to(device) # doctest: +IGNORE_RESULT >>> url = "https://raw.githubusercontent.com/salesforce/LAVIS/main/docs/_static/Confusing-Pictures.jpg" >>> image = Image.open(requests.get(url, stream=True).raw).convert("RGB") >>> prompt = "What is unusual about this image?" >>> inputs = processor(images=image, text=prompt, return_tensors="pt").to(device) >>> outputs = model.generate( ... **inputs, ... do_sample=False, ... num_beams=5, ... max_length=256, ... min_length=1, ... top_p=0.9, ... repetition_penalty=1.5, ... length_penalty=1.0, ... temperature=1, ... ) >>> generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0].strip() >>> print(generated_text) The unusual aspect of this image is that a man is ironing clothes on the back of a yellow SUV, which is parked in the middle of a busy city street. This is an unconventional approach to ironing clothes, as it requires the man to balance himself and his ironing equipment on top of the vehicle while navigating through traffic. Additionally, the presence of taxis and other vehicles in the scene further emphasizes the unusual nature of this situation. ```TrrrqrNrrr)r*rryr)rrrrr)r)r*r+r,r-r6)rrr?rrvr%rtrrrCrr loss_functionrryr)r*r()r5rrrrrrrrrrrqrrr+rrrr*r)r6r6r7r.sbM     z,InstructBlipForConditionalGeneration.forwardcKst|dr ||jd} |j||||dd\} } } |durK|durE|jjg|jj} | |jjjg}t j |gt j |j d}| | d}||}|durTt |}| |j |j} |j||d}||| }||d }|jjjsw||d <|jjd i||}|S) a Overrides `generate` function to be able to use the model as a conditional generator. Args: pixel_values (`torch.FloatTensor` of shape (batch_size, num_channels, height, width)): Input images to be processed. qformer_input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt to be fed to the Q-Former module. qformer_attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt for the generation. attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Embedded representation of the inputs. Should be float, not int tokens. interpolate_pos_encoding (`bool`, *optional*, defaults to `False`): Whether to interpolate the positional encoding of the image embeddings. Returns: captions (list): A list of strings of length batch_size * num_captions. rrTrNr$r#rrrr6)r rrcrrCimage_token_indexrr bos_token_idr?rr'r%repeatrrrvrtrrris_encoder_decodergenerate)r5rrrrrrrrqgenerate_kwargsrzrr+r image_tokens start_tokensrinputsrr6r6r7rs6 "        z-InstructBlipForConditionalGeneration.generate)NFF)NNNNNNNF)NNNNNF)%r;r<r=r$rArrrrIrrrrModulerrrrrr?r@rrrrrrrrrrr9r(rno_gradrrr6r6rWr7rs   1      r)rrrrr)r)Nr>r) dataclassesrtypingrrrrr?r activationsr generationr modeling_flash_attention_utilsr modeling_layersr modeling_outputsr rrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrrrutilsrrrrrr utils.genericrrautor r!r"configuration_instructblipr$r%r& get_loggerr;rr(rrBrfloatrrrrrrrrrrLrWrZr[rprrrr__all__r6r6r6r7s         R M"'#4{/Y'3-)  X