o ei@sddlZddlmZddlmZddlmZddlZddlmZddl m Z ddl m Z dd lmZdd lmZdd lmZdd lmZmZmZmZmZmZdd lmZmZddlmZddl m!Z!ddl"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*ddl+m,Z,m-Z-ddl.m/Z/m0Z0m1Z1ddl2m3Z3m4Z4m5Z5e'6e7Z8Gdddej9Z:Gdddej9Z;e%GdddeZ< dNdej9dej=d ej=d!ej=d"ej=dBd#e>d$e>fd%d&Z?Gd'd(d(ej9Z@Gd)d*d*ej9ZAGd+d,d,eZBGd-d.d.ej9ZCGd/d0d0e<ZDGd1d2d2ej9ZEGd3d4d4ej9ZFGd5d6d6ej9ZGGd7d8d8ej9ZHGd9d:d:ej9ZIGd;d<dd>ej9ZKGd?d@d@e<ZLee%dAdBGdCdDdDe#ZMe%dEdBGdFdGdGe<ZNee%GdHdIdIeZOe%dJdBGdKdLdLerAr'<s$(r'cs2eZdZdZfddZ    dddZZS) "InstructBlipVideoQFormerEmbeddingsz;Construct the embeddings from word and position embeddings.cs~ttj|j|j|jd|_t|j|j|_ tj |j|j d|_ t |j|_|jdt|jddd||_dS)N) padding_idxeps position_idsr#rFF) persistent)r-r.r Embedding vocab_sizer/ pad_token_idword_embeddingsmax_position_embeddingsposition_embeddings LayerNormlayer_norm_eps layernormDropouthidden_dropout_probdropoutregister_bufferr4arangercr(r<r>r@rAr.s  z+InstructBlipVideoQFormerEmbeddings.__init__NrcCs|dur |d}nd}|dur |jdd|||f}|durD||}|||j}||}|durCtj||fdd}n|}||j j j }| |}| |}|S)Nr#rrK) rHrwcloner}rr`devicer4rUrr_r^r)r= input_idsrw query_embedspast_key_values_length seq_lengthrBrr@r@rAris"   z*InstructBlipVideoQFormerEmbeddings.forwardNNNr)rkrlrm__doc__r.rirrr@r@r>rArss rscsVeZdZUeed<dZdZdZdZdZ dZ dZ dZ gdZ efddZZS) InstructBlipVideoPreTrainedModelr(blip)videotextT)rsInstructBlipVideoAttention*InstructBlipVideoQFormerMultiHeadAttention"InstructBlipVideoQFormerSelfOutputcst||jj}t|tr#tj|jd|dtj|j d|ddSt|t t fr2t |j dSt|trJt|jt|jjdddSdS)zInitialize the weights)meanstdrFrxN)r- _init_weightsr(initializer_range isinstancer'init trunc_normal_r;r6)InstructBlipVideoForConditionalGenerationInstructBlipVideoModelzeros_ query_tokensrscopy_rwr4rrMrc)r=modulefactorr>r@rArs   &z.InstructBlipVideoPreTrainedModel._init_weights)rkrlrmr$__annotations__base_model_prefixinput_modalitiessupports_gradient_checkpointing_supports_attention_backend_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_no_split_modulesr4no_gradrrrr@r@r>rArs rrrquerykeyvalueattention_maskscalingrc Kspt||dd|}|dur||}tjj|dd}tjj|||jd}t||} | dd} | |fS)NrFrK)ptrainingr#r) r4matmulrbrrRsoftmaxrr contiguous) rrrrrrrkwargs attn_weights attn_outputr@r@rAeager_attention_forwards  rcsheZdZdZfddZdejdedefddZd ejd e ejejd Be ejd Bffd d Z Z S)rz=Multi-headed attention from 'Attention Is All You Need' papercst||_|j|_|j|_|j|j|_|j|j|jkr-td|jd|jd|jd|_ d|_ |j |_ t j |jd|jdd|_|jr]t t|j}t t|j}nd}d}|duryt|tj|dd|f}t ||j_t |j|j|_dS) Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gFr)bias) requires_grad)r-r.r(r/r0num_attention_heads num_headshead_dim ValueErrorscale is_causalattention_dropoutrLinearqkvqkv_biasr3r4zerosrU zeros_liker projection)r=r(q_biasv_biasrr>r@rAr.s0   z#InstructBlipVideoAttention.__init__tensorseq_lenbszcCs ||||j|jddS)Nr#r)rTrrrbr)r=rrrr@r@rA_shapes z!InstructBlipVideoAttention._shape hidden_statesrENc Ks|\}}}||}|||d|j||jddddd}|d|d|d}}} t|jjt } | |||| fd|j sBdn|j |j d|\} } | ||d  } || } | | fS) z#Input shape: Batch x Time x Channelrrrr#Nr)rrrrF)rHrrPrrQr get_interfacer(_attn_implementationrrrrrr) r=rrrtgt_lenr0 mixed_qkv query_states key_states value_statesattention_interfacerrr@r@rAris0    z"InstructBlipVideoAttention.forward) rkrlrmrr.r4rnrortuplerirrr@r@r>rArs rc2eZdZfddZdejdejfddZZS)InstructBlipVideoMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSN) r-r.r(r hidden_act activation_fnrrr/intermediate_sizefc1fc2r<r>r@rAr.=s  zInstructBlipVideoMLP.__init__rrEcCs"||}||}||}|Sr)rrrr=rr@r@rAriDs   zInstructBlipVideoMLP.forwardrkrlrmr.r4rnrirrr@r@r>rAr<s rcsDeZdZdeffdd Zedejdee dej fddZ Z S) InstructBlipVideoEncoderLayerr(csRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSNru) r-r.r/r0r self_attnrrr layer_norm1rmlp layer_norm2r<r>r@rAr.Ls   z&InstructBlipVideoEncoderLayer.__init__rrrEcKsR|}||}|jdd|i|\}}||}|}||}||}||}|S)Nrr@)rrrr)r=rrresidualrer@r@rAriTs    z%InstructBlipVideoEncoderLayer.forward) rkrlrmr$r.rr4rnrrrprirrr@r@r>rArKsrcsDeZdZdZdeffdd Zedeede e BfddZ Z S) InstructBlipVideoEncodera" Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`InstructBlipVideoEncoderLayer`]. Args: config (`InstructBlipVideoConfig`): The corresponding vision configuration for the `InstructBlipVideoEncoder`. r(c:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr@)r).0rer(r@rA xsz5InstructBlipVideoEncoder.__init__..F) r-r.r(r ModuleListrangenum_hidden_layerslayersgradient_checkpointingr<r>rrAr.us   z!InstructBlipVideoEncoder.__init__rrEcKs*|}|jD] }||fi|}qt|dS)Nlast_hidden_state)rr )r= inputs_embedsrr encoder_layerr@r@rAri{s  z InstructBlipVideoEncoder.forward) rkrlrmrr$r.rrrrr rirrr@r@r>rArks rc seZdZUdZdZeed<eedZ deffdd Z e e dde ddejd Bd ed eed eeBfd dZddZZS)InstructBlipVideoVisionModelr\rr()r attentionscsJt|||_|j}t||_t||_tj ||j d|_ | dSr) r-r.r(r/r'rBrencoderrrrpost_layernorm post_init)r=r(r0r>r@rAr.s    z%InstructBlipVideoVisionModel.__init__F)tie_last_hidden_statesNr[rrEcKsn|durtd|j||d}|jdd|i|}|j}||}|dddddf}||}t||dS)Nz You have to specify pixel_values)r[rrr pooler_outputr@)rrBrrrr)r=r\r[rrencoder_outputsr pooled_outputr@r@rAris    z$InstructBlipVideoVisionModel.forwardcC|jSr)rBr=r@r@rAget_input_embeddingsz1InstructBlipVideoVisionModel.get_input_embeddingsNF)rkrlrmmain_input_namerr&rrr_can_record_outputsr.rrrr4rprqrrrrrir rrr@r@r>rArs.  rcs`eZdZdfdd ZddZddZdd Zd d Zd d Z   dde e fddZ Z S)rFcst||_|j|jdkrt|dstd|j|jf|j|_t|j|j|_|j|j|_ t |j|j |_ |rQt |j |j |_t |j |j |_nt |j|j |_t |j|j |_t |j|_d|_dS)Nrembedding_sizezLThe hidden size (%d) is not a multiple of the number of attention heads (%d)F)r-r.r(r/rhasattrrroattention_head_size all_head_sizerrrencoder_hidden_sizerrrattention_probs_dropout_probrsave_attentionr=r(is_cross_attentionr>r@rAr.s&   z3InstructBlipVideoQFormerMultiHeadAttention.__init__cC ||_dSrattn_gradients)r=rr@r@rAsave_attn_gradients z>InstructBlipVideoQFormerMultiHeadAttention.save_attn_gradientscCr rrr r@r@rAget_attn_gradientsr z=InstructBlipVideoQFormerMultiHeadAttention.get_attn_gradientscCrr attention_map)r=r r@r@rAsave_attention_maprz=InstructBlipVideoQFormerMultiHeadAttention.save_attention_mapcCr rrr r@r@rAget_attention_mapr zrArsrc8eZdZfddZdejdejdejfddZZS)rcsBtt|j|j|_tj|j|jd|_t|j |_ dSr) r-r.rrr/denserrrrrr<r>r@rAr.# z+InstructBlipVideoQFormerSelfOutput.__init__r input_tensorrEcC&||}||}|||}|Srr8rrr=rr:r@r@rAri)  z*InstructBlipVideoQFormerSelfOutput.forwardrr@r@r>rAr" $rc sbeZdZd fdd Z   ddejdejdBdejdBdejdBd eed ejf d d Z Z S)!InstructBlipVideoQFormerAttentionFcs$tt|||_t||_dSr)r-r.r attentionroutputrr>r@rAr.1s  z*InstructBlipVideoQFormerAttention.__init__Nrrr*r+rrEc Ks.|jd||||d|\}}|||}|S)N)rrr*r+r@)rArB) r=rrr*r+rrreattention_outputr@r@rAri6s z)InstructBlipVideoQFormerAttention.forwardrjr6) rkrlrmr.r4rnrprrrirrr@r@r>rAr@0s$r@cr)$InstructBlipVideoQFormerIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) r-r.rrr/rr8rrstrrintermediate_act_fnr<r>r@rAr.Js   z-InstructBlipVideoQFormerIntermediate.__init__rrEcCs||}||}|Sr)r8rFrr@r@rAriRs  z,InstructBlipVideoQFormerIntermediate.forwardrr@r@r>rArDIs rDcr7)InstructBlipVideoQFormerOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) r-r.rrrr/r8rrrrrr<r>r@rAr.Yr9z'InstructBlipVideoQFormerOutput.__init__rr:rEcCr;rr<r=r@r@rAri_r>z&InstructBlipVideoQFormerOutput.forwardrr@r@r>rArGXr?rGcsHeZdZfddZ    d deefddZdd Zd d ZZ S) InstructBlipVideoQFormerLayercs~t|j|_d|_t||_||_||jdkr&t|dd|_d|_ nd|_ t ||_ t ||_ t ||_t ||_dS)Nr#rT)rF)r-r.chunk_size_feed_forward seq_len_dimr@rA layer_idxcross_attention_frequencycrossattentionhas_cross_attentionrD intermediaterGrBintermediate_query output_query)r=r(rKr>r@rAr.gs     z&InstructBlipVideoQFormerLayer.__init__Nrrc Ks|j|fd|i|}|dkri|ddd|ddf}|jr4|dur'td|j|f|||d|}t|j|j|j|} |jd|krgt|j |j|j|dd|dddf | j } t j | | gdd} | St|j |j|j|} | S)Nrrz>encoder_hidden_states must be given for cross-attention layers)rr*r+r#rK)rArNrrMrfeed_forward_chunk_queryrIrJrMfeed_forward_chunkr`rr4rU) r=rrr*r+ query_lengthrrCquery_attention_output layer_outputlayer_output_textr@r@rAri{sZ z%InstructBlipVideoQFormerLayer.forwardcC||}|||}|Sr)rOrBr=rCintermediate_outputrVr@r@rArS  z0InstructBlipVideoQFormerLayer.feed_forward_chunkcCrXr)rPrQrYr@r@rArRr[z6InstructBlipVideoQFormerLayer.feed_forward_chunk_queryr) rkrlrmr.rrrirSrRrrr@r@r>rArHfs  5rHcs<eZdZfddZe    ddeefddZZS) InstructBlipVideoQFormerEncodercr)Ncsg|]}t|qSr@)rH)rrKrr@rArz.F) r-r.r(rrrrlayerrr<r>rrAr.s  z(InstructBlipVideoQFormerEncoder.__init__Nrrc Ks@t|jjD]}|j|}||||f||d|}qt|dS)N)r+rTr)rr(rr^r ) r=rrr*r+rTri layer_moduler@r@rAris  z'InstructBlipVideoQFormerEncoder.forwardr) rkrlrmr.rrrrirrr@r@r>rAr\s r\cseZdZdZdZdZdZdZee e dddge e dddgdZ de ffd d Z d d Zd dZ ddejdeedejdedejf ddZeee     d dejdejdBdejdBdejdBdejdBdejdBdeedeejeBfddZZ S)!InstructBlipVideoQFormerModelz Querying Transformer (Q-Former), used in InstructBlipVideo. Slightly modified from BLIP-2 as it also takes the instruction as input. Fr#z .attention)index layer_namez.crossattention)rrcross_attentionsr(cs2t|||_t||_t||_|dSr)r-r.r(rsrBr\rrr<r>r@rAr.s    z&InstructBlipVideoQFormerModel.__init__cCs|jjSrrBr}r r@r@rAr sz2InstructBlipVideoQFormerModel.get_input_embeddingscCs ||j_dSrrer=rr@r@rAset_input_embeddingss z2InstructBlipVideoQFormerModel.set_input_embeddingsr input_shaper has_queryrEcCs|dkr|dddddddf}n|dkr(|ddddddf}n td|d|jd|j|jd}d|d }|S) a> Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (`torch.Tensor`): Mask with ones indicating tokens to attend to, zeros for tokens to ignore. input_shape (`tuple[int]`): The shape of the input to the model. device: (`torch.device`): The device of the input to the model. Returns: `torch.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`. rNrz!Wrong shape for input_ids (shape z) or attention_mask (shape )r]g?g)rLrrMr`r^)r=rrhrriextended_attention_maskr@r@rAget_extended_attention_masks   z9InstructBlipVideoQFormerModel.get_extended_attention_maskNrrwrr*r+rc sT|dur |dur td|dur|jdnd}j|||d} | dd} | \} } | j} |dur;tj| | f| d}|| | }|durt|t rU|d\}}}n|\}}}||f}t|t rofdd |D}n|durtj|| d} |}n |}nd}j | f||||d |}|j }|dddddf}t ||d S) a$ query_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Hidden states to be used in the attention computation. If cross-attention, will be used for the query (i.e., key and value will use the encoder_hidden_states). Nz7You have to specify query_embeds when input_ids is Noner#r)rrwrrF)rcsg|]}|qSr@)invert_attention_mask)rmaskr r@rArar]z9InstructBlipVideoQFormerModel.forward..)rr*r+rTr)rrMrBrHrr4onesrlrlistrmrrr)r=rrrwrr*r+rrTembedding_outputrhrdrrrkencoder_batch_sizeencoder_sequence_lengthreencoder_hidden_shapeencoder_extended_attention_maskrsequence_outputrr@r rAri/sT    z%InstructBlipVideoQFormerModel.forwardrj)NNNNN)!rkrlrmrrrrrrHrrrr%r.r rgr4rnrrorrqrlrrr LongTensorrprrrrirrr@r@r>rArash     + razV Class defining the outputs of [`InstructBlipVideoForConditionalGeneration`]. ) custom_introc@seZdZUdZdZeejdBed<dZ eejdBed<dZ e dBed<dZ e dBed<dZeeBdBed<deefd d ZdS) 4InstructBlipVideoForConditionalGenerationModelOutputa loss (`torch.FloatTensor`, *optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Language modeling loss from the language model. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head of the language model. vision_outputs (`BaseModelOutputWithPooling`): Outputs of the vision encoder. qformer_outputs (`BaseModelOutputWithPoolingAndCrossAttentions`): Outputs of the Q-Former (Querying Transformer). language_model_outputs (`CausalLMOutputWithPast` or `Seq2SeqLMOutput`): Outputs of the language model. Nlosslogitsvision_outputsqformer_outputslanguage_model_outputsrEcstfddDS)Nc3s.|]}|dvr |nt|VqdS)r|r}r~N)getattrto_tuple)rkr r@rA s  zPInstructBlipVideoForConditionalGenerationModelOutput.to_tuple..)rkeysr r@r rArs z=InstructBlipVideoForConditionalGenerationModelOutput.to_tuple)rkrlrmrrzrr4rprr{r|rr}rr~rrrrr@r@r@rAry{s  ryz` InstructBlipVideo base Model consisting of language model, qformer and vision encoder. c"seZdZdZdgZdeffdd ZddZdd Zd d Z d e j d e j fddZ ee           d de j de j de j dBd e j dBde j dBde j dBde j dBd e jdBdedBdedBdedBdededBdeedeeBfddZZS)!rr\rr(cspt|t|j|_ttd|j |j j |_ t |j |_t|j j |jj |_t|j|_|dSNr#)r-r.r vision_config vision_modelrr3r4rnum_query_tokensqformer_configr/rraqformerr text_configlanguage_projectionr from_configlanguage_modelrr<r>r@rAr.s    zInstructBlipVideoModel.__init__cC |jSrrr r r@r@rAr rz+InstructBlipVideoModel.get_input_embeddingscC|j|dSrrrgrfr@r@rArgz+InstructBlipVideoModel.set_input_embeddingscCP|j}t|dkrd|vrtjdkrtdt|jdr&d|jj _ dSdSz Some pre-processing hacks to make the model `accelerate` compatible. Check https://github.com/huggingface/transformers/pull/21707 for more details. r#raThe `language_model` is not in the `hf_device_map` dictionary and you are running your script in a multi-GPU environment. this may lead to unexpected behavior when using `accelerate`. Please pass a `device_map` that contains `language_model` to remove this warning. Please refer to https://github.com/huggingface/blog/blob/main/accelerate-large-models.md for more details on creating a `device_map` for large models._hf_hookTN hf_device_maplenr4cuda device_countloggerwarningrrrio_same_devicer=rr@r@rA_preprocess_accelerate" z-InstructBlipVideoModel._preprocess_acceleraterrcC`|dur||tj|jjtj|jdk}|d}n||jjk}|d | |j}|SzZ Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`. Nr^rrF) r r4rr(image_token_idlongrall unsqueeze expand_asr`r=rrspecial_image_maskr@r@rAget_placeholder_mask  z+InstructBlipVideoModel.get_placeholder_maskNFqformer_input_idsqformer_attention_maskrdecoder_input_idsdecoder_attention_maskoutput_attentionsoutput_hidden_states return_dictr[ use_cacherrEc Ks:| dur| n|jj} |j\}}}}}||||||}|j|| | | | d}|d}tj|ddtj|j d}|j |jddd}tj|ddtj|j d}|dur^t |}|j |dd}|j |dd}tj||gdd}|j|||||| | | d}|dddd|dddf}||}|||jj|d}|dur|j|}||jjk}|durt |}n||tj|jjtj|j dk}|d}|d||j }||j |j}|||}|jjr|jd ||| | | | d |}n|jd ||||| | | | d |}t|||d S) a qformer_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the Q-Former. Input tokens can optionally be provided to serve as text prompt, which the Q-Former model will encode. Indices can be obtained using [`InstructBlipVideoProcessor`]. See [`InstructBlipVideoProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) qformer_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Only relevant in case an encoder-decoder language model (like T5) is used. N)r\rrrr[rrFrrKr#)rrrr*r+rrrrrrrrr)rrrrrrrrrr@)r(use_return_dictrMrPrr4rorHrrrrc ones_likerepeat_interleaverUrrrrr video_token_idrrrrr`r^masked_scatteruse_decoder_only_language_modelry)r=r\rrrrrrrrrrr[rrrdframeschannelrCrDr| image_embedsimage_attention_maskrquery_attention_mask query_outputs query_outputlanguage_model_inputsroutputsr@r@rAris)   $         zInstructBlipVideoModel.forward) NNNNNNNNNFN)rkrlrmr_keep_in_fp32_modulesr$r.r rgrr4rwrprrrrnrqrr rryrirrr@r@r>rArsh      rc@s2eZdZUdZdZedBed<dZedBed<dS)'BaseModelOutputWithVisionQformerOutputsz vision_outputs (`BaseModelOutputWithPooling`): Outputs of the vision encoder. qformer_outputs (`BaseModelOutputWithPoolingAndCrossAttentions`): Outputs of the Q-Former (Querying Transformer). Nr|r}) rkrlrmrr|rrr}rr@r@r@rAras ra InstructBlipVideo Model for generating text given an image and an optional text prompt. The model consists of a vision encoder, Querying Transformer (Q-Former) and a language model. One can optionally pass `input_ids` to the model, which serve as a text prompt, to make the language model continue the prompt. Otherwise, the language model starts generating text from the [BOS] (beginning-of-sequence) token. c$seZdZUeed<dZdZdgZdeffdd ZddZ d d Z d d Z d e j fddZd.fdd ZddZddZdejdejfddZee            d/dejdejdejdBdejdBdejdBdejdBd ejdBdejdBd!edBd"edBd#ejdBd$edBd%ed&edBd'eed eeBf d(d)Ze      d0dejdejdBdejdBdejdBdejdBdejdBd%ed ejfd*d+Zee  d1dejdejdejdBd%edBd'eed ee Bf d,d-Z!Z"S)2rr(r\Trcst|t|j|_tt d|j |j j |_ t|j |_t|j j |jj |_|jr7t|j}nt|j}||_|dSr)r-r.r _from_configrrrr3r4rrrr/rrarrrrrr!rr"rr)r=r(rr>r@rAr.s   z2InstructBlipVideoForConditionalGeneration.__init__cCrrrr r@r@rAr rz>InstructBlipVideoForConditionalGeneration.get_input_embeddingscCrrrrfr@r@rArgrz>InstructBlipVideoForConditionalGeneration.set_input_embeddingscCrr)rset_output_embeddings)r=new_embeddingsr@r@rArrz?InstructBlipVideoForConditionalGeneration.set_output_embeddingsrEcCrr)rget_output_embeddingsr r@r@rArrz?InstructBlipVideoForConditionalGeneration.get_output_embeddingsNcs |dur |jStj|dS)N)modality)r get_encoderr-)r=rr>r@rArs z5InstructBlipVideoForConditionalGeneration.get_encodercCrr)r get_decoderr r@r@rArrz5InstructBlipVideoForConditionalGeneration.get_decodercCrrrrr@r@rArrz@InstructBlipVideoForConditionalGeneration._preprocess_acceleraterrcCrr) r r4rr(rrrrrrr`rr@r@rArrz>InstructBlipVideoForConditionalGeneration.get_placeholder_maskFrrrrrrrlabelsrr[rrc KsP| dur| n|jj} |j|f||| dd|}|j}|j}|j}|dur+||}|dur4t|}| |j |j }|j ||d}| ||}|jjr{|jd ||| | | |d|}| ra|jn|d}d}| durz|jd || |jjjd|}n$|jd ||||| | | | |d |}| r|jn|d}| r|jn|d }t|||||d S) a qformer_input_ids (`torch.LongTensor` of shape (batch_size, sequence_length)): The sequence used as a prompt to be fed to the Q-Former module. qformer_attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. Examples: ```python >>> from transformers import InstructBlipVideoProcessor, InstructBlipVideoForConditionalGeneration >>> import torch >>> from huggingface_hub import hf_hub_download >>> import av >>> import numpy as np >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> model = InstructBlipVideoForConditionalGeneration.from_pretrained("Salesforce/instructblip-vicuna-7b", device_map="auto") >>> processor = InstructBlipVideoProcessor.from_pretrained("Salesforce/instructblip-vicuna-7b") >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample uniformly 4 frames from the videWhy is this video funny?o >>> total_frames = container.streams.video[0].frames >>> indices = np.arange(0, total_frames, total_frames / 4).astype(int) >>> clip = read_video_pyav(container, indices) >>> prompt = "What is happening in the video?" >>> inputs = processor(text=prompt, images=clip, return_tensors="pt").to(model.device) >>> outputs = model.generate( ... **inputs, ... do_sample=False, ... num_beams=5, ... max_length=256, ... repetition_penalty=1.5, ... length_penalty=1.0, ... ) >>> generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0].strip() >>> print(generated_text) "A person is eating a bowl of pasta, and they are using a fork to eat it. The person is sitting at a table, and the plate of pasta is on the table in front" ```NTrrr[rrrr)r{rr{) rrrrrrrrrr#)rzr{r|r}r~r@)r(rget_video_featuresrr}r|r r4rr`rr^rrrrr{ loss_functionrr{rzry)r=r\rrrrrrrrrrrr[rrvideo_featuresrr}r|rrr{rzr@r@rAris~S       z1InstructBlipVideoForConditionalGeneration.forwardcKs t|dr ||jd} |j||||dd} | j} |durM|durG|jjg|jjd} | |jjj g} t j | gt j |j d}|| d}||}|durVt |}| |j |j} |j||d }||| }||d }|jjjsy||d <|jjd i||}|S) a Overrides `generate` function to be able to use the model as a conditional generator. Args: pixel_values (`torch.FloatTensor` of shape (batch_size, num_channels, height, width) or (batch_size, num_frames, num_channels, height, width)): Input images or videos to be processed. qformer_input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt to be fed to the Q-Former module. qformer_attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt for the generation. attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Embedded representation of the inputs. Should be float, not int tokens. interpolate_pos_encoding (`bool`, *optional*, defaults to `False`): Whether to interpolate the positional encoding of the image embeddings. Returns: captions (list): A list of strings of length batch_size * num_captions. rrTrNrrr#r)rrrr@)rrrMrrr(video_token_indexrr bos_token_idr4rrrrepeatr rr`r^rrris_encoder_decodergenerate)r=r\rrrrrr[generate_kwargsrdrr video_tokens start_tokensrinputsrr@r@rAr_s8 "       z2InstructBlipVideoForConditionalGeneration.generatec Ksh|j\}}}} } ||||| | }|jd ||dd|} t| j| j| j| j| dd} | d} tj | ddtj | j d} |j | jddd}tj | ddtj | j d}|durdt|}|j|dd}|j|dd}tj||gd d}|jd |||| | dd |}|| _|dddd| d ddf}||}|||jj|d}|| _| S) a pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input images. qformer_input_ids (`torch.LongTensor` of shape (batch_size, sequence_length)): The sequence used as a prompt to be fed to the Q-Former module. qformer_attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. T)r\r[rN)rrrrr|r}rrFrrKr#)rrrr*r+rr@)rMrPrrrrrrr4rorHrrrrcrrrUrr}rr(r)r=r\rrr[rrdrrrCrDr|rrrrr}rrr@r@rArsT    $ zrAros        Fr)rrrarr)r)Rr&collections.abcr dataclassesrtypingrr4rrr activationsr generationr modeling_flash_attention_utilsr modeling_layersr modeling_outputsr r rrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsrrrrrr utils.genericrutils.output_capturingrrautor r!r"configuration_instructblipvideor$r%r& get_loggerrkrrr'rsrrnfloatrrrrrrrrr@rDrGrHr\raryrrr__all__r@r@r@rAs               J2) J  6aU$?   z