o Gii@sddlZddlmZddlmZddlmZddlmZm Z m Z ddl m Z m Z ddlmZmZmZmZmZddlmZdd lmZmZeeZGd d d ejZGd d d ejZGdddejZGdddejZ GdddejZ!GdddeZ"GdddeZ#dS)N)nn) BertTokenizer)QuickGELUActivation))BaseModelOutputWithPastAndCrossAttentionsBaseModelOutputWithPooling,BaseModelOutputWithPoolingAndCrossAttentions) Blip2ConfigBlip2VisionConfig) Blip2EncoderBlip2PreTrainedModelBlip2QFormerAttentionBlip2QFormerIntermediateBlip2QFormerOutput)apply_chunking_to_forward)loggingreplace_return_docstringscs2eZdZdZfddZ    dddZZS) Blip2TextEmbeddingsz;Construct the embeddings from word and position embeddings.csttj|j|j|jd|_t|j|j|_ tj |j|j d|_ t |j |_|dt|jdt|dd|_||_dS)N) padding_idxeps position_ids)position_embedding_typeabsolute)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandgetattrrconfigselfr. __class__e/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/pipelines/blip_diffusion/modeling_blip2.pyr/s  zBlip2TextEmbeddings.__init__Nrc Cs|dur |d}nd}|dur |jdd|||f}|durQ||}|jdkr7||}||}|durP|jd}||dd}tj ||fdd}n|}| |j }| |}| |}|S)Nrrrdim)sizercloner!rr#shaperepeatr*cattodtyper$r() r0 input_idsr query_embedspast_key_values_length seq_length embeddingsr# batch_sizer3r3r4forward?s(       zBlip2TextEmbeddings.forward)NNNr)__name__ __module__ __qualname____doc__rrD __classcell__r3r3r1r4r,s rcs8eZdZdeffdd ZdejdejfddZZS)Blip2VisionEmbeddingsr.cst||_|j|_|j|_|j|_tt dd|j|_ tj d|j|j|jdd|_ |j|jd|_|jd|_tt d|j|j|_dS)NrF) in_channels out_channels kernel_sizestridebias)rrr.r embed_dim image_size patch_sizer Parameterr*randnclass_embeddingConv2dpatch_embedding num_patches num_positionsposition_embeddingr/r1r3r4rcs  zBlip2VisionEmbeddings.__init__ pixel_valuesreturncCs|jd}|jjj}||j|d}|ddd}|j|dd|}t j ||gdd}||j ddd| dddf|}|S)Nrr=rQrrr5) r9rYweightr=r<flatten transposerWr,r*r;r\r7)r0r]rC target_dtype patch_embeds class_embedsrBr3r3r4rDus  ,zBlip2VisionEmbeddings.forward) rErFrGr rr*TensorrDrIr3r3r1r4rJbsrJcs:eZdZfddZ          d ddZZS) Blip2QFormerEncodercs:t|_tfddtjD|_d|_dS)Ncsg|]}t|qSr3)Blip2QFormerLayer).0 layer_idxr.r3r4 z0Blip2QFormerEncoder.__init__..F) rrr.r ModuleListrangenum_hidden_layerslayergradient_checkpointingr/r1rkr4rs  zBlip2QFormerEncoder.__init__NFTrc  Cs\| rdnd} |r dnd} |rdnd}|rdnd}t|jjD]o}|j|}| r,| |f} |dur4||nd}|dur>||nd}t|jddrbtrb|rTtdd}| ||||||||| }n ||||||||| }|d}|rz||df7}|r| |df} |j r||df}q| r| |f} | st d d ||| | |fDSt ||| | |d S) Nr3rrFzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...rrrrQcss|] }|dur|VqdSNr3)rivr3r3r4 sz.Blip2QFormerEncoder.forward..)last_hidden_statepast_key_values hidden_states attentionscross_attentions) ror.rprqr-r*is_grad_enabledloggerwarning_gradient_checkpointing_funchas_cross_attentiontupler)r0rxattention_mask head_maskencoder_hidden_statesencoder_attention_maskrw use_cacheoutput_attentionsoutput_hidden_states return_dict query_lengthall_hidden_statesall_self_attentionsall_cross_attentionsnext_decoder_cachei layer_modulelayer_head_maskpast_key_value layer_outputsr3r3r4rDs~          zBlip2QFormerEncoder.forward) NNNNNNFFTrrErFrGrrDrIr3r3r1r4rgs  rgcsDeZdZfddZ       d ddZdd Zd d ZZS) rhcs~t|j|_d|_t||_||_||jdkr&t|dd|_d|_ nd|_ t ||_ t ||_ t ||_t ||_dS)NrrT)is_cross_attentionF)rrchunk_size_feed_forward seq_len_dimr attentionrjcross_attention_frequencycrossattentionrr intermediateintermediate_queryr output_queryoutput)r0r.rjr1r3r4rs     zBlip2QFormerLayer.__init__NFrc  Cs:|dur |ddnd} |j||||| d} | d} | dd} | d} |dkr| ddd|ddf}|jrW|dur@td|j||||||d}|d}| |dd} t|j|j|j|}| jd|krt|j |j|j| dd|dddf}t j ||gdd}n t|j |j|j| }|f| } | | f} | S) NrQ)rrrrrz>encoder_hidden_states must be given for cross-attention layers)rr5) rr ValueErrorrrfeed_forward_chunk_queryrrr9feed_forward_chunkr*r;)r0rxrrrrrrrself_attn_past_key_valueself_attention_outputsattention_outputoutputspresent_key_valuequery_attention_outputcross_attention_outputs layer_outputlayer_output_textr3r3r4rDsd    zBlip2QFormerLayer.forwardcC||}|||}|Srs)rrr0rintermediate_outputrr3r3r4rA  z$Blip2QFormerLayer.feed_forward_chunkcCrrs)rrrr3r3r4rFrz*Blip2QFormerLayer.feed_forward_chunk_query)NNNNNFr)rErFrGrrDrrrIr3r3r1r4rhs  Grhcs&eZdZdfdd ZddZZS) ProjLayer皙?-q=csNtt|||_t|_t|||_t||_ tj ||d|_ dSNr) rrrLineardense1 QuickGELUact_fndense2r&r(r$)r0in_dimout_dim hidden_dimdrop_prr1r3r4rNs  zProjLayer.__init__c Cs2|}||}||||||}|Srs)r$r(rrr)r0xx_inr3r3r4rDYs  zProjLayer.forward)rrrr3r3r1r4rMs rc s|eZdZdZeZdeffdd Zeeed    dde j dBde dBde dBd e dBd e eBf d d Z d dZZS)Blip2VisionModelr]r.cs\t|||_|j}t||_tj||jd|_ t ||_ tj||jd|_ | dSr)rrr.rrJrBrr$r% pre_layernormr encoderpost_layernorm post_init)r0r.rRr1r3r4rgs    zBlip2VisionModel.__init__) output_type config_classNrrrr^c Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td||}||}|j||||d}|d}||}|dddddf}||}|s^||f|ddSt |||j |j dS)z Returns: Nz You have to specify pixel_values) inputs_embedsrrrrr)rv pooler_outputrxry) r.rruse_return_dictrrBrrrrrxry) r0r]rrrrxencoder_outputsrv pooled_outputr3r3r4rDrs4     zBlip2VisionModel.forwardcCs|jSrs)rBr0r3r3r4get_input_embeddingssz%Blip2VisionModel.get_input_embeddings)NNNN)rErFrGmain_input_namer rrrrr*rfboolrrDrrIr3r3r1r4rcs*  ,rc seZdZdZdeffdd ZddZddZd d Z dd e j d e e de j dede j f ddZ          dddZZS)Blip2QFormerModelz: Querying Transformer (Q-Former), used in BLIP-2. r.cst|||_t|j|_t|j|_t t d|j |jj|_t|dr-|jdur6tjddd|_n tj|jdd|_|jddit|jj|jj|jjdd d d |_t|j|_|dS) Nr tokenizerzbert-base-uncasedright)truncation_side bos_tokenz[DEC]rr)rrrrr)rrr.rqformer_configrBr vision_configvisual_encoderrrUr*zerosnum_query_tokensr query_tokenshasattrrrfrom_pretrainedadd_special_tokensr proj_layerrgrrr/r1r3r4rs$      zBlip2QFormerModel.__init__cCs|jjSrsrBr!rr3r3r4rsz&Blip2QFormerModel.get_input_embeddingscCs ||j_dSrsr)r0valuer3r3r4set_input_embeddingss z&Blip2QFormerModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrqr prune_heads)r0heads_to_prunerqheadsr3r3r4 _prune_headsszBlip2QFormerModel._prune_headsFr input_shapedevice has_queryr^cCs|dkr|dddddddf}n|dkr(|ddddddf}n td||j|j|jd}d|d}|S)a= Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (`torch.Tensor`): Mask with ones indicating tokens to attend to, zeros for tokens to ignore. input_shape (`tuple[int]`): The shape of the input to the model. device (`torch.device`): The device of the input to the model. Returns: `torch.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`. rKNrQzAWrong shape for input_ids (shape {}) or attention_mask (shape {})r_g?g)r6rformatr9r<r=)r0rrrrextended_attention_maskr3r3r4get_extended_attention_masks   z-Blip2QFormerModel.get_extended_attention_maskNc s^j|ddd} | j} | j} | jd} tj| jdftj dj}tj || j gdd}|dur9|nj j }| durC| nj j} | durM| nj j} |durb|ddjd j jnd}jjd}j| j|d }|dd }|\} }|j}|j}|}|durtj| ||f|d }|||}|durt|tr|d\}}}n|\}}}||f}t|tr͇fd d|D}n|durtj||d }|}n|}nd}|j jj}j||||||||| | |d }|d}|dddddf}| s!|ddd|ddfSt|||j|j|j |j!dS)a  encoder_hidden_states (`torch.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, `optional`): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, `optional`): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`tuple(tuple(torch.Tensor))` of length `config.n_layers` with each tuple having 4 tensors of: shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, `optional`): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). ptT)return_tensorspaddingrrr_r5NrQ)r>r?r@r)rcsg|]}|qSr3)invert_attention_mask)rimaskrr3r4rlRrmz-Blip2QFormerModel.forward..) rrrrrwrrrrr)rvrrwrxryrz)"rr<rr>r9r*onesrr7longr;rr.rrrrrBrrvr isinstancelistr get_head_maskrrprrrrwrxryrz) r0 text_input image_inputrrrrwrrrrtextr>rC query_attsrr@rembedding_outputrrArimage_embeds_frozenrencoder_batch_sizeencoder_sequence_length_encoder_hidden_shapeencoder_extended_attention_maskrsequence_outputrr3rr4rDs!  &$         zBlip2QFormerModel.forward)F) NNNNNNNNNN)rErFrGrHrrrrrr*rfrintrrrrDrIr3r3r1r4rs:  /r)$r*r transformersrtransformers.activationsrrtransformers.modeling_outputsrrr/transformers.models.blip_2.configuration_blip_2rr *transformers.models.blip_2.modeling_blip_2r r r r rtransformers.pytorch_utilsrtransformers.utilsrr get_loggerrEr|ModulerrJrgrhrrrr3r3r3r4s"     6 dgA