o wi@sdZddlZddlmZddlmZmZmZddlZddl Zddlm Z ddl m Z ddl mZmZdd lmZdd lmZdd lmZdd lmZmZmZmZdd lmZmZddlmZm Z m!Z!ddl"m#Z#m$Z$m%Z%m&Z&ddl'm(Z(m)Z)e%*e+Z,ee$ddGddde#Z-Gddde j.Z/Gddde j.Z0Gddde j.Z1de0iZ2Gddde j.Z3Gdd d e j.Z4Gd!d"d"e j.Z5Gd#d$d$eZ6Gd%d&d&e j.Z7e$Gd'd(d(eZ8Gd)d*d*e j.Z9Gd+d,d,e j.Z: -dKd.e j.d/ej;d0ej;d1ej;d2eej;d3eGd9d:d:eZ?Gd;d<dd>e j.ZAe$d?dGd@dAdAe8ZBGdBdCdCe j.ZCe$dDdGdEdFdFe8ZDe$dGdGdHdIdIe8eZEgdJZFdS)LzPyTorch GIT model.N) dataclass)CallableOptionalUnion)nn)ACT2FN)Cache DynamicCache)GenerationMixin)_prepare_4d_attention_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPastBaseModelOutputWithPoolingCausalLMOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputauto_docstringlogging torch_int) GitConfigGitVisionConfigz} Base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. ) custom_introc@sjeZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dS)GitVisionModelOutputz image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. N image_embedslast_hidden_state. hidden_states attentions) __name__ __module__ __qualname____doc__r rtorch FloatTensor__annotations__r!r"tupler#r,r,a/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/transformers/models/git/modeling_git.pyr3s rc s\eZdZdZfddZ    d deejdeejdeejd e d ej f d d Z Z S) GitEmbeddingsz;Construct the embeddings from word and position embeddings.csttj|j|j|jd|_t|j|j|_ tj |j|j d|_ t |j |_t|dd|_|jdt|jddddS) N) padding_idxepsposition_embedding_typeabsolute position_idsrF persistent)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutgetattrr2register_bufferr(arangeexpandselfconfig __class__r,r-r:Is  zGitEmbeddings.__init__Nr input_idsr4 inputs_embedspast_key_values_lengthreturnc Cs|dur |}n|dd}|d}|dur&|jdd|||f}|dur0||}n|}|jdkr@||}||7}||}||}|S)Nr6rr3)sizer4r?r2rArBrF) rLrPr4rQrR input_shape seq_length embeddingsrAr,r,r-forwardXs      zGitEmbeddings.forward)NNNr) r$r%r&r'r:rr( LongTensorr)intTensorrX __classcell__r,r,rNr-r.Fs$ r.cseZdZdfdd ZdejdejfddZ     dd ejd eejd eejd ee d ee dee de ejfddZ Z S)GitSelfAttentionNcsVt|j|jdkrt|dstd|jd|jd||_|dur1td|j j d|j|_t |j|j|_ |j|j |_ t |jj|jjdd |_|jdura|j|j9_t|j|j |_t|j|j |_t|j|j |_t|j|_|pt|d d |_|jd ks|jd kr|j|_td|jd |j |_dSdS)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()zInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.rr2r3 relative_keyrelative_key_query) r9r:r=num_attention_headshasattr ValueError layer_idxlogger warning_oncerOr$rZattention_head_size all_head_size vision_config image_size patch_sizeimage_patch_tokensnum_image_with_embeddingrLinearquerykeyvaluerDattention_probs_dropout_probrFrGr2r@r;distance_embeddingrLrMr2rfrNr,r-r:ws:   zGitSelfAttention.__init__xrScCs6|dd|j|jf}||}|ddddS)Nr6rr`rr)rTrcriviewpermute)rLrw new_x_shaper,r,r-transpose_for_scoress z%GitSelfAttention.transpose_for_scoresFr"attention_mask head_maskpast_key_valueoutput_attentionspixel_values_presentc Cs||}|r |jnd}|||} |||} |durt|| dddd|dddf| dddd|dddf|j\} } tj| ddddd|ddf| gdd} tj| ddddd|ddf| gdd} ||} t | | dd}|j dks|j dkr | j d| j d}}|durtj |dtj|jd dd}ntj|tj|jd dd}tj|tj|jd dd}||}|||jd}|j| jd }|j dkrtd | |}||}n|j dkr td | |}td | |}|||}|t|j}|dur||}tjj|dd}||}|dur1||}t || }|dddd }| dd|j!f}||}|rY||fn|f}||f}|S)Nrr`dimr6rarbrdtypedevicerzbhld,lrd->bhlrzbhrd,lrd->bhlrr)"rqrnr{rrrsupdaterfr(catmatmul transposer2shapetensorlongrrxrIrur@toreinsummathsqrtrir functionalsoftmaxrFry contiguousrTrj)rLr"r|r}r~rrmixed_query_layercutoff key_layer value_layerkey_layer_pastvalue_layer_past query_layerattention_scores query_length key_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layernew_context_layer_shapeoutputsr,r,r-rXsX @..           zGitSelfAttention.forwardNNNNNFF)r$r%r&r:r(r[r{rr)r boolr+rXr\r,r,rNr-r]vs."r]c8eZdZfddZdejdejdejfddZZS) GitSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNr0) r9r:rrpr=denserBrCrDrErFrKrNr,r-r: zGitSelfOutput.__init__r" input_tensorrScC&||}||}|||}|SNrrFrBrLr"rr,r,r-rX  zGitSelfOutput.forwardr$r%r&r:r(r[rXr\r,r,rNr-r $reagercsxeZdZdfdd ZddZ     ddejdeejd eejd ee d ee d ee d e ejfddZ Z S) GitAttentionNcs6tt|j|||d|_t||_t|_dS)N)r2rf) r9r:GIT_SELF_ATTENTION_CLASSES_attn_implementationrLroutputset pruned_headsrvrNr,r-r:s   zGitAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr)lenrrLrcrirrrqrrrsrrrjunion)rLheadsindexr,r,r- prune_heads s zGitAttention.prune_headsFr"r|r}r~rrrSc Cs:|||||||}||d|}|f|dd} | S)Nrr)rLr) rLr"r|r}r~rr self_outputsattention_outputrr,r,r-rXs zGitAttention.forwardrr)r$r%r&r:rr(r[rr)r rr+rXr\r,r,rNr-rs. rc2eZdZfddZdejdejfddZZS)GitIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) r9r:rrpr=intermediate_sizer isinstance hidden_actstrrintermediate_act_fnrKrNr,r-r:3s   zGitIntermediate.__init__r"rScCs||}||}|Sr)rrrLr"r,r,r-rX;s  zGitIntermediate.forwardrr,r,rNr-r2s rcr) GitOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) r9r:rrprr=rrBrCrDrErFrKrNr,r-r:CrzGitOutput.__init__r"rrScCrrrrr,r,r-rXIrzGitOutput.forwardrr,r,rNr-rBrrcsxeZdZdfdd Z     ddejdeejdeejdeed ee d ee d e ejfd d Z ddZ Z S)GitLayerNcs>t|j|_d|_t||d|_t||_t||_ dS)Nr)rf) r9r:chunk_size_feed_forward seq_len_dimr attentionr intermediaterr)rLrMrfrNr,r-r:Qs  zGitLayer.__init__Fr"r|r}r~rrrSc Cs^|j||||||d}|d}|dd} |d} t|j|j|j|} | f| } | | f} | S)N)rr~rrrr6)rrfeed_forward_chunkrr) rLr"r|r}r~rrself_attention_outputsrrpresent_key_value layer_outputr,r,r-rXYs"    zGitLayer.forwardcCs||}|||}|Sr)rr)rLrintermediate_outputrr,r,r-r{s  zGitLayer.feed_forward_chunkrr)r$r%r&r:r(r[rr)r rr+rXrr\r,r,rNr-rPs.  "rcseZdZfddZ        ddejdeejdeejd eee e e ejfd ee d ee d ee d ee dee dee eje ffddZ ZS) GitEncoderc:t|_tfddtjD|_d|_dS)Ncsg|]}t|qSr,)r).0irMr,r- sz'GitEncoder.__init__..F) r9r:rMr ModuleListrangenum_hidden_layerslayergradient_checkpointingrKrNrr-r:   zGitEncoder.__init__NFTr"r|r}past_key_values use_cacheroutput_hidden_statesr return_dictrSc Cs6|jr|jr|rtdd}d} |r,t|ts,d} |dur"t}n t|}td|r0dnd} |r6dnd} d} t|j D]1\}}|rJ| |f} |durR||nd}|||||||}|d}|rg|d} |rp| |df} q?|rx| |f} |r|| nd}| r| }| st d d ||| | fDSt ||| | d S) NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FTzWe detected that you are passing `past_key_values` as a tuple of tuples. This is deprecated and will be removed in v4.47. Please convert your cache or use an appropriate `Cache` class (https://huggingface.co/docs/transformers/kv_cache#legacy-cache-format)r,rr6rcs|] }|dur|VqdSrr,rvr,r,r- sz%GitEncoder.forward..r!rr"r#) rtrainingrgrhrr r from_legacy_cache enumeraterto_legacy_cacher+r)rLr"r|r}rrrrrrreturn_legacy_cacheall_hidden_statesall_self_attentionsnext_decoder_cacher layer_modulelayer_head_mask layer_outputs next_cacher,r,r-rXsn         zGitEncoder.forward)NNNNFFFT)r$r%r&r:r(r[rr)rr r+rrrXr\r,r,rNr-rs>     rc@s(eZdZeZdZdZdZdZddZ dS)GitPreTrainedModelgitTcCst|tr)tjj|jd|jjdtjj|jj |jjdtjj|j j |jjdt|tj rI|j j jd|jjd|j durG|j j dSdSt|tjrl|j j jd|jjd|jdurj|j j |jdSdSt|tjr|j j |j j ddSdS)zInitialize the weights)meanstd)rNg?)rGitVisionEmbeddingsrinitnormal_class_embeddingrMinitializer_rangepatch_embeddingweightposition_embeddingrpdatabiaszero_r;r/rBfill_)rLmoduler,r,r- _init_weightss$       z GitPreTrainedModel._init_weightsN) r$r%r&r config_classbase_model_prefixsupports_gradient_checkpointing_supports_cache_class_supports_quantized_cacherr,r,r,r-rs rcsXeZdZdeffdd Zdejdededejfdd Zdd ej dejfd d Z Z S)rrMcst||_|j|_|j|_|j|_tt |j|_ tj |j |j|j|jdd|_|j|jd|_|jd|_t|j|j|_|jdt |jddddS)NF) in_channels out_channels kernel_sizestrider r`rr4r5r7)r9r:rMr= embed_dimrlrmr Parameterr(randnrConv2d num_channelsr num_patches num_positionsr;r rHrIrJrKrNr,r-r:s"  "zGitVisionEmbeddings.__init__rWheightwidthrSc Cs|jdd}|jjd}|jdd}tjs(||kr(||kr(||jS|ddddf}|ddddf}|jd} ||j} ||j} t |d} | d| | | }| dddd}t j j|| | fdd d }| dddddd| }tj||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNr6g?rr`bicubicF)rTmode align_cornersr)rr r unsqueezer(jit is_tracingr4rmrreshaperyrr interpolaterxr) rLrWr r!rr rclass_pos_embedpatch_pos_embedr new_height new_widthsqrt_num_positionsr,r,r-interpolate_pos_encodings*      z,GitVisionEmbeddings.interpolate_pos_encodingF pixel_valuesc Cs|j\}}}}|s&||jks||jkr&td|d|d|jd|jd |jjj}||j|d}|ddd}|j |dd} t j | |gdd } |r[| | | ||} | S| ||j} | S) NzInput image size (*z) doesn't match model ().rr`rr6r)rrlrerrrrflattenrrrJr(rr/r r4) rLr0r/ batch_size_r r! target_dtype patch_embeds class_embedsrWr,r,r-rX9s  zGitVisionEmbeddings.forwardF) r$r%r&rr:r(r[rZr/r)rXr\r,r,rNr-rs )rcr) GitVisionMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSr) r9r:rMrr activation_fnrrpr=rfc1fc2rKrNr,r-r:Ms  zGitVisionMLP.__init__r"rScCs"||}||}||}|Sr)r<r;r=rr,r,r-rXTs   zGitVisionMLP.forwardrr,r,rNr-r:Ls r:rrrqrrrsr|scalingrFc Ks|t||dd|}|dur||}tjj|dtjd|j}tjj |||j d}t||} | dd } | |fS)Nr6r)rr)prrr`) r(rrrrrfloat32rrrFrr) rrqrrrsr|r>rFkwargs attn_weights attn_outputr,r,r-eager_attention_forward\s  rDcsheZdZdZfddZ   d dejdeejdeejd eed e ejeejff d d Z Z S)GitVisionAttentionz=Multi-headed attention from 'Attention Is All You Need' papercst||_|j|_|j|_|j|j|_|j|j|jkr-td|jd|jd|jd|_ |j |_ d|_ t |j|j|_t |j|j|_t |j|j|_t |j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: r2gF)r9r:rMr=rrc num_headshead_dimrescaleattention_dropoutrF is_causalrrpk_projv_projq_projout_projrKrNr,r-r:us$   zGitVisionAttention.__init__NFr"r|causal_attention_maskrrSc CsL|j\}}}||}||} ||} ||||j|jdd}| |||j|jdd} | |||j|jdd} |jj dkrY|durR|durR||}n |durX|}n|du|_ t } |jj dkrz|jj dkrt|rtt dnt|jj } | ||| | ||j |j|jsdn|jd \} } | |||} || } |sd} | | fS) z#Input shape: Batch x Time x Channelrr`flash_attention_2Nrsdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.r)rJr>rF)rrMrKrLrxrFrGrrMrrJrDrgrhrrHrrFr(rrN)rLr"r|rOrr4rVrquerieskeysvaluesattention_interfacerCrBr,r,r-rXsH          zGitVisionAttention.forward)NNF) r$r%r&r'r:r(r[rrr+rXr\r,r,rNr-rErs" rEc sTeZdZdeffdd Z d dejdejdejdeed e ej f d d Z Z S) GitVisionEncoderLayerrMcsRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSr) r9r:r=rrE self_attnrrBrC layer_norm1r:mlp layer_norm2rKrNr,r-r:s   zGitVisionEncoderLayer.__init__Fr"r|rOrrScCsd|}||}|j||||d\}}||}|}||}||}||}|f}|r0||f7}|S)aI Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )r"r|rOr)rXrWrZrY)rLr"r|rOrresidualrBrr,r,r-rXs"     zGitVisionEncoderLayer.forwardr9) r$r%r&rr:r(r[rrr+r)rXr\r,r,rNr-rVs rVcsteZdZdZdeffdd Z     ddeejdeejdee d ee d ee d e e e ff d d Z ZS)GitVisionEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`GitVisionEncoderLayer`]. Args: config: GitVisionConfig rMcr)Ncsg|]}tqSr,)rVrr5rr,r-rsz-GitVisionEncoder.__init__..F) r9r:rMrrrrlayersrrKrNrr-r:rzGitVisionEncoder.__init__Nr|rOrrrrSc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|r"dnd}|r(dnd}|} t|jD] \} } |r<|| f}| | |||d} | d} |rQ|| df}q1|rY|| f}|sgtdd| ||fDSt| ||dS) a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr,)rrrcsrrr,rr,r,r-rGsz+GitVisionEncoder.forward..r!r"r#)rMrruse_return_dictrr^r+r) rLrQr|rOrrrencoder_statesall_attentionsr"idx encoder_layerrr,r,r-rXs6&    zGitVisionEncoder.forward)NNNNN)r$r%r&r'rr:rr(r[rrr+rrXr\r,r,rNr-r\s*  r\csreZdZdeffdd Ze     ddeejdee dee d ee d ee d e e e ff d d Z ZS)GitVisionTransformerrMcsRt||_|j}t||_tj||jd|_ t ||_ tj||jd|_ dSr) r9r:rMr=rrWrrBrC pre_layrnormr\encoderpost_layernorm)rLrMrrNr,r-r:Os   zGitVisionTransformer.__init__NFr0rrr/rrSc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td|j||d}||}|j||||d}|d}||}|sO|f|ddSt ||j |j dS)Nz You have to specify pixel_valuesr/)rQrrrrrr_) rMrrr`rerWrfrgrhrr"r#) rLr0rrr/rr"encoder_outputsr!r,r,r-rXYs.   zGitVisionTransformer.forwardNNNFN)r$r%r&rr:rrr(r)rrr+rrXr\r,r,rNr-reMs*  rezY The vision model from CLIP, used in GIT, without any head or projection on top. cseZdZeZdZdeffdd ZdejfddZ e    dde e j d e ed e ed ed e edeeeff ddZZS)GitVisionModelr0rMcs"t|t||_|dSr)r9r:re vision_model post_initrKrNr,r-r:s   zGitVisionModel.__init__rScCs |jjjSr)rmrWrrLr,r,r-get_input_embeddings z#GitVisionModel.get_input_embeddingsNFrrr/rcCs(|dur|n|jj}|j|||||dS)a{ Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, GitVisionModel >>> processor = AutoProcessor.from_pretrained("microsoft/git-base") >>> model = GitVisionModel.from_pretrained("microsoft/git-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state ```N)r0rrr/r)rMr`rm)rLr0rrr/rr,r,r-rXszGitVisionModel.forwardrk)r$r%r&rrmain_input_namer:rModulerprrr(r)rrr+rrXr\r,r,rNr-rls0 rlcs8eZdZdeffdd ZdejdejfddZZS) GitProjectionrMcs@t||_tt|jj|jtj|j|jj d|_ dSr) r9r:rMr Sequentialrprkr=rBrCvisual_projectionrKrNr,r-r:s  zGitProjection.__init__rWrScCs ||Sr)rv)rLrWr,r,r-rXrqzGitProjection.forward) r$r%r&rr:r(r[rXr\r,r,rNr-rtsrtzy The bare GIT Model transformer consisting of a CLIP image encoder and text decoder outputting raw hidden-states cseZdZfddZddZddZddZd ed ej d ej d ej fd dZ d!ddZ e            d"deej deej deej deej deej deej deeeeejfdeedeedeededeed eeej effdd ZZS)#GitModelcsrt|_t|_tj|_t|_ t |_ j dur3t fddtj D|_|dS)Nc3s(|]}ttddjjVqdS)rN)rrr(zerosrkr=r]rr,r-rs  z$GitModel.__init__..)r9r:rMr.rWrlrk image_encoderrrgrtrvror ParameterListrimg_temperal_embeddingrnrKrNrr-r:s       zGitModel.__init__cCs|jjSrrWr?ror,r,r-rpszGitModel.get_input_embeddingscCs ||j_dSrr|)rLrsr,r,r-set_input_embeddingss zGitModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrgrrr)rLheads_to_prunerrr,r,r- _prune_headsszGitModel._prune_headsrTrrrScCs4tjtj||||ddd}||dktd}|S)Nrrr)diagonal-inf)r(triuones masked_fillfloat)rLrTrrmaskr,r,r-_generate_future_maskszGitModel._generate_future_maskNcCs|jd}|jd}|j}|j} tj||f|| d} tj|||ftd|j| d} tj||f| |jd} |dkrLtj|jd|jd|f| |jd}tj| | fdd} tj| || fdd}tj| |fdddddf}|durtj|jd|jdfd|d}|jtj krt d tj ||jd }td||<| |jd|||||f}| }|ddddd|f}|dddddf}|||ddddd|f<|dddddddf}|S) NrrrrrrF) fill_valuerz1Memory key padding mask must be a boolean tensor.r)rrrr(rxfullrrrrre zeros_likerJclone)rLtgtmemorytgt_maskrRmemory_key_padding_masknum_tgt num_memoryrrtop_left top_right bottom_leftleftrightfull_attention_maskzero_negative_infinity origin_leftrr,r,r-create_attention_masksP      zGitModel.create_attention_maskFrPr|r4r0r}rQrrrrr/rc  Cs| dur| n|jj} | dur| n|jj} |dur|n|jj}| dur$| n|jj} |dur4|dur4td|durC||||} n|durP|dd} ntd| d}d}|durpt|t sl|ddj dn| }| ||jj }d}|dur|jdkr|j|| d j}n=|jd krg}t|j dD]"}|j|dd|ddddf| d j}||j|7}||qtj|dd }ntd ||}|j||||d }|durtj|j dd|j df|j|jd}||d|ddd}tj||fdd }|||j|j}|j||||d}|dur\t||j| dd|j}|dkrB|dddd| dddf}n|dddd| d d| d df|7<|j |||||| | | |dud }|d}| s||f|ddSt!||j"|j#|j$dS)a Examples: ```python >>> from transformers import AutoProcessor, AutoModel >>> import requests >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base") >>> model = AutoModel.from_pretrained("microsoft/git-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "this is an image of two cats" >>> inputs = processor(images=image, text=text, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer6z5You have to specify either input_ids or inputs_embedsrrr`rirz#pixel_values must be of rank 4 or 5)rPr4rQrRr)rrrrR)tgt_len)r|r}rrrrrrr)%rMrrrr`re%warn_if_padding_and_no_attention_maskrTrr rget_seq_length get_head_maskrndimryr!rr{appendr(rrvrWrxrrrepeatrrr rrgrrr"r#)rLrPr|r4r0r}rQrrrrr/rrUrVrRprojected_visual_featuresvisual_features frame_idxvisual_features_frameembedding_outputr"rcombined_attention_maskexpanded_attn_maskrjsequence_outputr,r,r-rX)s%          $4 zGitModel.forwardr) NNNNNNNNNNFN)r$r%r&r:rpr}rrZr(rrr[rrrrrr listr)rr+rrXr\r,r,rNr-rws^   2     rwz` GIT Model with a `language modeling` head on top for autoregressive language modeling. c seZdZdgZfddZddZddZe            dd ee j d ee j d ee j d ee j dee j dee j dee j dee e e e j fdeedeedeededeede ee j effddZ dddZddZZS) GitForCausalLMz output.weightcs4t|t||_t|j|j|_| dSr) r9r:rwrrrpr=r<rrnrKrNr,r-r:s   zGitForCausalLM.__init__cCs|jSrrror,r,r-get_output_embeddingssz$GitForCausalLM.get_output_embeddingscCs ||_dSrr)rLnew_embeddingsr,r,r-set_output_embeddingsrqz$GitForCausalLM.set_output_embeddingsNFrPr|r4r0r}rQlabelsrrrrr/rrScKs| dur| n|jj} |durd} |j|||||||| | | | | d }|d}||}d}|durl|jjjdjjj}|dd|dddf }|ddddf }|j | d|jj | dfd|jj i|}| s|f|dd}|dur|f|S|St |||j|j|jdS) a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` Examples: Image captioning example: ```python >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> import requests >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-coco") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-coco") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> pixel_values = processor(images=image, return_tensors="pt").pixel_values >>> generated_ids = model.generate(pixel_values=pixel_values, max_length=50) >>> generated_caption = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> print(generated_caption) two cats sleeping on a pink blanket next to remotes. ``` Visual question answering (VQA) example: ```python >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> from huggingface_hub import hf_hub_download >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-textvqa") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-textvqa") >>> file_path = hf_hub_download(repo_id="nielsr/textvqa-sample", filename="bus.png", repo_type="dataset") >>> image = Image.open(file_path).convert("RGB") >>> pixel_values = processor(images=image, return_tensors="pt").pixel_values >>> question = "what does the front of the bus say at the top?" >>> input_ids = processor(text=question, add_special_tokens=False).input_ids >>> input_ids = [processor.tokenizer.cls_token_id] + input_ids >>> input_ids = torch.tensor(input_ids).unsqueeze(0) >>> generated_ids = model.generate(pixel_values=pixel_values, input_ids=input_ids, max_length=50) >>> print(processor.batch_decode(generated_ids, skip_special_tokens=True)) ['what does the front of the bus say at the top? special'] ``` Video captioning example: ```python >>> import av >>> import numpy as np >>> from PIL import Image >>> from huggingface_hub import hf_hub_download >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-vatex") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-vatex") >>> # set seed for reproducibility >>> np.random.seed(45) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`list[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # load video >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample frames >>> num_frames = model.config.num_image_with_embedding >>> indices = sample_frame_indices( ... clip_len=num_frames, frame_sample_rate=4, seg_len=container.streams.video[0].frames ... ) >>> frames = read_video_pyav(container, indices) >>> pixel_values = processor(images=list(frames), return_tensors="pt").pixel_values >>> generated_ids = model.generate(pixel_values=pixel_values, max_length=50) >>> print("Generated caption:", processor.batch_decode(generated_ids, skip_special_tokens=True)) Generated caption: ['a woman is sitting at a table and she is talking about the food she is holding.'] ``` NF) r|r4r0r}rQrrrrr/rrr6rr<)losslogitsrr"r#)rMr`rrrgrrrLrnr loss_functionrxr<rrr"r#)rLrPr|r4r0r}rQrrrrrr/rrArrrrnum_image_tokensshifted_logitsrr,r,r-rXsV zGitForCausalLM.forwardc Ksv|dur#|}|jd|kr|}n|jdd}|dd|df}|j}|dur/||}|||dd||dS)Nrr0)rPr|r0rr)rrnew_onesget) rLrPrr|rrA past_lengthremove_prefix_lengthrUr,r,r-prepare_inputs_for_generations  z,GitForCausalLM.prepare_inputs_for_generationcs.d}|D]}|tfdd|Df7}q|S)Nr,c3s$|] }|d|jVqdS)rN) index_selectrr)r past_statebeam_idxr,r-rs"z0GitForCausalLM._reorder_cache..)r+)rLrrreordered_past layer_pastr,rr-_reorder_caches zGitForCausalLM._reorder_cache) NNNNNNNNNNNFN)NNN)r$r%r&_tied_weights_keysr:rrrrr(r[rr rrr+rrXrrr\r,r,rNr-rsh        E r)rrwrrl)r)Gr'r dataclassesrtypingrrrr(torch.utils.checkpointr activationsr cache_utilsr r generationr modeling_attn_mask_utilsr modeling_layersr modeling_outputsrrrrmodeling_utilsrr pytorch_utilsrrrutilsrrrrconfiguration_gitrr get_loggerr$rgrrsr.r]rrrrrrrrrr:r[rrDrErVr\rerlrtrwr__all__r,r,r,r-s        0v31ZS P3X65