o iD@sdZddlZddlmZddlmZmZmZddlZddlm Z ddl m Z ddl m Z mZdd lmZdd lmZdd lmZdd lmZmZmZmZdd lmZmZddlmZmZm Z ddl!m"Z"m#Z#m$Z$m%Z%m&Z&ddl'm(Z(ddl)m*Z*m+Z+e%,e-Z.ee#ddGddde"Z/Gddde j0Z1Gddde j0Z2Gddde j0Z3de2iZ4Gddde j0Z5Gd d!d!e j0Z6Gd"d#d#e j0Z7Gd$d%d%eZ8Gd&d'd'e j0Z9e#Gd(d)d)eZ:Gd*d+d+e j0Z;Gd,d-d-e j0Z< .dLd/e j0d0ej=d1ej=d2ej=d3eej=d4e>d5e>fd6d7Z?Gd8d9d9e j0Z@Gd:d;d;eZAGdd?d?e j0ZCe#d@dGdAdBdBe:ZDGdCdDdDe j0ZEe#dEdGdFdGdGe:ZFe#dHdGdIdJdJe:eZGgdKZHdS)MzPyTorch GIT model.N) dataclass)CallableOptionalUnion)nn)ACT2FN)Cache DynamicCache)GenerationMixin)_prepare_4d_attention_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPastBaseModelOutputWithPoolingCausalLMOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputauto_docstringcan_return_tuplelogging torch_int)deprecate_kwarg) GitConfigGitVisionConfigz} Base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. ) custom_introc@sjeZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dS)GitVisionModelOutputz image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. N image_embedslast_hidden_state. hidden_states attentions) __name__ __module__ __qualname____doc__r"rtorch FloatTensor__annotations__r#r$tupler%r.r.X/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/git/modeling_git.pyr!4s r!c s\eZdZdZfddZ    d deejdeejdeejd e d ej f d d Z Z S) GitEmbeddingsz;Construct the embeddings from word and position embeddings.csttj|j|j|jd|_t|j|j|_ tj |j|j d|_ t |j |_t|dd|_|jdt|jddddS) N) padding_idxepsposition_embedding_typeabsolute position_idsrF persistent)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutgetattrr4register_bufferr*arangeexpandselfconfig __class__r.r/r<Js  zGitEmbeddings.__init__Nr input_idsr6 inputs_embedspast_key_values_lengthreturnc Cs|dur |}n|dd}|d}|dur&|jdd|||f}|dur0||}n|}|jdkr@||}||7}||}||}|S)Nr8rr5)sizer6rAr4rCrDrH) rNrRr6rSrT input_shape seq_length embeddingsrCr.r.r/forwardYs      zGitEmbeddings.forward)NNNr) r&r'r(r)r<rr* LongTensorr+intTensorrZ __classcell__r.r.rPr/r0Gs$ r0cs~eZdZdfdd Zedddd     dd ejd eejd eejdee d ee d ee de ejfddZ Z S)GitSelfAttentionNcsVt|j|jdkrt|dstd|jd|jd||_|dur1td|j j d|j|_t |j|j|_ |j|j |_ t |jj|jjdd |_|jdura|j|j9_t|j|j |_t|j|j |_t|j|j |_t|j|_|pt|d d |_|jd ks|jd kr|j|_td|jd |j |_dSdS)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()zInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.rr4r5 relative_keyrelative_key_query) r;r<r?num_attention_headshasattr ValueError layer_idxlogger warning_oncerQr&r\attention_head_size all_head_size vision_config image_size patch_sizeimage_patch_tokensnum_image_with_embeddingrLinearquerykeyvaluerFattention_probs_dropout_probrHrIr4rBr=distance_embeddingrNrOr4rhrPr.r/r<xs:   zGitSelfAttention.__init__past_key_valuepast_key_values4.58new_nameversionFr$attention_mask head_maskoutput_attentionspixel_values_presentrUc Cs|j\}}} |||d|j|jdd} |r|jnd} |||d|j|jdd} |||d|j|jdd} |dur| | dddd| dddf| dddd| dddf|j \}}t j | ddddd| ddf|gdd} t j | ddddd| ddf|gdd} t | | dd}|jdks|jdkr*| jd| jd}}|durt j|dt j|jd dd}nt j|t j|jd dd}t j|t j|jd dd}||}|||jd}|j| jd }|jdkrt d | |}||}n|jdkr*t d | |}t d | |}|||}|t|j}|dur;||}tjj|dd}||}|durQ||}t || }|dddd }| dd|j!f}||}||fS)Nr8rrbrdimrcrddtypedevicerzbhld,lrd->bhlrzbhrd,lrd->bhlrr)"shapersviewrerk transposerprtruupdaterhr*catmatmulr4tensorlongrrKrwrBtoreinsummathsqrtr functionalsoftmaxrHpermute contiguousrVrl)rNr$rrrzrr batch_sizerX_ query_layercutoff key_layer value_layerkey_layer_pastvalue_layer_pastattention_scores query_length key_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layernew_context_layer_shaper.r.r/rZsf @..         zGitSelfAttention.forwardNNNNNFF)r&r'r(r<rr*r]rr+r boolr-rZr^r.r.rPr/r_ws. "r_c8eZdZfddZdejdejdejfddZZS) GitSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNr2) r;r<rrrr?denserDrErFrGrHrMrPr.r/r< zGitSelfOutput.__init__r$ input_tensorrUcC&||}||}|||}|SNrrHrDrNr$rr.r.r/rZ  zGitSelfOutput.forwardr&r'r(r<r*r]rZr^r.r.rPr/r $reagercseZdZdfdd ZddZedddd     dd ejd eej d eej dee dee dee de ejfddZ ZS) GitAttentionNcs6tt|j|||d|_t||_t|_dS)N)r4rh) r;r<GIT_SELF_ATTENTION_CLASSES_attn_implementationrNroutputset pruned_headsrxrPr.r/r<s   zGitAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr)lenrrNrerkrrrsrtrurrrlunion)rNheadsindexr.r.r/ prune_headss zGitAttention.prune_headsryrzr{r|Fr$rrrrrUc Cs,|||||||\}}|||} | |fSr)rNr) rNr$rrrzrr attn_outputself_attn_weightsattention_outputr.r.r/rZ s  zGitAttention.forwardrr)r&r'r(r<rrr*r]rr+r rr-rZr^r.r.rPr/rs0 rc2eZdZfddZdejdejfddZZS)GitIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) r;r<rrrr?intermediate_sizer isinstance hidden_actstrrintermediate_act_fnrMrPr.r/r<8s   zGitIntermediate.__init__r$rUcCs||}||}|Sr)rrrNr$r.r.r/rZ@s  zGitIntermediate.forwardrr.r.rPr/r7s rcr) GitOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) r;r<rrrrr?rrDrErFrGrHrMrPr.r/r<HrzGitOutput.__init__r$rrUcCrrrrr.r.r/rZNrzGitOutput.forwardrr.r.rPr/rGrrcseZdZdfdd Zedddd     dd ejd eejd eejdee d ee d ee de ejfddZ ddZ ZS)GitLayerNcs>t|j|_d|_t||d|_t||_t||_ dS)Nr)rh) r;r<chunk_size_feed_forward seq_len_dimr attentionr intermediaterr)rNrOrhrPr.r/r<Vs  zGitLayer.__init__ryrzr{r|Fr$rrrrrUc Cs6|j||||||d\}}t|j|j|j|} | |fS)N)rrzr)rrfeed_forward_chunkrr) rNr$rrrzrrrself_attention_weights layer_outputr.r.r/rZ^s   zGitLayer.forwardcCs||}|||}|Sr)rr)rNrintermediate_outputrr.r.r/rws  zGitLayer.feed_forward_chunkrr)r&r'r(r<rr*r]rr+r rr-rZrr^r.r.rPr/rUs0  rcseZdZfddZ        ddejdeejdeejd eee e e ejfd ee d ee d ee d ee dee dee eje ffddZ ZS) GitEncoderc:t|_tfddtjD|_d|_dS)Ncsg|]}t|qSr.)r).0irOr.r/ sz'GitEncoder.__init__..F) r;r<rOr ModuleListrangenum_hidden_layerslayergradient_checkpointingrMrPrr/r<~   zGitEncoder.__init__NFTr$rrrz use_cacheroutput_hidden_statesr return_dictrUc Cs|jr|jr|rtdd}|r|durt|jd}|rdnd} |r%dnd} t|jD]+\} } |r7| |f} |dur?|| nd}| ||||||}|d}|rW| |df} q,|r_| |f} | sntdd||| | fDSt ||| | d S) NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Frr.rrcss|] }|dur|VqdSrr.)rvr.r.r/ sz%GitEncoder.forward..r#rzr$r%) rtrainingrirjr rO enumeraterr-r)rNr$rrrzrrrrrall_hidden_statesall_self_attentionsr layer_modulelayer_head_mask layer_outputsr.r.r/rZsT         zGitEncoder.forward)NNNNFFFT)r&r'r(r<r*r]rr+rr r-rrrZr^r.r.rPr/r}s>     rc@s&eZdZUeed<dZdZddZdS)GitPreTrainedModelrOgitTcCst|tr)tjj|jd|jjdtjj|jj |jjdtjj|j j |jjdt|tj rI|j j jd|jjd|j durG|j j dSdSt|tjrl|j j jd|jjd|jdurj|j j |jdSdSt|tjr|j j |j j ddSdS)zInitialize the weights)meanstd)rNg?)rGitVisionEmbeddingsrinitnormal_class_embeddingrOinitializer_rangepatch_embeddingweightposition_embeddingrrdatabiaszero_r=r1rDfill_)rNmoduler.r.r/ _init_weightss$       z GitPreTrainedModel._init_weightsN)r&r'r(rr,base_model_prefixsupports_gradient_checkpointingr r.r.r.r/rs  rcsXeZdZdeffdd Zdejdededejfdd Zdd ej dejfd d Z Z S)rrOcst||_|j|_|j|_|j|_tt |j|_ tj |j |j|j|jdd|_|j|jd|_|jd|_t|j|j|_|jdt |jddddS)NF) in_channels out_channels kernel_sizestriderrbrr6r7r9)r;r<rOr? embed_dimrnror Parameterr*randnrConv2d num_channelsr num_patches num_positionsr=rrJrKrLrMrPr.r/r<s"  "zGitVisionEmbeddings.__init__rYheightwidthrUc Cs|jdd}|jjd}|jdd}tjs(||kr(||kr(||jS|ddddf}|ddddf}|jd} ||j} ||j} t |d} | d| | | }| dddd}t j j|| | fdd d }| dddddd| }tj||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNr8g?rrbbicubicF)rVmode align_cornersr)rrr unsqueezer*jit is_tracingr6rorreshaperrr interpolaterr) rNrYrrrrrclass_pos_embedpatch_pos_embedr new_height new_widthsqrt_num_positionsr.r.r/interpolate_pos_encodings*      z,GitVisionEmbeddings.interpolate_pos_encodingF pixel_valuesc Cs|j\}}}}|s&||jks||jkr&td|d|d|jd|jd |jjj}||j|d}|ddd}|j |dd} t j | |gdd } |r[| | | ||} | S| ||j} | S) NzInput image size (*z) doesn't match model ().rrbrr8r)rrnrgrrrrflattenrrrLr*rr(rr6) rNr)r(rrrr target_dtype patch_embeds class_embedsrYr.r.r/rZ!s  zGitVisionEmbeddings.forwardF) r&r'r(rr<r*r]r\r(r+rZr^r.r.rPr/rs )rcr) GitVisionMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSr) r;r<rOrr activation_fnrrrr?rfc1fc2rMrPr.r/r<5s  zGitVisionMLP.__init__r$rUcCs"||}||}||}|Sr)r3r2r4rr.r.r/rZ<s   zGitVisionMLP.forwardrr.r.rPr/r14s r1rr rsrtrurscalingrHc Ks|t||dd|}|dur||}tjj|dtjd|j}tjj |||j d}t||} | dd } | |fS)Nr8r)rr)prrrb) r*rrrrrfloat32rrrHrr) r rsrtrurr5rHkwargs attn_weightsrr.r.r/eager_attention_forwardDs  r:csheZdZdZfddZ   d dejdeejdeejd eed e ejeejff d d Z Z S)GitVisionAttentionz=Multi-headed attention from 'Attention Is All You Need' papercst||_|j|_|j|_|j|j|_|j|j|jkr-td|jd|jd|jd|_ |j |_ d|_ t |j|j|_t |j|j|_t |j|j|_t |j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: r+gF)r;r<rOr?rre num_headshead_dimrgscaleattention_dropoutrH is_causalrrrk_projv_projq_projout_projrMrPr.r/r<^s$   zGitVisionAttention.__init__NFr$rcausal_attention_maskrrUc Cs0|j\}}}||}||} ||} ||||j|jdd}| |||j|jdd} | |||j|jdd} |jj dkrY|durR|durR||}n |durX|}n|du|_ t } |jj dkrlt |jj } | ||| | ||j |j |js{dn|jd\} } | |||} || } |sd} | | fS)z#Input shape: Batch x Time x Channelrrbflash_attention_2Nrr)r@r5rH)rrCrArBrr<r=rrOrr@r:rr>rrHr!rrD)rNr$rrErrrXrquerieskeysvaluesattention_interfacerr9r.r.r/rZrs@          zGitVisionAttention.forward)NNF) r&r'r(r)r<r*r]rrr-rZr^r.r.rPr/r;[s" r;c sTeZdZdeffdd Z d dejdejdejdeed e ej f d d Z Z S) GitVisionEncoderLayerrOcsRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSr) r;r<r?rr; self_attnrrDrE layer_norm1r1mlp layer_norm2rMrPr.r/r<s   zGitVisionEncoderLayer.__init__Fr$rrErrUcCsd|}||}|j||||d\}}||}|}||}||}||}|f}|r0||f7}|S)aI Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )r$rrEr)rMrLrOrN)rNr$rrErresidualr9outputsr.r.r/rZs"     zGitVisionEncoderLayer.forwardr0) r&r'r(rr<r*r]rrr-r+rZr^r.r.rPr/rKs rKcsxeZdZdZdeffdd Ze     ddeej deej dee d ee d ee d e e e ff d d ZZS)GitVisionEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`GitVisionEncoderLayer`]. Args: config: GitVisionConfig rOcr)Ncsg|]}tqSr.)rKrrrr.r/rsz-GitVisionEncoder.__init__..F) r;r<rOrrrrlayersrrMrPrr/r<rzGitVisionEncoder.__init__NrrErrrrUc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|r"dnd}|r(dnd}|} t|jD] \} } |r<|| f}| | |||d} | d} |rQ|| df}q1|rY|| f}t| ||dS)a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr.)rrrr#r$r%)rOrruse_return_dictrrTr) rNrSrrErrrencoder_statesall_attentionsr$idx encoder_layerrr.r.r/rZs2'    zGitVisionEncoder.forward)NNNNN)r&r'r(r)rr<rrr*r]rrr-rrZr^r.r.rPr/rRs, rRcsreZdZdeffdd Ze     ddeejdee dee d ee d ee d e e e ff d d Z ZS)GitVisionTransformerrOcsRt||_|j}t||_tj||jd|_ t ||_ tj||jd|_ dSr) r;r<rOr?rrYrrDrE pre_layrnormrRencoderpost_layernorm)rNrOrrPr.r/r<1s   zGitVisionTransformer.__init__NFr)rrr(rrUc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td|j||d}||}|j||||d}|d}||}|sO|f|ddSt ||j |j dS)Nz You have to specify pixel_valuesr()rSrrrrrrU) rOrrrVrgrYr\r]r^rr$r%) rNr)rrr(rr$encoder_outputsr#r.r.r/rZ;s.   zGitVisionTransformer.forwardNNNFN)r&r'r(rr<rrr*r+rrr-rrZr^r.r.rPr/r[/s*  r[zY The vision model from CLIP, used in GIT, without any head or projection on top. cseZdZUeed<dZdeffdd ZdejfddZ e    dde e j d e ed e ed ed e edeeeff ddZZS)GitVisionModelrOr)cs"t|t||_|dSr)r;r<r[ vision_model post_initrMrPr.r/r<os   zGitVisionModel.__init__rUcCs |jjjSr)rcrYrrNr.r.r/get_input_embeddingsu z#GitVisionModel.get_input_embeddingsNFrrr(rcCs(|dur|n|jj}|j|||||dS)a{ Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, GitVisionModel >>> processor = AutoProcessor.from_pretrained("microsoft/git-base") >>> model = GitVisionModel.from_pretrained("microsoft/git-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state ```N)r)rrr(r)rOrVrc)rNr)rrr(rr.r.r/rZxszGitVisionModel.forwardra)r&r'r(rr,main_input_namer<rModulerfrrr*r+rrr-rrZr^r.r.rPr/rbes0  rbcs8eZdZdeffdd ZdejdejfddZZS) GitProjectionrOcs@t||_tt|jj|jtj|j|jj d|_ dSr) r;r<rOr Sequentialrrrmr?rDrEvisual_projectionrMrPr.r/r<s  zGitProjection.__init__rYrUcCs ||Sr)rl)rNrYr.r.r/rZrgzGitProjection.forward) r&r'r(rr<r*r]rZr^r.r.rPr/rjsrjzy The bare GIT Model transformer consisting of a CLIP image encoder and text decoder outputting raw hidden-states cseZdZfddZddZddZddZd ed ej d ej d ej fd dZ d!ddZ e            d"deej deej deej deej deej deej deeeeejfdeedeedeededeed eeej effdd ZZS)#GitModelcsrt|_t|_tj|_t|_ t |_ j dur3t fddtj D|_|dS)Nc3s(|]}ttddjjVqdS)rN)rrr*zerosrmr?rSrr.r/rs  z$GitModel.__init__..)r;r<rOr0rYrbrm image_encoderrr]rjrlrqr ParameterListrimg_temporal_embeddingrdrMrPrr/r<s       zGitModel.__init__cCs|jjSrrYrArer.r.r/rfszGitModel.get_input_embeddingscCs ||j_dSrrr)rNrur.r.r/set_input_embeddingss zGitModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr]rrr)rNheads_to_prunerrr.r.r/ _prune_headsszGitModel._prune_headsrVrrrUcCs4tjtj||||ddd}||dktd}|S)Nrrr)diagonal-inf)r*triuones masked_fillfloat)rNrVrrmaskr.r.r/_generate_future_maskszGitModel._generate_future_maskNcCs|jd}|jd}|j}|j} tj||f|| d} tj|||ftd|j| d} tj||f| |jd} |dkrLtj|jd|jd|f| |jd}tj| | fdd} tj| || fdd}tj| |fdddddf}|durtj|jd|jdfd|d}|jtj krt d tj ||jd }td||<| |jd|||||f}| }|ddddd|f}|dddddf}|||ddddd|f<|dddddddf}|S) NrrwryrrrF) fill_valuerz1Memory key padding mask must be a boolean tensor.r)rrrr*rnfullr}rrrrg zeros_likerLclone)rNtgtmemorytgt_maskrTmemory_key_padding_masknum_tgt num_memoryrrtop_left top_right bottom_leftleftrightfull_attention_maskzero_negative_infinity origin_leftrr.r.r/create_attention_masksP      zGitModel.create_attention_maskFrRrr6r)rrSrzrrrr(rc  Cs| dur| n|jj} | dur| n|jj} |dur|n|jj}| dur$| n|jj} |dur4|dur4td|durC||||} n|durP|dd} ntd| d}d}|durkt|t sg| n| }| ||jj }d}|dur|j dkr|j|| dj}n=|j d krg}t|jdD]"}|j|dd|ddddf| dj}||j|7}||qtj|dd }ntd ||}|j||||d }|durtj|jdd|jd f|j|jd}||d|ddd}tj||fdd }|||j|j}|j||||d}|durWt||j| dd|j}|dkr=|dddd| dddf}n|dddd| d d| d df|7<|j |||||| | | |dud }|d}| sw|f|ddSt!||j"|j#|j$dS)a Examples: ```python >>> from transformers import AutoProcessor, AutoModel >>> import requests >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base") >>> model = AutoModel.from_pretrained("microsoft/git-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "this is an image of two cats" >>> inputs = processor(images=image, text=text, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer8z5You have to specify either input_ids or inputs_embedsrrr_rz#pixel_values must be of rank 4 or 5)rRr6rSrTrbr)rrrrT)tgt_len)rrrzrrrrrr)%rOrrrrVrg%warn_if_padding_and_no_attention_maskrVrr get_seq_length get_head_maskrndimror#rrrqappendr*rrlrYrnrrrepeatrrr rr]rrzr$r%)rNrRrr6r)rrSrzrrrr(rrWrXrTprojected_visual_featuresvisual_features frame_idxvisual_features_frameembedding_outputr$rcombined_attention_maskexpanded_attn_maskr`sequence_outputr.r.r/rZ s%           $4 zGitModel.forwardr) NNNNNNNNNNFN)r&r'r(r<rfrsrvr\r*rrr]rrrrrr listr+rr-rrZr^r.r.rPr/rms^   2     rmz` GIT Model with a `language modeling` head on top for autoregressive language modeling. c seZdZdgZfddZddZddZe            dd ee j d ee j d ee j d ee j dee j dee j dee j dee e e e j fdeedeedeededeede ee j effddZ dddZZS)GitForCausalLMz output.weightcs4t|t||_t|j|j|_| dSr) r;r<rmrrrrr?r>rrdrMrPr.r/r<s   zGitForCausalLM.__init__cCs|jSrrrer.r.r/get_output_embeddingssz$GitForCausalLM.get_output_embeddingscCs ||_dSrr)rNnew_embeddingsr.r.r/set_output_embeddingsrgz$GitForCausalLM.set_output_embeddingsNFrRrr6r)rrSlabelsrzrrrr(rrUcKs| dur| n|jj} |durd} |j|||||||| | | | | d }|d}||}d}|durl|jjjdjjj}|dd|dddf }|ddddf }|j | d|jj | dfd|jj i|}| s|f|dd}|dur|f|S|St |||j|j|jdS) a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` Examples: Image captioning example: ```python >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> import requests >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-coco") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-coco") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> pixel_values = processor(images=image, return_tensors="pt").pixel_values >>> generated_ids = model.generate(pixel_values=pixel_values, max_length=50) >>> generated_caption = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> print(generated_caption) two cats sleeping on a pink blanket next to remotes. ``` Visual question answering (VQA) example: ```python >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> from huggingface_hub import hf_hub_download >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-textvqa") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-textvqa") >>> file_path = hf_hub_download(repo_id="nielsr/textvqa-sample", filename="bus.png", repo_type="dataset") >>> image = Image.open(file_path).convert("RGB") >>> pixel_values = processor(images=image, return_tensors="pt").pixel_values >>> question = "what does the front of the bus say at the top?" >>> input_ids = processor(text=question, add_special_tokens=False).input_ids >>> input_ids = [processor.tokenizer.cls_token_id] + input_ids >>> input_ids = torch.tensor(input_ids).unsqueeze(0) >>> generated_ids = model.generate(pixel_values=pixel_values, input_ids=input_ids, max_length=50) >>> print(processor.batch_decode(generated_ids, skip_special_tokens=True)) ['what does the front of the bus say at the top? special'] ``` Video captioning example: ```python >>> import av >>> import numpy as np >>> from PIL import Image >>> from huggingface_hub import hf_hub_download >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-vatex") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-vatex") >>> # set seed for reproducibility >>> np.random.seed(45) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`list[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # load video >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample frames >>> num_frames = model.config.num_image_with_embedding >>> indices = sample_frame_indices( ... clip_len=num_frames, frame_sample_rate=4, seg_len=container.streams.video[0].frames ... ) >>> frames = read_video_pyav(container, indices) >>> pixel_values = processor(images=list(frames), return_tensors="pt").pixel_values >>> generated_ids = model.generate(pixel_values=pixel_values, max_length=50) >>> print("Generated caption:", processor.batch_decode(generated_ids, skip_special_tokens=True)) Generated caption: ['a woman is sitting at a table and she is talking about the food she is holding.'] ``` NF) rr6r)rrSrzrrrr(rrr8rr>)losslogitsrzr$r%)rOrVrrr]rrrNrpr loss_functionrr>rrzr$r%)rNrRrr6r)rrSrrzrrrr(rr8rQrrrnum_image_tokensshifted_logitsrr.r.r/rZsV zGitForCausalLM.forwardc Ks|dur#|}|jd|kr|}n|jdd}|dd|df}|j}|dur/||}|||d||d} |D] \} } | | vrJ| | | <q>| S)Nrr))rRrr)rzr)rrnew_onesgetrt) rNrRrzrrr8 past_lengthremove_prefix_lengthrW model_inputsrtrur.r.r/prepare_inputs_for_generations(  z,GitForCausalLM.prepare_inputs_for_generation) NNNNNNNNNNNFN)NNN)r&r'r(_tied_weights_keysr<rrrrr*r]rr rrr-rrZrr^r.r.rPr/rsf        Er)rrmrrb)r)Ir)r dataclassesrtypingrrrr*r activationsr cache_utilsr r generationr modeling_attn_mask_utilsr modeling_layersr modeling_outputsrrrrmodeling_utilsrr pytorch_utilsrrrutilsrrrrrutils.deprecationrconfiguration_gitrr get_loggerr&rir!rir0r_rrrrrrrrrr1r]r}r:r;rKrRr[rbrjrmr__all__r.r.r.r/s         0z3(HS J3W65  ~