o ei @sdZddlZddlmZddlmZddlZddlmZddlm Z ddl m Z dd l mZmZdd lmZdd lmZdd lmZdd lmZddlmZmZmZmZddlmZmZddl m!Z!ddl"m#Z#ddl$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*ddl+m,Z,ddl-m.Z.m/Z/e)0e1Z2ee'ddGddde%Z3dej4dBdej4dBdedBfddZ5e,dd d!d"   # dbd$ed!ej4d%ej4dBd&ej4d'edBd(ej4dBdej4dBd)ej6dBd*e7d+e7dBde8fd,d-Z9Gd.d/d/ej:Z;Gd0d1d1ej:ZGd5d6d6ej:Z?Gd7d8d8ej:Z@Gd9d:d:ej:ZAGd;d<dd>ej:ZCe'Gd?d@d@eZDGdAdBdBej:ZEGdCdDdDej:ZF EdcdFej:dGej4dHej4dIej4d%ej4dBdJeGdKeGfdLdMZHGdNdOdOej:ZIGdPdQdQeZJGdRdSdSej:ZKGdTdUdUej:ZLe'dVdGdWdXdXeDZMGdYdZdZej:ZNe'd[dGd\d]d]eDZOe'd^dGd_d`d`eDeZPgdaZQdS)dzPyTorch GIT model.N)Callable) dataclass)nn)initialization)ACT2FN)Cache DynamicCache)PreTrainedConfig)GenerationMixin)create_masks_for_generate)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPastBaseModelOutputWithPoolingCausalLMOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging torch_int)deprecate_kwarg) GitConfigGitVisionConfigz} Base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. ) custom_introc@sjeZdZUdZdZejdBed<dZejdBed<dZ e ejdfdBed<dZ e ejdfdBed<dS)GitVisionModelOutputz image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. N image_embedslast_hidden_state. hidden_states attentions) __name__ __module__ __qualname____doc__r"torch FloatTensor__annotations__r#r$tupler%r.r.b/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/git/modeling_git.pyr!7s r!token_type_idsimage_group_idsreturnc s4durdSdtdtdtdtdtf fdd }|S) z This function adds the correct offsets to the `q_idx` and `kv_idx` as the torch API can only accept lengths, not start and end indices. N batch_idxhead_idxq_idxkv_idxr2c st|jdk|d}t|jdk|d}||f}t|jdk|d}||f}t|jdk|d}||f}t|jdk|d}||f} t|jdk| d} |dk|dk@} || k} | | @S)Nrr)r*whereshape) r3r4r5r6 safe_q_idx safe_kv_idxtoken_type_ids_at_q_idxtoken_type_ids_at_kv_idximage_group_ids_at_q_idximage_group_ids_at_kv_idxis_image_blocksame_image_blockr1r0r.r/ inner_maskWs    z0token_type_ids_mask_function..inner_mask)intbool)r0r1rCr.rBr/token_type_ids_mask_functionKs $rF input_embedsz5.6.0 inputs_embeds)versionnew_nameFconfigattention_maskcache_positionpast_key_values position_ids pixel_values is_trainingis_first_iterationc Ks|r |dur td||||||d} | dur| n |dup&|j p&|du} |durh| rh|dk|j} tjj| dddddddf} | | @}tj | dd d}t | |d}t ||j|| d <t d i| S) a Overwrites the base `create_masks_for_generate` with `token_type_ids` masking to create the causal mask mapping for all kinds of forward passes. Gemma3 uses a bidirectional mask for images. Uses `pixel_values` as an optional input to disambiguate edge cases. Nz;`token_type_ids` is required as a model input when training)rKrHrLrMrNrOr)rrr)valuer7dimor_mask_functionr.) ValueErrorget_text_configis_initializedtodevicer functionalpadr*cumsumrDr8rFr )rKrHrLrMrNrOr0rPrQrRkwargs mask_kwargsis_imageis_previous_imagenew_image_startr1r.r.r/create_causal_mask_mappingss.   "  rdc s\eZdZdZfddZ    d dejdBdejdBdejdBd ed ej f d d Z Z S) GitEmbeddingsz;Construct the embeddings from word and position embeddings.csxttj|j|j|jd|_t|j|j|_ tj |j|j d|_ t |j |_|jdt|jddddS)N) padding_idxepsrOrr7F persistent)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_bufferr*arangeexpandselfrK __class__r.r/rms  zGitEmbeddings.__init__Nr input_idsrOrHpast_key_values_lengthr2c Cs|dur |}n|dd}|d}|dur&|jdd|||f}|dur0||}n|}||}||7}||}||}|S)Nr7r)sizerOrrrtrury) r~rrOrHr input_shape seq_length embeddingsrtr.r.r/forwards     zGitEmbeddings.forward)NNNr) r&r'r(r)rmr* LongTensorr+rDTensorr __classcell__r.r.rr/res$ rec s\eZdZd fdd Z   d dejdejdBdedBdejdBdeejf d d Z Z S) GitSelfAttentionNcs t|j|jdkrt|dstd|jd|jd||_|dur1td|j j d|j|_t |j|j|_ |j|j |_ t |jj|jjdd |_|jdura|j|j9_t|j|j |_t|j|j |_t|j|j |_t|j|_dS) Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()zInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.r)rlrmrpnum_attention_headshasattrrW layer_idxlogger warning_oncerr&rDattention_head_size all_head_size vision_config image_size patch_sizeimage_patch_tokensnum_image_with_embeddingrLinearquerykeyrSrwattention_probs_dropout_probryr~rKrrr.r/rms,   zGitSelfAttention.__init__r$rLrNrMr2c Cs,|jd}|||d|j|jdd}|||d|j|jdd}|||d|j|jdd}|durJ|j|||j d|id\}}t ||dd} | t |j} |durd| |} tjj| dd} || } t | |} | dddd } | dd|jf} | | } | | fS) Nrr7rrrM) cache_kwargsrTr)r9rviewrr transposerrSupdaterr*matmulmathsqrtrr\softmaxrypermute contiguousrr) r~r$rLrNrM batch_size query_layer key_layer value_layerattention_scoresattention_probs context_layernew_context_layer_shaper.r.r/rs8     zGitSelfAttention.forwardN)NNN) r&r'r(rmr*rr+rr-rrr.r.rr/rs rc8eZdZfddZdejdejdejfddZZS) GitSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNrg) rlrmrrrpdenserurvrwrxryr}rr.r/rm. zGitSelfOutput.__init__r$ input_tensorr2cC&||}||}|||}|Srrryrur~r$rr.r.r/r4  zGitSelfOutput.forwardr&r'r(rmr*rrrr.r.rr/r- $reagercsfeZdZd fdd Z    ddejdejdBdedBdejdBd edBd e ejf d d Z Z S) GitAttentionNcs,tt|j||d|_t||_dS)Nr)rlrmGIT_SELF_ATTENTION_CLASSES_attn_implementationr~routputrrr.r/rmAs zGitAttention.__init__Fr$rLrNrMoutput_attentionsr2c Cs*|j||||d\}}|||}||fS)N)rM)r~r) r~r$rLrNrMr attn_outputself_attn_weightsattention_outputr.r.r/rFs zGitAttention.forwardrNNNF) r&r'r(rmr*rr+rrEr-rrr.r.rr/r@s&rc2eZdZfddZdejdejfddZZS)GitIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) rlrmrrrpintermediate_sizer isinstance hidden_actstrrintermediate_act_fnr}rr.r/rmZs   zGitIntermediate.__init__r$r2cCs||}||}|Sr)rrr~r$r.r.r/rbs  zGitIntermediate.forwardrr.r.rr/rYs rcr) GitOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) rlrmrrrrprrurvrwrxryr}rr.r/rmjrzGitOutput.__init__r$rr2cCrrrrr.r.r/rprzGitOutput.forwardrr.r.rr/rirrcsneZdZdfdd Z    ddejdejdBdedBdejdBd edBd e ejf d d Z d dZ Z S)GitLayerNcs>t|j|_d|_t||d|_t||_t||_ dS)Nrr) rlrmchunk_size_feed_forward seq_len_dimr attentionr intermediaterrrrr.r/rmxs  zGitLayer.__init__Fr$rLrNrMrr2c Cs4|j|||||d\}}t|j|j|j|}||fS)N)rrNrM)rrfeed_forward_chunkrr) r~r$rLrNrMrrself_attention_weights layer_outputr.r.r/rs  zGitLayer.forwardcCs||}|||}|Sr)rr)r~rintermediate_outputrr.r.r/rs  zGitLayer.feed_forward_chunkrr) r&r'r(rmr*rr+rrEr-rrrr.r.rr/rws(  rcseZdZfddZ       ddejdejdBdedBd edBd edBd edBd edBd ejdBde eje BfddZ Z S) GitEncoderc:t|_tfddtjD|_d|_dS)Ncsg|]}t|qSr.)r).0irKr.r/ sz'GitEncoder.__init__..F) rlrmrKr ModuleListrangenum_hidden_layerslayergradient_checkpointingr}rrr/rm   zGitEncoder.__init__NFTr$rLrN use_cacheroutput_hidden_states return_dictrMr2c Cs|jr|jr|rtdd}|r|durt|jd}|rdnd} |r%dnd} t|jD] \} } |r7| |f} | |||||} | d}|rL| | df} q,|rT| |f} |sctdd||| | fDSt ||| | d S) NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Frr.rrcss|] }|dur|VqdSrr.)rvr.r.r/ sz%GitEncoder.forward..r#rNr$r%) rtrainingrrr rK enumeraterr-r)r~r$rLrNrrrrrMall_hidden_statesall_self_attentionsr layer_module layer_outputsr.r.r/rsP        zGitEncoder.forward)NNNFFTN) r&r'r(rmr*rr+rrEr-rrrr.r.rr/rs8    rc@s2eZdZUeed<dZdZdZe ddZ dS)GitPreTrainedModelrKgit)imagetextTcCsbt|tr7tj|jd|jjdtj|jj|jjdtj|j j|jjdt |j t |j jddt|tjrWtj|jd|jjd|jdurUt|jdSdSt|tjrtj|jd|jjd|jdurt|jddst|j|jdSdSdSt|tjrt|jt|jdSt|trt |j t |j jdddSdS) zInitialize the weights)meanstd)rr7riN_is_hf_initializedF)rGitVisionEmbeddingsinitnormal_class_embeddingrKinitializer_rangepatch_embeddingweightposition_embeddingcopy_rOr*r{r9r|rrbiaszeros_rnrfgetattrruones_re)r~moduler.r.r/ _init_weightss* "      &z GitPreTrainedModel._init_weightsN) r&r'r(rr,base_model_prefixinput_modalitiessupports_gradient_checkpointingr*no_gradr r.r.r.r/rs rcsXeZdZdeffdd Zdejdededejfdd Zdd ej dejfd d Z Z S)rrKcst||_|j|_|j|_|j|_tt |j|_ tj |j |j|j|jdd|_|j|jd|_|jd|_t|j|j|_|jdt |jddddS)NF) in_channels out_channels kernel_sizestriderrrrOrirj)rlrmrKrp embed_dimrrr Parameterr*randnrConv2d num_channelsr num_patches num_positionsrnrrzr{r|r}rr.r/rms"  "zGitVisionEmbeddings.__init__rheightwidthr2c Cs|jdd}|jjd}|jdd}tjs(||kr(||kr(||jS|ddddf}|ddddf}|jd} ||j} ||j} t |d} | d| | | }| dddd}t j j|| | fdd d }| dddddd| }tj||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNr7g?rrbicubicF)rmode align_cornersrT)r9rr unsqueezer*jit is_tracingrOrrreshaperrr\ interpolatercat) r~rrrrrrclass_pos_embedpatch_pos_embedrU new_height new_widthsqrt_num_positionsr.r.r/interpolate_pos_encodings*      z,GitVisionEmbeddings.interpolate_pos_encodingFrPc Cs|j\}}}}|s&||jks||jkr&td|d|d|jd|jd |jjj}||j|d}|ddd}|j |dd} t j | |gdd } |r[| | | ||} | S| ||j} | S) NzInput image size (*z) doesn't match model ().dtyperrr7rT)r9rrWrrr/rZflattenrrr|r*r%r+rrO) r~rPr+r_rr target_dtype patch_embeds class_embedsrr.r.r/r@s  zGitVisionEmbeddings.forwardF) r&r'r(rrmr*rrDr+r+rrr.r.rr/rs )rcr) GitVisionMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSr) rlrmrKrr activation_fnrrrprfc1fc2r}rr.r/rmTs  zGitVisionMLP.__init__r$r2cCs"||}||}||}|Sr)r8r7r9rr.r.r/r[s   zGitVisionMLP.forwardrr.r.rr/r6Ss r6rr rrrSscalingryc Ks|t||dd|}|dur||}tjj|dtjd|j}tjj |||j d}t||} | dd } | |fS)Nr7r)rUr/)prrr) r*rrrr\rfloat32rZr/ryrr) r rrrSrLr:ryr_ attn_weightsrr.r.r/eager_attention_forwardcs  r>c s\eZdZdZfddZ  d dejdejdBdedBd eejejdBffd d Z Z S) GitVisionAttentionz=Multi-headed attention from 'Attention Is All You Need' papercst||_|j|_|j|_|j|j|_|j|j|jkr-td|jd|jd|jd|_ |j |_ d|_ t |j|j|_t |j|j|_t |j|j|_t |j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: r-gF)rlrmrKrprr num_headshead_dimrWscaleattention_dropoutry is_causalrrk_projv_projq_projout_projr}rr.r/rm}s$   zGitVisionAttention.__init__NFr$rLrr2c Cs|j\}}}||}||}||} ||||j|jdd}||||j|jdd}| |||j|jdd} t |j j t } | |||| ||j |j|jsVdn|jd\} } | |||} || } |sod} | | fS)z#Input shape: Batch x Time x Channelrrr)rDr:ryN)r9rGrErFrr@rArr get_interfacerKrr>rDrBrryr#rrH) r~r$rLrrrrquerieskeysvaluesattention_interfacerr=r.r.r/rs2      zGitVisionAttention.forward)NF) r&r'r(r)rmr*rrEr-rrr.r.rr/r?zs r?c sVeZdZdeffdd Z d dejdejdedBd ee d e ej f d d Z Z S)GitVisionEncoderLayerrKcsRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSr) rlrmrprr? self_attnrrurv layer_norm1r6mlp layer_norm2r}rr.r/rms   zGitVisionEncoderLayer.__init__Fr$rLrNr_r2cKsj|}||}|jd|||d|\}}||}|}||}||}||}|f}|r3||f7}|S)aI Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )r$rLrNr.)rPrOrRrQ)r~r$rLrr_residualr=outputsr.r.r/rs$     zGitVisionEncoderLayer.forwardr5)r&r'r(rrmr*rrErrr-r+rrr.r.rr/rNs rNcspeZdZdZdeffdd Ze    ddejdBde dBde dBd e dBd e e d e e Bf d d ZZS)GitVisionEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`GitVisionEncoderLayer`]. Args: config: GitVisionConfig rKcr)Ncsg|]}tqSr.)rNrr1rr.r/rsz-GitVisionEncoder.__init__..F) rlrmrKrrrrlayersrr}rrr/rmrzGitVisionEncoder.__init__NrLrrrr_r2c Ks|dur|n|jj}|dur|n|jj}|dur|n|jj}|r"dnd}|r(dnd}|} t|jD]#\} } |r<|| f}| | |fd|i|} | d} |rT|| df}q1|r\|| f}t| ||dS)ad Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr.rrrr#r$r%)rKrruse_return_dictrrWr) r~rHrLrrrr_encoder_statesall_attentionsr$idx encoder_layerrr.r.r/rs6     zGitVisionEncoder.forward)NNNN)r&r'r(r)rrmrr*rrErrr-rrrr.r.rr/rUs*rUcsneZdZdeffdd Ze     ddejdBdedBdedBd edBd edBd e e Bf d d Z Z S)GitVisionTransformerrKcsRt||_|j}t||_tj||jd|_ t ||_ tj||jd|_ dSr) rlrmrKrprrrrurv pre_layrnormrUencoderpost_layernorm)r~rKrrr.r/rm?s   zGitVisionTransformer.__init__NFrPrrr+rr2c Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td|j||d}||}|j||||d}|d}||}|sO|f|ddSt ||j |j dS)Nz You have to specify pixel_valuesr+)rHrrrrrrX) rKrrrYrWrr_r`rarr$r%) r~rPrrr+rr$encoder_outputsr#r.r.r/rIs.   zGitVisionTransformer.forwardNNNFN) r&r'r(rrmrr*r+rEr-rrrr.r.rr/r^=s* r^zY The vision model from CLIP, used in GIT, without any head or projection on top. cseZdZUeed<dZdZdeffdd Zdej fddZ e dde j d Bd ed Bd ed Bd eded BdeeBf ddZZS)GitVisionModelrKrP)rcs"t|t||_|dSr)rlrmr^ vision_model post_initr}rr.r/rm~s   zGitVisionModel.__init__r2cCs |jjjSr)rfrrr~r.r.r/get_input_embeddings z#GitVisionModel.get_input_embeddingsNFrrr+rcKs(|dur|n|jj}|j|||||dS)a Examples: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, GitVisionModel >>> processor = AutoProcessor.from_pretrained("microsoft/git-base") >>> model = GitVisionModel.from_pretrained("microsoft/git-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state ```N)rPrrr+r)rKrYrf)r~rPrrr+rr_r.r.r/rszGitVisionModel.forwardrd)r&r'r(rr,main_input_namer rmrModulerirr*r+rEr-rrrr.r.rr/ress2 recs8eZdZdeffdd ZdejdejfddZZS) GitProjectionrKcs@t||_tt|jj|jtj|j|jj d|_ dSr) rlrmrKr Sequentialrrrprurvvisual_projectionr}rr.r/rms  zGitProjection.__init__rr2cCs ||Sr)ro)r~rr.r.r/rrjzGitProjection.forward) r&r'r(rrmr*rrrr.r.rr/rmsrmzy The bare GIT Model transformer consisting of a CLIP image encoder and text decoder outputting raw hidden-states cseZdZfddZddZddZe            dd ejdBd ejdBd ejdBd ejdBd ejdBde dBde dBde dBde dBde de dBdejdBde eje BfddZ ZS)GitModelcsrt|_t|_tj|_t|_ t |_ j dur3t fddtj D|_|dS)Nc3s(|]}ttddjjVqdS)rN)rrr*zerosrrprVrr.r/rs  z$GitModel.__init__..)rlrmrKrerrer image_encoderrr`rmrorr ParameterListrimg_temporal_embeddingrgr}rrr/rms       zGitModel.__init__cCs|jjSrrrrrhr.r.r/riszGitModel.get_input_embeddingscCs ||j_dSrru)r~rSr.r.r/set_input_embeddingss zGitModel.set_input_embeddingsNFrrLrOrPrHrNrrrr+rrMr2c  Ks|dur|n|jj}| dur| n|jj} |dur|n|jj}| dur$| n|jj} |dur4|dur4tdd}|durGt|tsC|n|}|durZ|durZ|j ddkrZ||}|j ||||d}| durut j |||j d|j d} t j|t jdd}|dur|jd kr|j|| d j}n=|jd krg}t|j dD]"}|j|dd|ddddf| d j}||j|7}||qt j|dd }ntd ||}||d|ddd}t j||fdd }t j|t jdd}t j||gdd }t j |j d|j t jd} |durt jt ||gdd }n7|durU|j ddkrUt j|g| j| j d} t j|j d||j ddf|j|j d}t j||gdd }t|j||| |d||}|}|j|||||| | | d}|d}| s|f|ddSt ||j!|j"|j#dS)a Examples: ```python >>> from transformers import AutoProcessor, AutoModel >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base") >>> model = AutoModel.from_pretrained("microsoft/git-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> text = "this is an image of two cats" >>> inputs = processor(images=image, text=text, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timerr)rrOrHr)r[r.).rrbrTz#pixel_values must be of rank 4 or 5r7)r[r/)r/r[)rLrNrrrrrMr)$rKrrrrYrWrrget_seq_lengthr9rr*r{r[ zeros_likerDndimrrr#rrtappendr%rorepeatr ones_liketensorr/onesrdr`rrNr$r%)r~rrLrOrPrHrNrrrr+rrMr_rembedding_outputr0visual_features frame_idxvisual_features_frameprojected_visual_featuresimage_token_type_idsextended_attention_mask causal_maskr$rcsequence_outputr.r.r/rs(           zGitModel.forward) NNNNNNNNNFNN)r&r'r(rmrirvrr*rrrEr-rrrr.r.rr/rpsX       rpz` GIT Model with a `language modeling` head on top for autoregressive language modeling. c!seZdZddiZfddZddZddZe dd ej d Bd ej d Bdej d Bdej d Bdej d Bdej d Bde d Bde d Bde d Bde d Bde de d Bde ej Bdej d Bde ej eBfddZ d fdd ZZS)!GitForCausalLMz output.weightz%git.embeddings.word_embeddings.weightcs4t|t||_t|j|j|_| dSr) rlrmrprrrrprorrgr}rr.r/rms   zGitForCausalLM.__init__cCs|jSrrrhr.r.r/get_output_embeddingssz$GitForCausalLM.get_output_embeddingscCs ||_dSrr)r~new_embeddingsr.r.r/set_output_embeddingsrjz$GitForCausalLM.set_output_embeddingsNFrrrLrOrPrHlabelsrNrrrr+rlogits_to_keeprMr2cKsH| dur| n|jj} |durd}|j|||||||| | | | |d }|d}t| tr0t| dn| }||dd|ddf}d}|dur|jjjdj j j }|dd|dddf }|ddddf }|j |d|jj|dfd|jji|}| s|f|dd}|dur|f|S|St|||j|j|jdS) a0 labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` Examples: Image captioning example: ```python >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-coco") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-coco") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> pixel_values = processor(images=image, return_tensors="pt").pixel_values >>> generated_ids = model.generate(pixel_values=pixel_values, max_length=50) >>> generated_caption = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> print(generated_caption) two cats sleeping on a pink blanket next to remotes. ``` Visual question answering (VQA) example: ```python >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> from huggingface_hub import hf_hub_download >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-textvqa") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-textvqa") >>> file_path = hf_hub_download(repo_id="nielsr/textvqa-sample", filename="bus.png", repo_type="dataset") >>> image = Image.open(file_path).convert("RGB") >>> pixel_values = processor(images=image, return_tensors="pt").pixel_values >>> question = "what does the front of the bus say at the top?" >>> input_ids = processor(text=question, add_special_tokens=False).input_ids >>> input_ids = [processor.tokenizer.cls_token_id] + input_ids >>> input_ids = torch.tensor(input_ids).unsqueeze(0) >>> generated_ids = model.generate(pixel_values=pixel_values, input_ids=input_ids, max_length=50) >>> print(processor.batch_decode(generated_ids, skip_special_tokens=True)) ['what does the front of the bus say at the top? special'] ``` Video captioning example: ```python >>> import av >>> import numpy as np >>> from PIL import Image >>> from huggingface_hub import hf_hub_download >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-vatex") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-vatex") >>> # set seed for reproducibility >>> np.random.seed(45) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`list[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # load video >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample frames >>> num_frames = model.config.num_image_with_embedding >>> indices = sample_frame_indices( ... clip_len=num_frames, frame_sample_rate=4, seg_len=container.streams.video[0].frames ... ) >>> frames = read_video_pyav(container, indices) >>> pixel_values = processor(images=list(frames), return_tensors="pt").pixel_values >>> generated_ids = model.generate(pixel_values=pixel_values, max_length=50) >>> print("Generated caption:", processor.batch_decode(generated_ids, skip_special_tokens=True)) Generated caption: ['a woman is sitting at a table and she is talking about the food she is holding.'] ``` NF) rLrOrPrHrNrrrr+rrMrr7rro)losslogitsrNr$r%)rKrYrrrDslicerr`rrr~rr loss_functionrrorrNr$r%)r~rrLrOrPrHrrNrrrr+rrrMr_rTr$ slice_indicesrrnum_image_tokensshifted_logitsrr.r.r/rsXzGitForCausalLM.forwardc  s4tj|f|||||d|} |s|s|| d<| S)N)rNrLrrMrRrP)rlprepare_inputs_for_generation) r~rrNrPrLrrMrRr_ model_inputsrr.r/rds  z,GitForCausalLM.prepare_inputs_for_generation)NNNNNNNNNNFNrN)NNNNNF)r&r'r(_tied_weights_keysrmrrrr*rrrErDr-rrrrr.r.rr/rsv         Lr)rrprre)NNFN)r)Rr)rcollections.abcr dataclassesrr*rrr activationsr cache_utilsrr configuration_utilsr generationr masking_utilsr modeling_layersr modeling_outputsrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsrrrrrrutils.deprecationrconfiguration_gitrr get_loggerr&rr!rrFr+rEdictrdrlrerrrrrrrrrrr6floatr>r?rNrUr^rermrpr__all__r.r.r.r/s                (    8-S%DS @3P69 B y