o ۷i$@sdZddlZddlZddlmZmZmZddlZddlmZddl m Z ddl m Z ddl mZmZmZdd lmZdd lmZdd lmZmZmZdd lmZdd lmZmZmZddlm Z m!Z!m"Z"ddl#m$Z$e!%e&Z'ddZ(Gdddej)Z*Gdddej)Z+Gdddej)Z,GdddeZ-e GdddeZ.e Gddde.Z/e dd Gd!d"d"e.eZ0e d#d Gd$d%d%e.Z1gd&Z2dS)'zPyTorch OpenAI ImageGPT model.N)AnyOptionalUnion)nn)CrossEntropyLoss)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentions SequenceClassifierOutputWithPast)PreTrainedModel)Conv1D find_pruneable_heads_and_indicesprune_conv1d_layer)auto_docstringlogging torch_float)ImageGPTConfigc Csz ddl}ddl}Wn tytdwtj|}td||j |}g}g}|D]"\} } td| d| |j || } | | | | q2t||D]\} } | dd} | d} td d | Dsy| d d vrtd d| qZ|} | d dvrt| d} | D]} |d| r|d| }n| g}|ddks|ddkrt| d} n|ddkrt| d} nw|ddks|ddkrt| |d} t| d} n^|ddvrt| d} t| d} nMt| dkr| ddkr|ddkrt| |d} t| d} n+|ddkr"t| d } t| d} n|dd!kr4t| d} t| d} nt| |d} t|d"krLt|d}| |} qt| dkr[| ddksp| d dksp| d d!ksp| d dkrqn%z | j| jks{JWnty}z |j| j| jf7_d}~wwtd#| | d d$krt| |j|jj| jddd|jf<qZ| d d%krt| |j|jj| jdd|jd"|jf<qZ| d d&krt| |j|jj| jddd"|jdf<qZt| dkr| ddkr| d"dkrt| |j|j| _qZ| d dkr-t| | _qZ| d dkrFt| | jd|j dddf<qZ| d d!krVt| | jd <qZt| | _qZ|S)'z0 Load tf checkpoints in a pytorch model rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from zLoading TF weight z with shape /css|]}|dvVqdS))adam_vadam_mAdamWeightDecayOptimizerAdamWeightDecayOptimizer_1 global_stepN).0nr!r!d/home/ubuntu/vllm_env/lib/python3.10/site-packages/transformers/models/imagegpt/modeling_imagegpt.py Ps  z.load_tf_weights_in_imagegpt..)_stepz Skipping {})wtet transformerz [A-Za-z]+\d+z(\d+)wgweightbbiaswpewte)q_projk_projv_projc_attnrrattnc_projr(lm_headsoszInitialize PyTorch weight r1r2r3)!re tensorflow ImportErrorloggererrorospathabspathinfotrainlist_variables load_variableappendsqueezezipsplitanyformatjoingetattr fullmatchlenintshapeAssertionErrorargstorch from_numpyreshapen_embdTdata vocab_size)modelconfigimagegpt_checkpoint_pathr:tftf_path init_varsnamesarraysnamerQarraypointerm_name scope_namesnumer!r!r$load_tf_weights_in_imagegpt/s                   *      F.2*$rjcsBeZdZd deedeffdd Zdejdejfdd Z Z S) ImageGPTLayerNormh㈵> hidden_sizeepscs&t||_tt||_dSN)super__init__rnr ParameterrTTensorr,)selfrmrn __class__r!r$rqs zImageGPTLayerNorm.__init__tensorreturncCs4|ttjt|ddd|j}||j}|S)Nr&T)axiskeepdim)rTsqrtmeansquarernr,)rtrwr!r!r$forwards& zImageGPTLayerNorm.forward)rl) __name__ __module__ __qualname__tuplerPfloatrqrTrsr~ __classcell__r!r!rur$rksrkcseZdZddeedeeffdd ZddZdd d Zdd d Z d dZ ddZ        dde j deedee j dee j dee j dee j deedeedee j defddZZS) ImageGPTAttentionFNis_cross_attention layer_idxcsFt|j}|jdttj||ftjddd||dd|jdt ddd|j |_ |j |_ |j |j |_|j |_|j|j |j krUtd|j d |j d |j|_||_|j|_||_|j|_|jr}td |j |j |_t|j |j |_n td |j |j |_t|j |j |_t|j|_t|j|_t |_!dS) Nr.dtyperF) persistent masked_biasgz=`embed_dim` must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r9r)"rprqmax_position_embeddingsregister_bufferrTtrilonesboolviewrwrm embed_dimnum_attention_heads num_headshead_dim split_size ValueErrorscale_attn_weightsrscale_attn_by_inverse_layer_idxrreorder_and_upcast_attnrr4q_attnr6rDropout attn_pdrop attn_dropout resid_pdrop resid_dropoutset pruned_heads)rtr\rr max_positionsrur!r$rqsB   zImageGPTAttention.__init__cCst|dkrdSt||j|j|j\}}t|||j|d|jg}t|j |dd|_ t|j |dd|_ |j|j|jt||_|jt||_|j ||_dS)Nrr9rdim) rOrrrrrTcatrrr4r6union)rtheadsindex index_attnr!r!r$ prune_headss  zImageGPTAttention.prune_headsc Cst||dd}|jr|t|dd}|jr$|t|jd}|j s]|d|d}}|j dddd|||d|f} t |j j } tj| |j |jd} t| || }|dure||}tjdd|}||j }||}|dur||}t||} | |fS)Nr&?rrdevicer)rTmatmul transposerrsizerrrrr.finforminrwrwhererSoftmaxtyper) rtquerykeyvalueattention_mask head_mask attn_weights query_length key_length causal_mask mask_value attn_outputr!r!r$_attns(&   zImageGPTAttention._attncCs|\}}}} |\} } } } tj|||| tj|jd} d} |jr.| t|dd} |jr:| t|jd} tj |jj dd1| d|| | dd d| | }}tj | ||d | d } | |||| } Wdn1swwY|js|d|d}}|jdddd|||d|f}t| jj}tj|| j| jd}t|| |} |dur| |} tjdd | } | jtjkrtd | |j} || } |dur| |} t| |}|| fS) Nr?r&rrF)enabledrr)betaalpharzDError with upcasting, attn_weights does not have dtype torch.float32)rrTemptyfloat32rrrrrautocastrrVrbaddbmmrr.rrrrwrrr RuntimeErrorrr)rtrrrrrbszr q_seq_lendk_ k_seq_lenr scale_factorqkrrrrrr!r!r$_upcast_and_reordered_attn s<&&    z,ImageGPTAttention._upcast_and_reordered_attncCs2|dd||f}|j|}|ddddS)zJ Splits hidden_size dim into attn_head_size and num_heads Nr&rr9rr)rrpermutertrwrattn_head_size new_shaper!r!r$ _split_heads=s zImageGPTAttention._split_headscCs8|dddd}|dd||f}||S)zS Merges attn_head_size dim and num_attn_heads dim into hidden_size rr9rrNr)r contiguousrrrr!r!r$ _merge_headsEs zImageGPTAttention._merge_heads hidden_states layer_pastrrencoder_hidden_statesencoder_attention_mask use_cacheoutput_attentionscache_positionrxc Cs|du} |j\} } } |dur&t|tr$|j|j}| r |j}n|j}n|}| r*|n|}| rt|ds7t d|durQ|rQ| |}|j |jj }|j |jj }nY| |}||j|jdd\}}|| d|j|jdd}|| d|j|jdd}n*||j|jdd\}}}|| d|j|jdd}|| d|j|jdd}|dur| s| nd} ||||jd| i\}}| rd|j|j<|| | |j|jdd}|jr||||||\}}n ||||||\}}|||j|j}||}||}||fS) NrzIf class is used as cross attention, the weights `q_attn` have to be defined. Please make sure to instantiate class with `ImageGPTAttention(..., is_cross_attention=True)`.r9rr&rrT)rQ isinstancer is_updatedgetrcross_attention_cacheself_attention_cachehasattrrrlayerskeysvaluesr4rIrrrrrupdaterrrrr6r)rtrrrrrrrrrrrseq_lenrrcurr_past_key_valuecurrent_statesrrrrrr!r!r$r~MsN           zImageGPTAttention.forward)FN)NNNNNNNFFN)rrrrrrPrqrrrrrrTrsr rr~rr!r!rur$rsH +  &4    rcs2eZdZfddZdejdejfddZZS) ImageGPTMLPcsFt|j}t|||_t|||_t|j|_t |j |_ dSro) rprqrmrc_fcr6ractivation_functionactrrrdropout)rtintermediate_sizer\rrur!r$rqs    zImageGPTMLP.__init__rrxcCs,||}||}||}||}|Sro)rrr6r)rtrr!r!r$r~s    zImageGPTMLP.forward)rrrrqrTrsr~rr!r!rur$rs rcseZdZdfdd Z        ddejdeedeejdeejd eejd eejd eed eed eejde fddZ Z S) ImageGPTBlockNcst|j}|jdur|jnd|}t||jd|_t||d|_t||jd|_ |j r>t|d|d|_ t||jd|_ t |||_dS)NrnrT)rr)rprqrmn_innerrklayer_norm_epsilonln_1rr5ln_2add_cross_attentioncrossattention ln_cross_attnrmlp)rtr\rrm inner_dimrur!r$rqs zImageGPTBlock.__init__Frrrrrrrrrrxc  Cs|} ||}|j||||||| d} | d} | dd} | | }|durVt|ds2td|d|} ||}|j|||||||| d}|d} | | }| |dd} |} ||}||}| |}|f| S)N)rrrrrrrrrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`)rrrrrrr)rr5rrrrrr)rtrrrrrrrrrresidual attn_outputsroutputscross_attn_outputsfeed_forward_hidden_statesr!r!r$r~sN         zImageGPTBlock.forwardror) rrrrqrTrsrr rrr~rr!r!rur$rs>   rcsDeZdZUeed<eZdZdZdZ dgZ fddZ dd Z Z S) ImageGPTPreTrainedModelr\r) input_idsTrcstj|i|dSro)rprq)rtinputskwargsrur!r$rqsz ImageGPTPreTrainedModel.__init__cCst|tjtfr|jjjd|jjd|j dur|j j n,t|tj r?|jjjd|jjd|j dur>|jj|j  n t|t rK|jjd|D]\}}d|vrnd|vrn|jjd|jjtd|jjdqOdS)zInitialize the weights.g)r|stdNrr6r,r9)rrLinearrr,rYnormal_r\initializer_ranger.zero_ Embedding padding_idxrkfill_named_parametersmathr{n_layer)rtmodulercpr!r!r$ _init_weightss"     &z%ImageGPTPreTrainedModel._init_weights)rrrr__annotations__rjload_tf_weightsbase_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modulesrqrrr!r!rur$r s  r c$seZdZdeffdd ZddZddZdd Ze dd e e j d e e d e e j de e j de e j de e j de e j de e j de e j de e de e de e de e de e j dedeeeff ddZZS) ImageGPTModelr\cstj|_tj|j|_tj|j|_ t j |_ t fddtjD|_t|jjd|_d|_d|_d|_|dS)Ncsg|]}t|dqS)r)r)r"ir\r!r$ $sz*ImageGPTModel.__init__..rF)rprqrmrrrrZr0rr/r embd_pdropdrop ModuleListrangenum_hidden_layershrkrln_fmodel_parallel device_mapgradient_checkpointing post_initrtr\rur$r$rqs   zImageGPTModel.__init__cCs|jSror0)rtr!r!r$get_input_embeddings.sz"ImageGPTModel.get_input_embeddingscCs ||_dSror2)rtnew_embeddingsr!r!r$set_input_embeddings1s z"ImageGPTModel.set_input_embeddingscCs(|D] \}}|j|j|qdS)zv Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} N)itemsr+r5r)rtheads_to_prunelayerrr!r!r$ _prune_heads4szImageGPTModel._prune_headsNr past_key_valuesrtoken_type_ids position_idsr inputs_embedsrrrroutput_hidden_states return_dictrr rxc$ Ksh| dur| n|jj} | dur| n|jj} | dur| n|jj} | dur$| n|jj} |dur4|dur4td|durP||||}|d|d}|j d}n|durb|dd}|j d}ntd|durm|j n|j }|j r|j r| rt dd} | r|durtt|jdt|jd}| rt|trt d t|}|dur|n|}|dur|d|d}|durtj||d|tj|d }|d}|dur|dkrtd ||d}|ddddddf}|j|jd }d |t|jj}|jjr(|dur(|\}}}||f}| dur"tj||d} || } nd} | ||jj!}|dur<|"|}|#|}|||j }|durW|"|}||}|$|}||df}| ridnd}| ru|jjrudnd}| r|dnd}t%|j&D]\}} |j'rtj()|j |dur||j }t|tj*r||j }| r||f}| ||||||| | | |d }!|!d}| r||!df}|jjr||!df}|j'r|j+,D]!\}"}#||#dkrdt-|"|j.kr|dt-|"d}qq|/|}|j|}| r||f}| s+tdd|||||fDSt0|||||dS)a input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoImageProcessor`]. See [`ImageGPTImageProcessor.__call__`] for details. Examples: ```python >>> from transformers import AutoImageProcessor, ImageGPTModel >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("openai/imagegpt-small") >>> model = ImageGPTModel.from_pretrained("openai/imagegpt-small") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer&rz5You have to specify either input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fr$zPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.rz$batch_size has to be defined and > 0rr)rr!)rrrrrr9zcuda:css|] }|dur|VqdSror!)r"vr!r!r$r%sz(ImageGPTModel.forward..)last_hidden_stater:r attentionscross_attentions)1r\rr>ruse_return_dictr%warn_if_padding_and_no_attention_maskrrrQrr/trainingr= warning_oncer r rrfrom_legacy_cacheget_seq_lengthrTarangelong unsqueezetorrrrrinvert_attention_mask get_head_maskrr0r/r' enumerater+r-cuda set_devicersr.r6str last_devicer,r)$rtr r:rr;r<rr=rrrrr>r?rr  input_shape batch_sizer past_lengthencoder_batch_sizeencoder_sequence_lengthrencoder_hidden_shapeposition_embedsrtoken_type_embeds output_shapeall_self_attentionsall_cross_attentionsall_hidden_statesr#blockrrr@r!r!r$r~;s0                       "    zImageGPTModel.forward)NNNNNNNNNNNNNN)rrrrrqr3r5r9rrrTrsr rrrrrr~rr!r!rur$r"sj      r"z The ImageGPT Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introc&seZdZdgZdeffdd Ze               ddeej dee deej d eej d eej d eej d eej d eej deej deej dee dee dee dee deej de de eeff"ddZZS)ImageGPTForCausalImageModelingzlm_head.weightr\csHt|t||_tj|j|jddd|_d|_ d|_ | dS)NrFr.) rprqr"r)rrrWrZr7r-r.r0r1rur!r$rqs   z'ImageGPTForCausalImageModeling.__init__Nr r:rr;r<rr=rrlabelsrrr>r?rr rxcKs|dur|n|jj}|j||||||||| | | | ||d}|d}||}d}| durU|dddddf}| dddf}t}||d|d|d}|sk|f|dd}|duri|f|S|St|||j |j |j |j dS)a& input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoImageProcessor`]. See [`ImageGPTImageProcessor.__call__`] for details. labels (`torch.LongTensor` of shape `(batch_size, input_ids_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import AutoImageProcessor, ImageGPTForCausalImageModeling >>> import torch >>> import matplotlib.pyplot as plt >>> import numpy as np >>> image_processor = AutoImageProcessor.from_pretrained("openai/imagegpt-small") >>> model = ImageGPTForCausalImageModeling.from_pretrained("openai/imagegpt-small") >>> device = torch.device("cuda" if torch.cuda.is_available() else "cpu") >>> model.to(device) # doctest: +IGNORE_RESULT >>> # unconditional generation of 8 images >>> batch_size = 4 >>> context = torch.full((batch_size, 1), model.config.vocab_size - 1) # initialize with SOS token >>> context = context.to(device) >>> output = model.generate( ... input_ids=context, max_length=model.config.n_positions + 1, temperature=1.0, do_sample=True, top_k=40 ... ) >>> clusters = image_processor.clusters >>> height = image_processor.size["height"] >>> width = image_processor.size["width"] >>> samples = output[:, 1:].detach().cpu().numpy() >>> samples_img = [ ... np.reshape(np.rint(127.5 * (clusters[s] + 1.0)), [height, width, 3]).astype(np.uint8) for s in samples ... ] # convert color cluster tokens back to pixels >>> f, axes = plt.subplots(1, batch_size, dpi=300) >>> for img, ax in zip(samples_img, axes): # doctest: +IGNORE_RESULT ... ax.axis("off") ... ax.imshow(img) ```N) r:rr;r<rr=rrrrr>r?rr.r&r)losslogitsr:rrBrC) r\rDr)r7rrrrrr:rrBrC)rtr r:rr;r<rr=rrrerrr>r?rr transformer_outputsr lm_logitsrf shift_logits shift_labelsloss_fctoutputr!r!r$r~sHG z&ImageGPTForCausalImageModeling.forward)NNNNNNNNNNNNNNN)rrr_tied_weights_keysrrqrrrTrsr rrrrrr~rr!r!rur$rc sl       rcz The ImageGPT Model transformer with an image classification head on top (linear layer). [`ImageGPTForImageClassification`] average-pools the hidden states in order to do the classification. c seZdZdeffdd Ze            ddeejdee deejdeejd eejd eejd eejd eejd ee dee dee dee de de e effddZZS)ImageGPTForImageClassificationr\cs@t||j|_t||_tj|j|jdd|_| dS)NFrd) rprq num_labelsr"r)rrrWscorer0r1rur!r$rqs   z'ImageGPTForImageClassification.__init__Nr r:rr;r<rr=rerrr>r?r rxc  Ks| dur| n|jj} |j|||||||| | | | d }|d}|jdd}||}d}|dur7||||j}| sM|f|dd}|durK|f|S|St|||j|j|j dS)ay input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoImageProcessor`]. See [`ImageGPTImageProcessor.__call__`] for details. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, ImageGPTForImageClassification >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("openai/imagegpt-small") >>> model = ImageGPTForImageClassification.from_pretrained("openai/imagegpt-small") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits ```N) r:rr;r<rr=rrr>r?rrr)rfrgr:rrB) r\rDr)r|rq loss_functionrr:rrB)rtr r:rr;r<rr=rerrr>r?r rhrpooled_hidden_statesrgrfrmr!r!r$r~s<2  z&ImageGPTForImageClassification.forward) NNNNNNNNNNNN)rrrrrqrrrTrsr rrrrrr~rr!r!rur$rosX       ro)rcror"r rj)3__doc__rr?typingrrrrTrtorch.nnr activationsr cache_utilsr r r generationr modeling_layersr modeling_outputsrrrmodeling_utilsr pytorch_utilsrrrutilsrrrconfiguration_imagegptr get_loggerrr=rjModulerkrrrr r"rcro__all__r!r!r!r$sL        l mM&q b