o ei@sdZddlZddlmZddlZddlmZddlmZddlm Z ddl m Z dd l mZmZmZdd lmZdd lmZdd lmZmZmZdd lmZddlmZddlmZmZm Z ddl!m"Z"ddl#m$Z$e%e&Z'Gdddej(Z)Gdddej(Z*Gdddej(Z+GdddeZ,eGdddeZ-eGddde-Z.edd Gd!d"d"e-eZ/ed#d Gd$d%d%e-Z0gd&Z1dS)'zPyTorch OpenAI ImageGPT model.N)Any)nn)CrossEntropyLoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentions SequenceClassifierOutputWithPast)PreTrainedModel)Conv1D)auto_docstringlogging torch_float)maybe_autocast)ImageGPTConfigcsBeZdZd deedeffdd Zdejdejfdd Z Z S) ImageGPTLayerNormh㈵> hidden_sizeepscs&t||_tt||_dSN)super__init__rr ParametertorchTensorweight)selfrr __class__l/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/imagegpt/modeling_imagegpt.pyr0s zImageGPTLayerNorm.__init__tensorreturncCs4|ttjt|ddd|j}||j}|S)NT)axiskeepdim)r sqrtmeansquarerr")r#r(r&r&r'forward5s& zImageGPTLayerNorm.forward)r) __name__ __module__ __qualname__tupleintfloatrr r!r0 __classcell__r&r&r$r'r/srcseZdZddedBdedBffdd ZdddZdd d Zd d Zd dZ       dde j de dBde j dBde j dBde j dBdedBdedBde j dBde fddZZS)ImageGPTAttentionFNis_cross_attention layer_idxcs.t||_|j}|jdttj||ftjd dd||dd|j |_ |j |_ |j |j |_|j |_|j|j |j krMtd|j d|j d|j|_||_|j|_||_|j|_|jrutd |j |j |_t|j |j |_n td |j |j |_t|j |j |_t|j|_t|j|_dS) NbiasdtyperF) persistentz=`embed_dim` must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r) rrconfigmax_position_embeddingsregister_bufferr trilonesboolviewr embed_dimnum_attention_heads num_headshead_dim split_size ValueErrorscale_attn_weightsr9scale_attn_by_inverse_layer_idxr:reorder_and_upcast_attnrc_attnq_attnc_projrDropout attn_pdrop attn_dropout resid_pdrop resid_dropout)r#r@r9r: max_positionsr$r&r'r=s@  zImageGPTAttention.__init__c Cst||dd}|jr|t|dd}|jr$|t|jd}|j s]|d|d}}|j dddd|||d|f}t |j j } tj| |j |jd} t||| }|dure||}tjdd|}||j }||}t||} | |fS)Nr*?rr=devicedim)r matmul transposerMrsizerNr6r:r9r;finfor=minr(r\whererSoftmaxtyperU) r#querykeyvalueattention_mask attn_weights query_length key_length causal_mask mask_value attn_outputr&r&r'_attnes$&   zImageGPTAttention._attncCs|\}}}}|\} } } } tj|||| tj|jd} d} |jr.| t|dd} |jr:| t|jd} t |jj dd1| d||| dd d|| } }tj | | |d | d } | |||| } Wdn1svwY|js|d|d}}|jdddd|||d|f}t| jj}tj|| j| jd}t|| |} |dur| |} tjdd | } | jtjkrtd | |j} || } t| |}|| fS) Nr[?r*rZrF)enabledrYr)betaalphar]zDError with upcasting, attn_weights does not have dtype torch.float32)rar emptyfloat32r\rMr6rNr:rrfreshaper`baddbmmr9r;rbr=rcr(rdrre RuntimeErrorrUr_)r#rgrhrirjbszrI q_seq_lendk_ k_seq_lenrk scale_factorqkrlrmrnrorpr&r&r'_upcast_and_reordered_attns8&&    z,ImageGPTAttention._upcast_and_reordered_attncCs2|dd||f}|j|}|ddddS)zJ Splits hidden_size dim into attn_head_size and num_heads Nr*rr?rr)rarFpermuter#r(rIattn_head_size new_shaper&r&r' _split_headss zImageGPTAttention._split_headscCs8|dddd}|dd||f}||S)zS Merges attn_head_size dim and num_attn_heads dim into hidden_size rr?rrNrY)r contiguousrarFrr&r&r' _merge_headss zImageGPTAttention._merge_heads hidden_states layer_pastrjencoder_hidden_statesencoder_attention_mask use_cacheoutput_attentionscache_positionr)c Cs |du} |j\} } } |dur&t|tr$|j|j} | r |j}n|j}n|}| r*|n|}| rt|ds7t d|durQ| rQ| |}|j |jj }|j |jj }nY| |}||j|jdd\}}|| d|j|jdd}|| d|j|jdd}n*||j|jdd\}}}|| d|j|jdd}|| d|j|jdd}|dur| s|nd}||||jd|i\}}| rd|j|j<|| | |j|jdd}|jr|||||\}}n |||||\}}|||j|j}||}||}||fS) NrQzIf class is used as cross attention, the weights `q_attn` have to be defined. Please make sure to instantiate class with `ImageGPTAttention(..., is_cross_attention=True)`.r?r]r*rrT)shape isinstancer is_updatedgetr:cross_attention_cacheself_attention_cachehasattrrLrQlayerskeysvaluesrPsplitrKrFrIrJr`updaterOrrqrrRrW)r#rrrjrrrrrr9r{seq_lenr~rcurr_past_key_valuescurrent_statesrgrhrirprkr&r&r'r0sN           zImageGPTAttention.forward)FNrNNNNFFN)r1r2r3rEr5rrqrrrr r!rr4r0r7r&r&r$r'r8<s@  ( "0   r8cs2eZdZfddZdejdejfddZZS) ImageGPTMLPcsFt|j}t|||_t|||_t|j|_t |j |_ dSr) rrrrc_fcrRractivation_functionactrrSrVdropout)r#intermediate_sizer@rGr$r&r'rs    zImageGPTMLP.__init__rr)cCs,||}||}||}||}|Sr)rrrRr)r#rr&r&r'r0s    zImageGPTMLP.forward)r1r2r3rr r!r0r7r&r&r$r'r s rcseZdZdfdd Z       ddejdedBdejdBdejdBd ejdBd edBd edBd ejdBd efddZ Z S) ImageGPTBlockNcst|j}|jdur|jnd|}t||jd|_t||d|_t||jd|_ |j r>t|d|d|_ t||jd|_ t |||_dS)Nrr:T)r9r:)rrrn_innerrlayer_norm_epsilonln_1r8attnln_2add_cross_attentioncrossattention ln_cross_attnrmlp)r#r@r:r inner_dimr$r&r'rs zImageGPTBlock.__init__Frrrjrrrrrr)c  Cs|} ||}|j||||||d} | d} | dd} | | }|durTt|ds1td|d|} ||}|j|||||||d} | d} | | }| | dd} |} ||}||}| |}|f| S)N)rrjrrrrrrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`)rrjrrrr)rrrrLrrrr)r#rrrjrrrrrresidual attn_outputsrpoutputscross_attn_outputsfeed_forward_hidden_statesr&r&r'r0.sJ         zImageGPTBlock.forwardrr) r1r2r3rr r!rrEr4r0r7r&r&r$r'rs8  rcsDeZdZUeed<dZdZdZdZdgZ e fddZ Z S) ImageGPTPreTrainedModelr@ transformer input_ids)imageTrc st|t|tr1|D]\}}d|vr.d|vr.tj|d|jjt d|jj dqdSt|t rT|jj }t|jttj||ftjddd||dSdS) zInitialize the weights.rRr"gr?)r.stdr<rN)r _init_weightsrrnamed_parametersinitnormal_r@initializer_rangemathr-n_layerr8rAcopy_r;r rCrDrErF)r#modulenameprXr$r&r'rqs  & z%ImageGPTPreTrainedModel._init_weights)r1r2r3r__annotations__base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_no_split_modulesr no_gradrr7r&r&r$r'rhs rc!seZdZdeffdd ZddZddZe             dd ej dBd e dBd ej dBd ej dBd ej dBdej dBdej dBdej dBde dBde dBde dBde dBdej dBde de eBfddZZS) ImageGPTModelr@cstj|_tj|j|_tj|j|_ t j |_ t fddtjD|_t|jjd|_d|_|dS)Ncsg|]}t|dqS)r)r).0ir@r&r' sz*ImageGPTModel.__init__..rF)rrrrGr Embedding vocab_sizewterAwperS embd_pdropdrop ModuleListrangenum_hidden_layershrrln_fgradient_checkpointing post_initr#r@r$rr'rs   zImageGPTModel.__init__cCs|jSrr)r#r&r&r'get_input_embeddingssz"ImageGPTModel.get_input_embeddingscCs ||_dSrr)r#new_embeddingsr&r&r'set_input_embeddingss z"ImageGPTModel.set_input_embeddingsNrpast_key_valuesrjtoken_type_ids position_ids inputs_embedsrrrroutput_hidden_states return_dictrkwargsr)c! Ks| dur| n|jj} | dur| n|jj} | dur| n|jj} | dur$| n|jj} |dur4|dur4td|durP||||}|d|d}|j d}n|durb|dd}|j d}ntd|durm|j n|j }|j r|j r| rt dd} |dur|d|d}| r|durt|jd}| dur|dur|nd}tj|d|d |} |dur| d}|dur|dkrtd ||d}|ddddddf}|j|jd }d |t|jj}|jjr|dur|\}}}||f}|dur tj||d }||}nd}|dur||}||}|||j }|dur7||}||}||}||df}| rId nd}| rU|jjrUd nd}| r\d nd}t|jD]3\}}| ro||f}||||||| | | d} | d}| r|| df}|jjr|| df}qc| |}|j|}| r||f}| st!dd|||||fDSt"|||||dS)a; input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoImageProcessor`]. See [`ImageGPTImageProcessor.__call__`] for details. Examples: ```python >>> from transformers import AutoImageProcessor, ImageGPTModel >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> image_processor = AutoImageProcessor.from_pretrained("openai/imagegpt-small") >>> model = ImageGPTModel.from_pretrained("openai/imagegpt-small") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer*rz5You have to specify either input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fr)r\z$batch_size has to be defined and > 0r<rrr&)rrrrrr?css|] }|dur|VqdSrr&)rvr&r&r' Isz(ImageGPTModel.forward..)last_hidden_staterr attentionscross_attentions)#r@rrruse_return_dictrL%warn_if_padding_and_no_attention_maskrarFrr\rtraininglogger warning_oncer get_seq_lengthr arange unsqueezetor=rbrcrrDinvert_attention_maskrrr enumeraterrr4r )!r#rrrjrrrrrrrrrrr input_shape batch_sizer\past_seen_tokensencoder_batch_sizeencoder_sequence_lengthr~encoder_hidden_shapeposition_embedsrtoken_type_embeds output_shapeall_self_attentionsall_cross_attentionsall_hidden_statesrblockrr&r&r'r0s1                       zImageGPTModel.forward) NNNNNNNNNNNNN)r1r2r3rrrrrr r!rrErr4r r0r7r&r&r$r'rsb     rz The ImageGPT Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introc#seZdZddiZdeffdd Ze              ddejdBde dBd ejdBd ejdBd ejdBd ejdBd ejdBdejdBdejdBde dBde dBde dBde dBdejdBde de e Bf ddZZS)ImageGPTForCausalImageModelingzlm_head.weightztransformer.wte.weightr@cs<t|t||_tj|j|jddd|_| dS)NrFr;) rrrrrLinearn_embdrlm_headrrr$r&r'ras   z'ImageGPTForCausalImageModeling.__init__Nrrrjrrrrrlabelsrrrrrrr)cKs| dur| n|jj} |j||||||||| | | | |d }|d}||}d}| durT|dddddf}| dddf}t}||d|d|d}| sj|f|dd}|durh|f|S|St|||j |j |j |j dS)a& input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoImageProcessor`]. See [`ImageGPTImageProcessor.__call__`] for details. labels (`torch.LongTensor` of shape `(batch_size, input_ids_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import AutoImageProcessor, ImageGPTForCausalImageModeling >>> import torch >>> import matplotlib.pyplot as plt >>> import numpy as np >>> image_processor = AutoImageProcessor.from_pretrained("openai/imagegpt-small") >>> model = ImageGPTForCausalImageModeling.from_pretrained("openai/imagegpt-small") >>> device = torch.device("cuda" if torch.cuda.is_available() else "cpu") >>> model.to(device) # doctest: +IGNORE_RESULT >>> # unconditional generation of 8 images >>> batch_size = 4 >>> context = torch.full((batch_size, 1), model.config.vocab_size - 1) # initialize with SOS token >>> context = context.to(device) >>> output = model.generate( ... input_ids=context, max_length=model.config.n_positions + 1, temperature=1.0, do_sample=True, top_k=40 ... ) >>> clusters = image_processor.clusters >>> height = image_processor.size["height"] >>> width = image_processor.size["width"] >>> samples = output[:, 1:].detach().cpu().numpy() >>> samples_img = [ ... np.reshape(np.rint(127.5 * (clusters[s] + 1.0)), [height, width, 3]).astype(np.uint8) for s in samples ... ] # convert color cluster tokens back to pixels >>> f, axes = plt.subplots(1, batch_size, dpi=300) >>> for img, ax in zip(samples_img, axes): # doctest: +IGNORE_RESULT ... ax.axis("off") ... ax.imshow(img) ```N) rrjrrrrrrrrrrr.r*r)losslogitsrrrr) r@rrrrrrFrarrrrr)r#rrrjrrrrrrrrrrrrtransformer_outputsr lm_logitsr shift_logits shift_labelsloss_fctoutputr&r&r'r0isFF z&ImageGPTForCausalImageModeling.forward)NNNNNNNNNNNNNN)r1r2r3_tied_weights_keysrrrr r!rrErr4rr0r7r&r&r$r'rXsf     rz The ImageGPT Model transformer with an image classification head on top (linear layer). [`ImageGPTForImageClassification`] average-pools the hidden states in order to do the classification. cseZdZdeffdd Ze           ddejdBdedBdejdBdejdBd ejdBd ejdBd ejdBd e dBd e dBde dBde dBde de e BfddZ ZS)ImageGPTForImageClassificationr@cs@t||j|_t||_tj|j|jdd|_| dS)NFr) rr num_labelsrrrrrscorerrr$r&r'rs   z'ImageGPTForImageClassification.__init__Nrrrjrrrrrrrrrr)c  Ks| dur| n|jj} |j|||||||| | | d } | d}|jdd}||}d}|dur6||||j}| sL|f| dd}|durJ|f|S|St||| j| j| j dS)a input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoImageProcessor`]. See [`ImageGPTImageProcessor.__call__`] for details. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, ImageGPTForImageClassification >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> image_processor = AutoImageProcessor.from_pretrained("openai/imagegpt-small") >>> model = ImageGPTForImageClassification.from_pretrained("openai/imagegpt-small") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits ```N) rrjrrrrrrrrrr])rrrrr) r@rrr.r loss_functionrrrr)r#rrrjrrrrrrrrrrrpooled_hidden_statesrrrr&r&r'r0s:3  z&ImageGPTForImageClassification.forward) NNNNNNNNNNN)r1r2r3rrrr r!rrErr4rr0r7r&r&r$r'rsR      r)rrrr)2__doc__rtypingrr rtorch.nnrrr activationsr cache_utilsrr r generationr modeling_layersr modeling_outputsr rrmodeling_utilsr pytorch_utilsrutilsrrr utils.genericrconfiguration_imagegptr get_loggerr1rModulerr8rrrrrr__all__r&r&r&r'sJ              RJ"M} b