o ei,@s dZddlmZddlZddlmZddlmZddlm Z ddl m Z dd l m Z dd lmZdd lmZmZmZmZdd lmZd dlmZeeZeeddGdddeZeGddde ZdddZGdddejZ GdddejZ!eddGdddeZ"ddgZ#dS) zPyTorch VitPose model.) dataclassN)nn)initialization) load_backbone)BackboneOutput)PreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringlogging)can_return_tuple) VitPoseConfigz6 Class for outputs of pose estimation models. ) custom_introc@sjeZdZUdZdZejdBed<dZejdBed<dZ e ejdfdBed<dZ e ejdfdBed<dS)VitPoseEstimatorOutputaH loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Loss is not supported at this moment. See https://github.com/ViTAE-Transformer/ViTPose/tree/main/mmpose/models/losses for further detail. heatmaps (`torch.FloatTensor` of shape `(batch_size, num_keypoints, height, width)`): Heatmaps as predicted by the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states (also called feature maps) of the model at the output of each stage. Nlossheatmaps. hidden_states attentions) __name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rrtuplerrrj/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/vitpose/modeling_vitpose.pyr$s  rc@sJeZdZUeed<dZdZdZdZe de j e j Be jBfddZd S) VitPosePreTrainedModelconfigvit pixel_values)imageTmodulecCsrt|tjtjfr#tj|jd|jjd|j dur!t |j dSdSt|tj r7t |j t |jdSdS)zInitialize the weightsg)meanstdN) isinstancerLinearConv2dinit trunc_normal_weightr"initializer_rangebiaszeros_ LayerNormones_)selfr&rrr _init_weightsDs   z$VitPosePreTrainedModel._init_weightsN)rrrrrbase_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointingrno_gradrr*r+r2r5rrrr r!<s "r!gaussian-heatmapc Cs|dvrtd|jdkrtd|j\}}}}d}|dkr7d}|dddddd f |dddddd f<||d |||}|}|D]$\} } |dd| d f|dd| d f<|dd| d f|dd| d f<qH|||||f}|d }|S) aFlip the flipped heatmaps back to the original form. Args: output_flipped (`torch.tensor` of shape `(batch_size, num_keypoints, height, width)`): The output heatmaps obtained from the flipped images. flip_pairs (`torch.Tensor` of shape `(num_keypoints, 2)`): Pairs of keypoints which are mirrored (for example, left ear -- right ear). target_type (`str`, *optional*, defaults to `"gaussian-heatmap"`): Target type to use. Can be gaussian-heatmap or combined-target. gaussian-heatmap: Classification target with gaussian distribution. combined-target: The combination of classification target (response map) and regression target (offset map). Paper ref: Huang et al. The Devil is in the Details: Delving into Unbiased Data Processing for Human Pose Estimation (CVPR 2020). Returns: torch.Tensor: heatmaps that flipped back to the original image )r;combined-targetz9target_type should be gaussian-heatmap or combined-targetzCoutput_flipped should be [batch_size, num_keypoints, height, width]rr<rN.) ValueErrorndimshapereshapeclonetolistflip) output_flipped flip_pairs target_type batch_size num_keypointsheightwidthchannelsoutput_flipped_backleftrightrrr flip_backPs" . " rQcsHeZdZdZdeffdd Zd dejdejdBdejfd d ZZ S) VitPoseSimpleDecoderz Simple decoding head consisting of a ReLU activation, 4x upsampling and a 3x3 convolution, turning the feature maps into heatmaps. r"csHtt|_tj|jddd|_tj|j j |j dddd|_ dS)NbilinearF) scale_factormode align_cornersrr kernel_sizestridepadding) super__init__rReLU activationUpsamplerT upsamplingr+backbone_config hidden_size num_labelsconvr4r" __class__rr r\~s   zVitPoseSimpleDecoder.__init__N hidden_staterGreturncCs4||}||}||}|durt||}|SN)r^r`rdrQr4rhrGrrrr forwards    zVitPoseSimpleDecoder.forwardrj rrrrrr\rTensorrl __classcell__rrrfr rRxs* rRcsBeZdZdZdeffdd Zd dejdejdBfdd ZZ S) VitPoseClassicDecoderz Classic decoding head consisting of a 2 deconvolutional blocks, followed by a 1x1 convolution layer, turning the feature maps into heatmaps. r"csttj|jjdddddd|_td|_t |_ tjddddddd|_ td|_ t |_ tjd|jdddd|_dS) Nr=rF)rXrYrZr0rrW)r[r\rConvTranspose2drarbdeconv1 BatchNorm2d batchnorm1r]relu1deconv2 batchnorm2relu2r+rcrdrerfrr r\s     zVitPoseClassicDecoder.__init__NrhrGcCs\||}||}||}||}||}||}||}|dur,t||}|Srj)rtrvrwrxryrzrdrQrkrrr rls        zVitPoseClassicDecoder.forwardrjrmrrrfr rps$rpz? The VitPose model with a pose estimation head on top. csleZdZdeffdd Zee   d dejdejdBdejdBdejdBd e e d e f d d Z Z S)VitPoseForPoseEstimationr"cs|t|t||_t|jjdstdt|jjds!tdt|jjds,td|jr3t|nt ||_ | dS)Nrbz0The backbone should have a hidden_size attribute image_sizez0The backbone should have an image_size attribute patch_sizez/The backbone should have a patch_size attribute) r[r\rbackbonehasattrr"r?use_simple_decoderrRrphead post_initrerfrr r\s   z!VitPoseForPoseEstimation.__init__Nr$ dataset_indexrGlabelskwargsric Ksd}|dur td|jj|fd|i|}|jd}|jd} |jjjd|jjjd} |jjjd|jjjd} | ddd}| | d| |  }|j ||d} t || |j|jd S) a dataset_index (`torch.Tensor` of shape `(batch_size,)`): Index to use in the Mixture-of-Experts (MoE) blocks of the backbone. This corresponds to the dataset index used during training, e.g. For the single dataset index 0 refers to the corresponding dataset. For the multiple datasets index 0 refers to dataset A (e.g. MPII) and index 1 refers to dataset B (e.g. CrowdPose). flip_pairs (`torch.tensor`, *optional*): Whether to mirror pairs of keypoints (for example, left ear -- right ear). Examples: ```python >>> from transformers import AutoImageProcessor, VitPoseForPoseEstimation >>> import torch >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> processor = AutoImageProcessor.from_pretrained("usyd-community/vitpose-base-simple") >>> model = VitPoseForPoseEstimation.from_pretrained("usyd-community/vitpose-base-simple") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> boxes = [[[412.8, 157.61, 53.05, 138.01], [384.43, 172.21, 15.12, 35.74]]] >>> inputs = processor(image, boxes=boxes, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) >>> heatmaps = outputs.heatmaps ```NzTraining is not yet supportedrr>rrrr)rG)rrrr)NotImplementedErrorr~forward_with_filtered_kwargs feature_mapsrAr"rar|r}permuterB contiguousrrrr) r4r$rrGrrroutputssequence_outputrI patch_height patch_widthrrrr rls.)  z VitPoseForPoseEstimation.forward)NNN)rrrrr\rr rrnr r rrlrorrrfr r{s(r{)r;)$r dataclassesrrrrr,backbone_utilsrmodeling_outputsrmodeling_utilsrprocessing_utilsr utilsr r r r utils.genericrconfiguration_vitposer get_loggerrloggerrr!rQModulerRrpr{__all__rrrr s8           (& X