o eiy*@sdZddlmZddlZddlmZddlmZddlm Z ddl m Z dd l m Z mZd d lmZeed d Gddde ZeGddde ZGdddejZGdddejZGdddejZGdddejZGdddejZedd GdddeZddgZdS)zPyTorch ViTMatte model.) dataclassN)nn)initialization) load_backbone)PreTrainedModel) ModelOutputauto_docstring)VitMatteConfigz4 Class for outputs of image matting models. ) custom_introc@sbeZdZUdZdZejdBed<dZejdBed<dZ e ejdBed<dZ e ejdBed<dS)ImageMattingOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Loss. alphas (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Estimated alpha values. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states (also called feature maps) of the model at the output of each stage. Nlossalphas hidden_states attentions) __name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rrtuplerrrl/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/vitmatte/modeling_vitmatte.pyr s  r c@s>eZdZUeed<dZdZdZgZe de j fddZ dS) VitMattePreTrainedModelconfig pixel_values)imageTmodulecCs~t|tjtjfr;tj|jd|jjd|j durt |j t |dddur=t |j t |jt |jdSdSdS)Ng)meanstd running_mean) isinstancerConv2d BatchNorm2dinitnormal_weightrinitializer_rangebiaszeros_getattrr#ones_ running_varnum_batches_tracked)selfr rrr _init_weights<s    z%VitMattePreTrainedModel._init_weightsN)rrrr rmain_input_nameinput_modalitiessupports_gradient_checkpointing_no_split_modulesrno_gradrModuler2rrrrr4s rcs*eZdZdZdfdd ZddZZS) VitMatteBasicConv3x3zP Basic convolution layers including: Conv3x3, BatchNorm2d, ReLU layers. r csBttj||d||dd|_tj||jd|_t|_ dS)NrF) in_channels out_channels kernel_sizestridepaddingr+)eps) super__init__rr%convr&batch_norm_eps batch_normReLUrelu)r1rr;r<r>r? __class__rrrBMs zVitMatteBasicConv3x3.__init__cCs"||}||}||}|SN)rCrErGr1 hidden_staterrrforwardZs   zVitMatteBasicConv3x3.forward)r:r rrrrrBrM __classcell__rrrHrr9Hs r9c(eZdZdZfddZddZZS)VitMatteConvStreamzc Simple ConvStream containing a series of basic conv3x3 layers to extract detail features. cstd}|jdur|jj}|j}t|_|g||_t t |jdD]}|j|}|j|d}|j t |||q'dS)Nr ) rArBbackbone_config num_channelsconvstream_hidden_sizesr ModuleListconvs conv_chansrangelenappendr9)r1rr;r<iin_chan_ out_chan_rHrrrBgs     zVitMatteConvStream.__init__cCsJd|i}|}tt|jD]}|j||}dt|d}|||<q |S)Ndetailed_feature_map_0detailed_feature_map_r )rYrZrWstr)r1rout_dict embeddingsr\name_rrrrMzs zVitMatteConvStream.forwardrNrrrHrrQbs rQcrP)VitMatteFusionBlockz\ Simple fusion block to fuse features from ConvStream and Plain Vision Transformer. cs"tt|||ddd|_dS)Nr )r>r?)rArBr9rC)r1rr;r<rHrrrBs zVitMatteFusionBlock.__init__cCs4tjj|dddd}tj||gdd}||}|S)Nr:bilinearF) scale_factormode align_cornersr )dim)r functional interpolatercatrC)r1featuresdetailed_feature_mapupscaled_featuresoutrrrrMs zVitMatteFusionBlock.forwardrNrrrHrres recrP) VitMatteHeadzJ Simple Matting Head, containing only conv3x3 and conv1x1 layers. c sZt|jd}d}ttj||ddddt|tdtj|ddddd|_dS)Nrr )r=r>r?Tr) rArBfusion_hidden_sizesr Sequentialr%r&rF matting_convs)r1rr; mid_channelsrHrrrBs   zVitMatteHead.__init__cCs||}|SrJ)rwrKrrrrMs zVitMatteHead.forwardrNrrrHrrrs  rrcrP)VitMatteDetailCaptureModulezG Simple and lightweight Detail Capture Module for ViT Matting. c stt|jt|jdkrtd||_t||_|jj |_ t |_ |j g|j|_tt|jdD]}|j t||j||j |d |j|ddq8t||_dS)Nr z_The length of fusion_hidden_sizes should be equal to the length of convstream_hidden_sizes + 1.)rr;r<)rArBrZrurU ValueErrorrrQ convstreamrXrrV fusion_blocks hidden_sizefusion_channelsrYr[rerr matting_head)r1rr\rHrrrBs&     z$VitMatteDetailCaptureModule.__init__cCs`||}tt|jD]}dtt|j|d}|j||||}q t||}|S)Nr`r )r{rYrZr|rarsigmoidr)r1rnrdetail_featuresr\detailed_feature_map_namerrrrrMs z#VitMatteDetailCaptureModule.forwardrNrrrHrrys ryzX ViTMatte framework leveraging any vision backbone e.g. for ADE20k, CityScapes. c sbeZdZfddZe     d dejdBdedBdedBdejdBdedBf d d ZZ S) VitMatteForImageMattingcs2t|||_t||_t||_|dSrJ)rArBrrbackbonerydecoder post_init)r1rrHrrrBs    z VitMatteForImageMatting.__init__Nroutput_attentionsoutput_hidden_stateslabels return_dictc Ks|dur|n|jj}|dur|n|jj}|dur|n|jj}d}|dur(td|jj|||d}|jd} || |} |sR| f|dd} |durP|f| S| St || |j |j dS)a8 labels (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*): Ground truth image matting for computing the loss. Examples: ```python >>> from transformers import VitMatteImageProcessor, VitMatteForImageMatting >>> import torch >>> from PIL import Image >>> from huggingface_hub import hf_hub_download >>> processor = VitMatteImageProcessor.from_pretrained("hustvl/vitmatte-small-composition-1k") >>> model = VitMatteForImageMatting.from_pretrained("hustvl/vitmatte-small-composition-1k") >>> filepath = hf_hub_download( ... repo_id="hf-internal-testing/image-matting-fixtures", filename="image.png", repo_type="dataset" ... ) >>> image = Image.open(filepath).convert("RGB") >>> filepath = hf_hub_download( ... repo_id="hf-internal-testing/image-matting-fixtures", filename="trimap.png", repo_type="dataset" ... ) >>> trimap = Image.open(filepath).convert("L") >>> # prepare image + trimap for the model >>> inputs = processor(images=image, trimaps=trimap, return_tensors="pt") >>> with torch.no_grad(): ... alphas = model(**inputs).alphas >>> print(alphas.shape) torch.Size([1, 1, 640, 960]) ```NzTraining is not yet supported)rrrsr )rrrr) ruse_return_dictrrNotImplementedErrorrforward_with_filtered_kwargs feature_mapsrr rr) r1rrrrrkwargsroutputsrnroutputrrrrMs**  zVitMatteForImageMatting.forward)NNNNN) rrrrBr rTensorboolrMrOrrrHrrs&  r)r dataclassesrrrrr'backbone_utilsrmodeling_utilsrutilsrr configuration_vitmatter r rr8r9rQrerrryr__all__rrrrs4      #) R