o ߥi?@sddlmZddlZddlmZddlmmZddlm Z m Z m Z m Z m Z ddlmZmZmZmZmZmZmZmZGdddejZGdd d ejZGd d d eZdS) )partialN)AdapterLoRAPrefixPromptSideTune) AttentionBlockDropPath LayerScaleMlp PatchEmbedVisionTransformercheckpoint_seqcs:eZdZdZ        d fdd Zdd ZZS) AttentionPETLa=Extend the parameter-efficient transfer learning (PETL) method to the original Attention. Prefix tuning optimizes the task-specific vector in the multi-head attention layer. 'Prefix-tuning: Optimizing continuous prompts for generation' by Li & Liang(2021) See https://arxiv.org/abs/2101.00190 LoRA constructs an additional layer with low-rank decomposition matrices of the weights in the network. 'LoRA: Low-Rank Adaptation of Large Language Models' by Hu et al.(2021) See https://arxiv.org/abs/2106.09685 Attributes: prefix_length: An integer indicating the length of prefix tuning. prefix_type: A string indicating the type of prefix tuning. lora_length: An integer indicating the length of LoRA tuning. lora_type: A string indicating the type of LoRA tuning. FNc st||dksJd||_||} | d|_tj||d|d|_t||_t|||_ t||_ |rI|dkrIt |||| d|_ nd|_ |r]|dkr]t ||||d|_dSd|_dS)Nrz$dim should be divisible by num_headsg)bias)dim num_heads lora_length lora_type)rr prefix_length prefix_type)super__init__rscalennLinearqkvDropout attn_dropproj proj_droprlorarprefix) selfrrqkv_biasr#r%rrrrhead_dim __class__i/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/models/cv/vision_efficient_tuning/backbone.pyr"s2        zAttentionPETL.__init__c Cs|j\}}}||||d|j||jddddd}|d\}}}|jdur5|||||\}}}|jdurE|||||\}}}||dd|j } | j dd} | | } | |dd|||}| |}| |}|S) Nrrrr)shaper!reshaperpermuteunbindr&r' transposersoftmaxr#r$r%) r(xBNCr!qkvattnr-r-r.forwardKs$       zAttentionPETL.forward)rFrrNNNN)__name__ __module__ __qualname____doc__rrB __classcell__r-r-r+r.rs)rcsReZdZdZddddddejejedddddddddffdd Zd d Z Z S) BlockPETLa)Extend the parameter-efficient transfer learning (PETL) method to the original Block. Visual prompt tuning (VPT) is proposed to initialize tunable prompt tokens and prepend to the original tokens in the first layer or multiple layers. 'Visual Prompt Tuning' by Jia et al.(2022) See https://arxiv.org/abs/2203.12119 Adapters project input tokens by an MLP layer. 'Parameter-Efficient Transfer Learning for NLP' by Houlsby et al.(2019) See http://arxiv.org/abs/1902.00751 Attributes: adapter_length: An integer indicating the length of adapter tuning. adapter_type: A string indicating the type of adapter tuning. prompt_length: An integer indicating the length of prompt tuning. prompt_type: A string indicating the type of prompt tuning. @FrNr2c s,t| |_| ||_| |||||||||d |_|r#t||dnt|_|dkr0t |nt|_ | ||_ t |t ||| |d|_|rOt||dnt|_|dkr\t |nt|_||_||_|rw|dkrwt|||| d|_nd|_| |_||_| r| dkrt|| | |d|_dSd|_dS)N)rr)r#r%rrrr) init_valuesr) in_featureshidden_features act_layerdropr)radapter_length adapter_typerM)r layer_num prompt_length prompt_type)rrrQnorm1rAr rIdentityls1r drop_path1norm2r intmlpls2 drop_path2rOrPradapterrRrSrprompt)r(rr mlp_ratior)rNr#rJ drop_pathrM norm_layer attn_layerrQrRrSrrrOrPrrr+r-r.rus         zBlockPETL.__init__c Cs|jdur|jr|jdkr||}||||||}|jdur<||||| | |}|S|||| | |}|S)Nr) r^rRrWrVrArTr]r\r[rZrX)r(r:r-r-r.rBs    zBlockPETL.forward) rCrDrErFrGELU LayerNormr rrBrGr-r-r+r.rHbs,LrHc!s|eZdZdZddddddddd d d d d d d d d d ded d ed d d d d d d d d d f!fdd ZddZddefddZ Z S)VisionTransformerPETLa] Extend the parameter-efficient transfer learning (PETL) method to the original Vision Transformer. A PyTorch impl of : `An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale` - https://arxiv.org/abs/2010.11929 The implementation of several tuning methods (prompt, prefix, adapter, and LoRA) based on ViT. ritokeni rITNFrc"%st|dvs J| s|dksJ|dur|dkn|}" p%ttjdd p*tj||_||_|_|_ | r;dnd|_ | |_ d |_ ||_ ||_| |_|_|_|_|_|_|_ |_ |_||||| d |_|jj}#| rttddnd|_| r|#n|#|j }$ttd|$d |_tj d |_!|r nt"|_#d dt$d||DdusLjdusLjdusLj durt%t&tj' fddt(|D|_)ntj' f ddt(|D|_)|"s nt"|_*|"r nt"|_+|dkr+t,|j |nt"|_-|dkr:|.||!durGt/| |!|_0dSd|_0dS)a Initialize a Parameter-efficient Transfer Learning Method based on Vision Transformer. Args: img_size (int, tuple): input image size patch_size (int, tuple): patch size in_chans (int): number of input channels num_classes (int): number of classes for classification head global_pool (str): type of global pooling for final sequence (default: 'token') embed_dim (int): embedding dimension depth (int): depth of transformer num_heads (int): number of attention heads mlp_ratio (int): ratio of mlp hidden dim to embedding dim qkv_bias (bool): enable bias for qkv if True init_values: (float): layer-scale init values class_token (bool): use class token fc_norm (Optional[bool]): pre-fc norm after pool, set if global_pool == 'avg' if None (default: None) drop_rate (float): dropout rate attn_drop_rate (float): attention dropout rate drop_path_rate (float): stochastic depth rate weight_init (str): weight init scheme embed_layer (nn.Module): patch embedding layer norm_layer: (nn.Module): normalization layer act_layer: (nn.Module): MLP activation layer prompt_length: An integer indicating the length of prompt tuning. prompt_type: A string indicating the type of prompt tuning. prefix_length: An integer indicating the length of prefix tuning. prefix_type: A string indicating the type of prefix tuning. adapter_length: An integer indicating the length of adapter tuning. adapter_type: A string indicating the type of adapter tuning. lora_length: An integer indicating the length of LoRA tuning. lora_type: A string indicating the type of LoRA tuning. sidetune_length: An integer indicating the linear dimension. sidetune_type: A string indicating the type of side network. )rjavgrhrhNrkgư>)epsrrF)img_size patch_sizein_chans embed_dimrg{Gz?)pcSsg|]}|qSr-)item).0r:r-r-r. Lsz2VisionTransformerPETL.__init__..c sg|]i}diddd dd ddd|d d d d |d ttr7|nd dttrF|nddttrU|nddt trd |n d qS)rrr_r)rJrNr#r`rarMrbrQrRrSrrrOrPrrr-) isinstancelistrsirMrOrPattn_drop_raterbblock_fndpr drop_raterprJrrr_rarrrrRrSr)r-r.rtQsx         cs,g|]}  |d qS)) rrr_r)rJrNr#r`rarMr-rw) rMrzr{r|r}rprJr_rarr)r-r.rtms skip)1rrrrrdrc num_classes global_pool num_featuresrpnum_prefix_tokensno_embed_classgrad_checkpointingdepthrm class_tokenrRrSrrrOrPrr patch_embed num_patches Parametertorchzeros cls_tokenrandn pos_embedr"pos_droprUnorm_prelinspacerrH Sequentialrangeblocksnormfc_normr head init_weightsrsidetune)%r(rmrnrorrrprrr_r)rJrrpre_normrr}rzdrop_path_rate weight_init embed_layerrarMr{rRrSrrrOrPrrsidetune_length sidetune_type use_fc_normr embed_lenr+ryr.rs E    4 "      zVisionTransformerPETL.__init__cCsft|d}||}||}||}|jr#tjs#t|j |}n| |}| |}||d<|S)a feature forward function of VisionTransformer. Args: x (Tensor): the input data. Returns: res (Dict): the output data, contains: - inputs: the original input. - x: the intermediate feature. )inputsr:) dictr _pos_embedrrrjit is_scriptingrrr)r(r:resr-r-r.forward_featuress     z&VisionTransformerPETL.forward_features pre_logitscCs|d}|jr#|jdkr|dd|jdfjddn|dddf}|jr2d|vr2||d|}||}|r;|S||S)a  head forward function of VisionTransformer. Args: res (Dict): the input data, contains: - inputs: the original input. - x: the intermediate feature. Returns: x (Tensor): the output data. r:rkNrr3rr)rrmeanrrr)r(rrr:r-r-r. forward_heads   z"VisionTransformerPETL.forward_head)F) rCrDrErFrr rrboolrrGr-r-r+r.resN 2re) functoolsrrtorch.nnrtorch.nn.functional functionalFpetlrrrrrtimm_vision_transformerr r r r r rrrModulerrHrer-r-r-r.s  (Rm