o ߥi|$@sddlZddlmZddlZddlmZddlZGdddejZGdddejZ GdddejZ Gd d d ejZ Gd d d ejZ Gd ddejZ dS)N) OrderedDictc*eZdZdZdfdd ZddZZS)PromptaQThe implementation of vision prompt tuning method. Visual prompt tuning (VPT) is proposed to initialize tunable prompt tokens and prepend to the original tokens in the first layer or multiple layers. 'Visual Prompt Tuning' by Jia et al.(2022) See https://arxiv.org/abs/2203.12119 Attributes: dim: An integer indicating the embedding dimension. layer_num: An integer indicating number of layers. prompt_length: An integer indicating the length of vision prompt tuning. prompt_type: A string indicating the type of vision prompt tuning. NcsNtt|||_||_||_||_tt d|||_ tj |j dSN)superr__init__dim layer_num prompt_length prompt_typenn Parametertorchzeros prompt_tokeninitxavier_uniform_)selfr r r r  __class__e/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/models/cv/vision_efficient_tuning/petl.pyrszPrompt.__init__cCsl|j\}}}|j|dd}|jdkrtj||fdd}|Stj|ddd|j ddf|fdd}|S)Nrrr )shaperexpandr rcatr )rxBNCrrrrforward#s  ,zPrompt.forwardNN__name__ __module__ __qualname____doc__rr" __classcell__rrrrr s rcs<eZdZdZddejffdd ZddZd ddZZ S) AdapteraThe implementation of adapter tuning method. Adapters project input tokens by an MLP layer. 'Parameter-Efficient Transfer Learning for NLP' by Houlsby et al.(2019) See http://arxiv.org/abs/1902.00751 Attributes: dim: An integer indicating the embedding dimension. adapter_length: An integer indicating the length of adapter tuning. adapter_type: A string indicating the type of adapter tuning. NcsPtt|||_||_||_t|||_||_ t|||_ | dSN) rr*rr adapter_length adapter_typer Linearln1activateln2 init_weights)rr r,r- act_layerrrrr;s zAdapter.__init__cCsdd}||dS)NcSs4t|tjrtj|jtjj|jdddSdS)Ngư>)std) isinstancer r.rrweightnormal_bias)mrrr _init_weightsMs z+Adapter.init_weights.._init_weights)apply)rr:rrrr2KszAdapter.init_weightscCs.||||}|dur|}||}|Sr+)r1r0r/)rridentityoutrrrr"Ts zAdapter.forwardr+) r%r&r'r(r GELUrr2r"r)rrrrr*.s r*c.eZdZdZ  dfdd ZddZZS)LoRAa'The implementation of LoRA tuning method. LoRA constructs an additional layer with low-rank decomposition matrices of the weights in the network. 'LoRA: Low-Rank Adaptation of Large Language Models' by Hu et al.(2021) See https://arxiv.org/abs/2106.09685 Attributes: dim: An integer indicating the embedding dimension. num_heads: An integer indicating number of attention heads. lora_length: An integer indicating the length of LoRA tuning. lora_type: A string indicating the type of LoRA tuning. Ncs|tt|||_||_tj||dd|_tjj |jj t ddtj||ddd|_ tj|j j ||_||_dS)NFr8)a)rr@rr num_headsr r.lora_arkaiming_uniform_r6mathsqrtlora_bzeros_ lora_length lora_type)rr rErLrMrrrrjs z LoRA.__init__c Csz|j\}}}|||}|||d|j||jddddd}|d\} } } || || || }}}|||fS)NrDrr)rrJrFreshaperEpermuteunbind) rrqkvrr r! qkv_deltaq_deltak_deltav_deltarrrr"|s   z LoRA.forwardr#r$rrrrr@\s r@cr?)Prefixa'The implementation of prefix tuning method. Prefix tuning optimizes the task-specific vector in the multi-head attention layer. 'Prefix-tuning: Optimizing continuous prompts for generation' by Li & Liang(2021) See https://arxiv.org/abs/2101.00190 Attributes: dim: An integer indicating the embedding dimension. num_heads: An integer indicating number of attention heads. prefix_length: An integer indicating the length of prefix tuning. prefix_type: A string indicating the type of prefix tuning. Ncsrtt|||_||_||_||_tt d|||_ tt d|||_ tj |j tj |j dSr)rrZrr rE prefix_length prefix_typer rrr prefix_key prefix_valuerr)rr rEr[r\rrrrszPrefix.__init__c Cs|j\}}}|j|dd||j|j|j|jdddd}|j|dd||j|j|j|jdddd} t j ||fddt j || fdd}}|||fS)NrrrNrrDr) rr]rrPr[rEr rQr^rr) rrrSrTrUrr r!r]r^rrrr"s"        zPrefix.forwardr#r$rrrrrZs rZcr)SideTuneaThe implementation of vision side-tuning method. Side-Tuning only needs to train one side network and weights the output of pre-trained model and side network. 'Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks' by Zhang et al.(2019) See https://arxiv.org/abs/1912.13503 Attributes: sidetune_length: An integer indicating the linear dimension. sidetune_type: A string indicating the type of side network. Nc stt|||_||_|dkrt|jd|_|dkrFtj j dd}t t d|jfd|jfdt fd t jd |jd d fg|_t td |_dS)Nfcn4)out_dimsalexnetT) pretrainedfeaturesavgpoolflattenfci$FrAg)rr_rsidetune_length sidetune_typelowerFCN4side torchvisionmodelsrbr SequentialrrdreFlattenr.rrtensoralpha)rrhrimmrrrrs   zSideTune.__init__cCs.t|j}||}||d||}|Sr)rsigmoidrrrl)rrx_basealpha_squashedx_sidex_outrrrr"s  zSideTune.forwardr#r$rrrrr_s r_cr)rkz@The implementation of simple FCN4 network for side network. rc stt|jd i|ttjddddddddtddt|_ttjddddddddtddt|_ ttjddddddddtddt|_ ttjdd ddddddtdd t|_ t d |_ |dkr~td ||_dSd|_dS) NrDrF) kernel_sizestridepaddingr8dilationrNr @)rrr)rrkrr roConv2d GroupNormReLUconv1conv2conv3conv4AdaptiveAvgPool2dpoolr.rg)rrakwargsrrrrsj     z FCN4.__init__cCs\||}||}||}||}||}||dd}|jdur,||}|S)Nrr)rrrrrviewsizerg)rrrrrr" s       z FCN4.forward)rr$rrrrrks-rk)rH collectionsrrtorch.nnr rmModulerr*r@rZr_rkrrrrs  $.+,%