o پif@s<dZddlmZmZmZmZddlZddlmZddlm Z ddl m Z m Z mZmZmZmZmZmZmZmZmZgdZeejZdeedefd d ZGd d d ejZGd ddejZGdddejZGdddejZGdddejZ GdddejZ!GdddejZ"GdddeZ#GdddejZ$dS)z[ EfficientNet, MobileNetV3, etc Blocks Hacked together by / Copyright 2019, Ross Wightman )CallableDictOptionalTypeN) functional) create_conv2dDropPathmake_divisiblecreate_act_layer create_aa to_2tuple LayerType ConvNormActget_norm_act_layerMultiQueryAttention2d Attention2d) SqueezeExcite ConvBnActDepthwiseSeparableConvInvertedResidualCondConvResidual EdgeResidualUniversalInvertedResidualMobileAttention group_sizechannelscCs |sdS||dks J||S)Nr)rrrrT/home/ubuntu/.local/lib/python3.10/site-packages/timm/models/_efficientnet_blocks.py num_groupssrcsdeZdZdZddejejddfdedede ede de d e e d e e ffd d Z d dZ ZS)ra Squeeze-and-Excitation w/ specific features for EfficientNet/MobileNet family Args: in_chs (int): input channels to layer rd_ratio (float): ratio of squeeze reduction act_layer (nn.Module): activation layer of containing block gate_layer (Callable): attention gate function force_act_layer (nn.Module): override block's activation fn if this is set/bound rd_round_fn (Callable): specify a fn to calculate rounding of reduced chs g?Nin_chsrd_ratio rd_channels act_layer gate_layerforce_act_layer rd_round_fncsvtt||dur|pt}|||}|p|}tj||ddd|_t|dd|_tj||ddd|_ t||_ dS)NrT)biasinplace) superr__init__roundnnConv2d conv_reducer act1 conv_expandgate)selfr r!r"r#r$r%r& __class__rrr++s  zSqueezeExcite.__init__cCs:|jddd}||}||}||}|||S)N)T)keepdim)meanr/r0r1r2)r3xx_serrrforward?s    zSqueezeExcite.forward)__name__ __module__ __qualname____doc__r-ReLUSigmoidintfloatrr rr+r< __classcell__rrr4rrs0rcs~eZdZdZdddddejejddf ded ed ed ed ed edede de de de e de ffdd Z ddZddZZS)rz@ Conv + Norm Layer + Activation w/ optional skip connection rrFNr out_chs kernel_sizestridedilationrpad_typeskipr# norm_layeraa_layerdrop_path_ratec  stt|t| | } t||}|o|dko||k|_| duo#|dk}t||||r,dn||||d|_| |dd|_t | |||d|_ | rMt | |_dSt |_dS)NrrJrKgroupspaddingTr(rrJenable)r*rr+rrhas_skiprconvbn1r aarr-Identity drop_path)r3r rHrIrJrKrrLrMr#rNrOrPnorm_act_layerrRuse_aar4rrr+Js    zConvBnAct.__init__cCs*|dkr tdd|jjdStd|jjdS)N expansionrXr<module hook_typenum_chsrFr`rb)dictrW out_channelsr3locationrrr feature_infogzConvBnAct.feature_infocCs:|}||}||}||}|jr|||}|SN)rWrXrYrVr[r3r:shortcutrrrr<ms   zConvBnAct.forwardr=r>r?r@r-rA BatchNorm2drCstrboolr rrDr+rhr<rErrr4rrGsL     rc!seZdZdZdddddddddejejdddfd ed ed ed ed ededede dede dede de de e de e de f fdd ZddZddZZS)rz Depthwise-separable block Used for DS convs in MobileNet-V1 and in the place of IR blocks that have no expansion (factor of 1.0). This is an alternative to having a IR with an optional first pw conv. r7rrFFrNrGr rHdw_kernel_sizerJrKrrLnoskippw_kernel_sizepw_acts2dr#rNrOse_layerrPc sXtt|t| | }|dko||ko| |_| |_|duo"|dk}| dkrPt|d}t||dddd|_||||_ |dd}|dkrIdn|}|}d}nd|_d|_ |}t ||}t||||redn||||d|_ ||dd |_ t ||||d |_|r||| d nt|_t||| |d |_||d|jd |_|rt||_dSt|_dS)Nrr6samerIrJrSF)rJrKrSrRTr(rTr#rS)r) apply_act)r*rr+rrV has_pw_actrCrconv_s2dbn_s2drconv_dwrXr rYr-rZseconv_pwbn2rr[)r3r rHrqrJrKrrLrrrsrtrur#rNrOrvrPr\r]sd_chs dw_pad_typerRr4rrr+|s8       zDepthwiseSeparableConv.__init__cC*|dkr tdd|jjdStd|jjdSNr^r forward_prer_rFrcrdr in_channelsrerfrrrrhriz#DepthwiseSeparableConv.feature_infocCsv|}|jdur||}||}||}||}||}||}||}||}|jr9| ||}|Srj) r~rrrXrYrrrrVr[rkrrrr<s         zDepthwiseSeparableConv.forward)r=r>r?r@r-rArnrCrorpr r ModuleTyperDr+rhr<rErrr4rrwsf     7rc%seZdZdZddddddddddejejdddd fd ed ed ed edededede de dededede de de e de e de ede f$fdd ZddZd d!ZZS)"ra Inverted residual block w/ optional SE Originally used in MobileNet-V2 - https://arxiv.org/abs/1801.04381v4, this layer is often referred to as 'MBConv' for (Mobile inverted bottleneck conv) and is also used in * MNasNet - https://arxiv.org/abs/1807.11626 * EfficientNet - https://arxiv.org/abs/1905.11946 * MobileNet-V3 - https://arxiv.org/abs/1905.02244 r7rrFF?rNrGr rHrqrJrKrrLrr exp_ratioexp_kernel_sizersrur#rNrOrv conv_kwargsrPcstt|t|| }|pi}||ko|dko| |_|duo#|dk}| dkrQt|d}t||dddd|_||||_|dd}|dkrJdn|}|}d}nd|_d|_|}t || }t ||}t||| fd|i||_ ||dd |_ t|||f|rdn||||d ||_ ||dd |_t||||d |_|r||| d nt|_t||| fd|i||_||dd |_|rt||_dSt|_dS)Nrrwr6rxryFrSTr(rQrTrzr|)r*rr+rrVrCrr~rr rrrXrrr rYr-rZrconv_pwlbn3rr[)r3r rHrqrJrKrrLrrrrrsrur#rNrOrvrrPr\r]rrmid_chsrRr4rrr+sD        zInvertedResidual.__init__cCrNr^rrr_rFrcrdrrrerfrrrrhrizInvertedResidual.feature_infocCs|}|jdur||}||}||}||}||}||}||}||}||}| |}|j rC| ||}|Srj) r~rrrXrrrYrrrrVr[rkrrrr<s           zInvertedResidual.forwardr=r>r?r@r-rArnrCrorprDr rrrr+rhr<rErrr4rrsr      @rcs4eZdZd dededeffdd Zdd ZZS) LayerScale2dh㈵>Fdim init_valuesr)cs*t||_t|t||_dSrj)r*r+r)r- Parametertorchonesgamma)r3rrr)r4rrr+,s zLayerScale2d.__init__cCs*|jdddd}|jr||S||S)Nr)rviewr)mul_)r3r:rrrrr<1szLayerScale2d.forward)rF) r=r>r?rCrDrpr+r<rErrr4rr+src&seZdZdZdddddddddejejdddd d fd ed ed ededededededede de de de de e de e de ede de e f$fdd Zdd Zd!d"ZZS)#ra Universal Inverted Residual Block (aka Universal Inverted Bottleneck, UIB) For MobileNetV4 - https://arxiv.org/abs/, referenced from https://github.com/tensorflow/models/blob/d93c7e932de27522b2fa3b115f58d06d6f640537/official/vision/modeling/layers/nn_blocks.py#L778 rr7rrFFrNrGrr rHdw_kernel_size_startdw_kernel_size_middw_kernel_size_endrJrKrrLrrrr#rNrOrvrrPlayer_scale_init_valuec stt||p i}||ko|dko| |_|dkr#|s#|s#|s#J|rE|s)|nd}t||}t|||f|||| d| | |d||_nt|_t || }t||df| | | d||_ |rzt||}t|||f|||| | | |d||_ nt|_ |r||| dnt|_ t||df| d| | d||_ |r|s|s|nd}t||}|dkr|rJt|||f|||| d| | d||_nt|_|durt|||_nt|_|rt||_dSt|_dS) NrF)rJrKrRrSr|r#rNrO)rSr#rN)rJrKrRrSr#rNrOrz)rSr|r#rN)rJrKrRrSr|r#rN)r*rr+rVrrdw_startr-rZr pw_expdw_midrpw_projdw_endr layer_scalerr[)r3r rHrrrrJrKrrLrrrr#rNrOrvrrPrdw_start_stridedw_start_groupsrrR dw_end_stride dw_end_groupsr4rrr+=s          z"UniversalInvertedResidual.__init__cCs.|dkrtdd|jjjdStd|jjjdS)Nr^z pw_proj.convrr_rFrc)rdrrWrrerfrrrrhsz&UniversalInvertedResidual.feature_infocCsb|}||}||}||}||}||}||}||}|jr/|||}|Srj) rrrrrrrrVr[rkrrrr<s       z!UniversalInvertedResidual.forwardrrrr4rr6sr      prc1seZdZdZdddddddddddddejejd d d d d ddfd ed ededededededededede dededede de de de e de de d e d!e e d"e d#e f0fd$d% Z d&d'Zd(d)ZZS)*rz Mobile Attention Block For MobileNetV4 - https://arxiv.org/abs/, referenced from https://github.com/tensorflow/models/blob/d93c7e932de27522b2fa3b115f58d06d6f640537/official/vision/modeling/layers/nn_blocks.py#L1504 rr7rF@F)rrNrGrr rHrJrqrKrrL num_headskey_dim value_dimuse_multi_query query_strides kv_stridecpe_dw_kernel_sizerrr#rNrOrP attn_drop proj_dropruse_biasuse_cpecs&tt|t||}|dko||ko| |_t| |_| |_tdd|jD|_ |r9t ||||ddd|_ nd|_ ||dd|_ |durS|| dksOJ|| }| rht |||| | | | ||||||d |_n t||||||d |_|dur~t|||_nt|_|rt||_dSt|_dS) NrcSsg|]}|dkqS)rr).0srrr sz,MobileAttention.__init__..T)rIrK depthwiser'Frr) dim_outrrrrrrqrKrSrrrN)rrrrr')r*rr+rrVr rranyhas_query_strider conv_cpe_dwnormrattnrrrr-rZrr[)r3r rHrJrqrKrrLrrrrrrrrrr#rNrOrPrrrrrr\r4rrr+s^       zMobileAttention.__init__cCrrrrfrrrrh!rizMobileAttention.feature_infocCsV|jdur||}||}|}||}||}||}|jr)|||}|Srj)rrrrrVr[)r3r:x_cperlrrrr<'s     zMobileAttention.forwardrmrrr4rrs      Zrc#seZdZdZdddddddddejejdddd fd ed ed ed edededede de dedede de de e de e dede f"fdd ZddZZS)rz, Inverted residual block w/ CondConv routingr7rrFFrNrrGr rHrqrJrKrrLrrrrrsr#rNrOrv num_expertsrPcs\||_t|jd}tt|j||f||||||| | | | | ||||dt||j|_dS)N)r)rqrJrKrrLrrrrrsr#rNrOrvrrP)rrdr*rr+r-Linear routing_fn)r3r rHrqrJrKrrLrrrrrsr#rNrOrvrrPrr4rrr+9s.  zCondConvResidual.__init__cCs|}t|dd}t||}|||}||}|||}| |}| |}| ||}| |}|j rC|||}|S)Nr)Fadaptive_avg_pool2dflattenrsigmoidrrrXrrrrrrVr[)r3r:rl pooled_inputsrouting_weightsrrrr<es       zCondConvResidual.forward)r=r>r?r@r-rArnrCrorprDr rrr+r<rErrr4rr6sj     ,rc!seZdZdZdddddddddejejddd fd ed ed ed ededededede de dede de de e de e de f fdd ZddZddZZS) ra( Residual block with expansion convolution followed by pointwise-linear w/ stride Originally introduced in `EfficientNet-EdgeTPU: Creating Accelerator-Optimized Neural Networks with AutoML` - https://ai.googleblog.com/2019/08/efficientnet-edgetpu-creating.html This layer is also called FusedMBConv in the MobileDet, EfficientNet-X, and EfficientNet-V2 papers * MobileDet - https://arxiv.org/abs/2004.14525 * EfficientNet-X - https://arxiv.org/abs/2102.05610 * EfficientNet-V2 - https://arxiv.org/abs/2104.00298 r7rrrFFrNrGr rHrrJrKrrL force_in_chsrrrrsr#rNrOrvrPc stt|t| | }|dkrt|| }nt|| }t||}||ko,|dko,| |_|duo5|dk}t||||r>dn||||d|_||dd|_ t ||||d|_ |r^||| dnt |_t||| |d|_||d d |_|r|t||_dSt |_dS) NrrrQTr(rTrzr{Fr)r*rr+rr rrVrconv_exprXr rYr-rZrrrrr[)r3r rHrrJrKrrLrrrrrsr#rNrOrvrPr\rrRr]r4rrr+s&     zEdgeResidual.__init__cCrrrrfrrrrhrizEdgeResidual.feature_infocCsX|}||}||}||}||}||}||}|jr*|||}|Srj)rrXrYrrrrVr[rkrrrr<s      zEdgeResidual.forward)r=r>r?r@r-rArnrCrorprDr rrr+rhr<rErrr4rrusf     .r)%r@typingrrrrrtorch.nnr-rr timm.layersrrr r r r r rrrr__all__ModulerrCrrrrrrrrrrrrrrs&  4  (0Rb  u?