o ߥiu@sddlZddlZddlmZddlmZmZddlZddl Z ddl m m Z ddl m Z ddlmZGddde jZGd d d e jZGd d d e jZGd dde jZGddde jZeddddddedddddddZdddZedkreZdSdS)N) OrderedDict)TupleUnion)nn)ViMcs$eZdZdejffdd ZZS) LayerNormxcs$|j}t|tj}||SN)dtypesuperforwardtypetorchfloat32)selfr orig_typeret __class__c/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/models/cv/vision_middleware/backbone.pyr s zLayerNorm.forward)__name__ __module__ __qualname__rTensorr __classcell__rrrrrsrc@seZdZdejfddZdS) QuickGELUr cCs|td|S)NgZd;?)rsigmoidrr rrrr szQuickGELU.forwardN)rrrrrr rrrrrsrcsTeZdZ d dededejffdd Zdejfdd Zdejd efd d Z Z S)ResidualAttentionBlockNd_modeln_head attn_maskc stt|||_t||_ttdt ||dfdt fdt |d|fg|_ t||_ ||_ t|_t|_dS)Nc_fcgeluc_proj)r __init__rMultiheadAttentionattnrln_1 SequentialrLinearrmlpln_2r#rvim_attvim_mlp)rr!r"r#rrrr(!s    zResidualAttentionBlock.__init__r cCs>|jdur|jj|j|jdnd|_|j|||d|jddS)N)r deviceF) need_weightsr#r)r#tor r2r*rrrr attention3s  z ResidualAttentionBlock.attention task_namecCsT||}|||}||||}||}|||}||||}|Sr )r+r5r0r/r.r1)rr r6 x_normed_1 x_normed_2rrrr :s  zResidualAttentionBlock.forwardr ) rrrintrrr(r5strr rrrrrr sr c sDeZdZ d dedededejffdd Zdejfd d ZZS) TransformerNwidthlayersheadsr#cs<t|_||_tfddt|D|_dS)Ncsg|]}tqSr)r ).0_r#r>r<rr Ps z(Transformer.__init__..)r r(r<r=r ModuleListrange resblocks)rr<r=r>r#rrArr(Hs zTransformer.__init__r c KsD|\}}}g}t|jD]\}}||fi|}||q|Sr )size enumeraterEappend) rr kwargsLBDfeaturesiblkrrrr Us  zTransformer.forwardr ) rrrr9rrr(r rrrrrr;Fs r;c sNeZdZdZ ddedededededef fd d Zd ejfd d ZZ S)VisionTransformeraf The Vision Transformer (ViT) model Args: - input_resolution (int): shape of input image - patch_width (int): size of patch tokens - width (int): feature channels - layers (int): number of transformer layers - heads (int): number of multi-head attention - output_dim (int): output feature channels input_resolution patch_sizer<r=r> output_dimcst||_tjd|||dd|_|d}t|t||_ t|t||dd||_ t ||_ |||_ t||||_t ||_t|t|||_||_dS)NF) in_channels out_channels kernel_sizestridebiasgr)r r(rRrConv2dconv1 Parameterrrandnclass_embeddingpositional_embeddingrln_prepatch_per_sider; transformerln_postprojrT)rrRrSr<r=r>rTscalerrrr(js(      zVisionTransformer.__init__r c KsN||}|d}|d}||jd|jdd}|ddd}|j|jddd|dd}t j ||gdd}||j |j}| |}|ddd}|j |fi|}|d}|ddd}||dddddf}|jdur~||j}g}|D]}||ddddddfddd|d||q|||S)Nrr[r)dim)r]rFreshapeshapepermuter`r4r repeatrcatrarbrdrerfrH) rr rIrKP cls_token x_per_layeroutputsoutputrrrr s:       "  zVisionTransformer.forward)rQ) rrr__doc__r9r(rrr rrrrrrP^s"rPi )rRrSr<r=r> ) vit_b16_224 vit_b32_224rycCs$t|}tdi|}|||S)z build a ViT + ViM model Args: arch: name of backbone pretrained: weights of pretrained model Nr) model_dictrPload_state_dict)arch pretrained model_argsmodelrrrbuild_backbones r__main__)ryN)mathos collectionsrtypingrrnumpynprtorch.nn.functionalr functionalFvimrrModulerr r;rPdictr{rrrrrrrs*   'L