o ߥi4@s ddlZddlmZddlZddlmZddlmmZ ddl m Z ddl m Z ddlmZddlmZddlmZmZddlmZdd lmZmZejeje jd Gd d d e ZGd dde ZGddde ZGddde Z Gddde Z!Gddde Z"dS)N)Models) TorchModel)MODELS)Config) ModelFileTasks) load_clip)get_state_dictset_seed) module_namecsBeZdZdZdeffdd Zd ddZdd Zd d d ZZ S) VoPa The implementation of 'VoP: Text-Video Co-operative Prompt Tuning for Cross-Modal Retrieval'. This model is dynamically initialized with the following parts: - clip: the upstream pre-trained backbone model (CLIP in this code) - pool_frames: the frames pooling method - visual_prompt_learner: visual prompt - ImageEncoder: get image encoder - TextPromptLearner: text prompt - TextEncoder: get text encoder model_dircstt|t|d}t|d}t|tj}t|j |_ t |d|_ t t|j jjj|j _t t|j jj|j _t|j j|j |_t|j |j |_t|j |j |_t|j |j |_t|j |j |_|t ||!dt"j#d<t$|j j%dS)zl Initialize a VoP Model Args: model_dir: model id or path, zVoP_msrvtt9k.pthz ViT-B-32.pt)namefalseTOKENIZERS_PARALLELISMN)&superr __init__ospjoinr CONFIGURATIONr from_file hyperparamconfigr cliplistrangevisual transformerlayers vpt_layers tpt_layersBaselinePooling pooling_type pool_framesVisualPromptLearnervisual_prompt_learner ImageEncoder image_encoderTextPromptLearnertext_prompt_learner TextEncoder text_encoderload_state_dictr evalosenvironr seed)selfrargskwargs model_path clip_arch config_path __class__\/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/models/cv/vop_retrieval/model.pyr!s.    z VoP.__init__FcCsv|jd}|dd|jj|jj}|}|||}||jddd}|||jjd}|d|}|r9||fS|S)z Get video Features Args: videos: the dim is [1, 12, 3, 224, 224] return_all_frames: default False rTdimkeepdimN) shapereshaper input_resr&r(norm num_framesr$)r2videosreturn_all_frames batch_size video_datavisual_promptsvideo_featuresvideo_features_pooledr:r:r;get_video_featuresFs   zVoP.get_video_featurescCs*|}|||}||jddd}|S)zh Get Text Features Args: text_data: the dim is [1, 69] r<Tr>)r*r,rD)r2 text_data text_prompts text_featuresr:r:r;get_text_featuresas  zVoP.get_text_featuresc Cs|djd}|d}|d}|dd|jj|jj}|}|||}|}|||} | | jddd} ||jddd}|||jj d}| | |} |rW| || fS| | fS)z Dynamic Forward Function of VoP Args: data: the input data return_all_frames: default False videortextr<r=Tr>) rArBrrCr&r(r*r,rDrEr$) r2datarGrHrNrIrJrKrOrPrLr:r:r;forwardos.    z VoP.forward)F) __name__ __module__ __qualname____doc__strrrMrQrU __classcell__r:r:r8r;r s  %r cs0eZdZdZfddZddZddZZS)r"z( Redefined Pooling Function cs&tt||dkr|j|_dSt)Navg)rr"r _avg_pooling pooling_funcNotImplementedError)r2r#rr8r:r;rs zBaselinePooling.__init__cCs|jdd}|S)a$ Pooling mean of frames Args: text_embeds: the input text embedding which is None here. video_embeds: the input video embedding with [1, 12, 512]. Returns: video_embeds_pooled: num_vids x embed_dim rr?)mean)r2 text_embeds video_embedsvideo_embeds_pooledr:r:r;r]s zBaselinePooling._avg_poolingcCs |||SN)r^)r2rbrcr:r:r;rUs zBaselinePooling.forward)rVrWrXrYrr]rUr[r:r:r8r;r"s  r"c(eZdZdZfddZddZZS)r%a The implementation of visual prompt. This module is used to define the learnable prompt parameters: the number of tokens is 8, the prompt dimension is 768, and the initialization weight std used is 0.02. csdtt||j}|jjjjd}|j}t j t |j d|||d}t jj|ddt ||_dS)Nrrdtype{Gz?std)rr%r vp_token_numrln_postweightrArhtorchemptylenr nninitnormal_ ParameterrJ)r2 clip_modelrrlvp_dimrhrJr8r:r;rszVisualPromptLearner.__init__cCs |j}|Sre)rJ)r2vpr:r:r;rUszVisualPromptLearner.forwardrVrWrXrYrrUr[r:r:r8r;r%s  r%crf)r)a The implementation of visual prompt. This module is used to define the learnable prompt parameters: the number of tokens is 4, the prompt dimension is 512, and the initialization weight std used is 0.02. cstt||j}|j}|dkr|dksJ|jjjd}|j}t j t |j ||||d}t jj|ddt ||_||_||_dS)Nrrgrirj)rr)rtp_prefix_token_numtp_suffix_token_numln_finalrnrArhrorprqr!rrrsrtrurO)r2rvrrzr{tp_dimrhrOr8r:r;rs   zTextPromptLearner.__init__cCs<|jddd|jddf|jdd|jdddffSre)rOrz)r2r:r:r;rUszTextPromptLearner.forwardryr:r:r8r;r)s r)crf)r'z The implementation of image encoder. This module is used to obtain the features of each frame of the video. csvtt|||_|j|_|j|_|j|_|jj|_|jj |_ |jj |_ |jj |_ |jj |_ |jj |_ |jj|_dSre)rr'rrr rlrErconv1class_embeddingpositional_embeddingln_prerrmprojr2rvrr8r:r;rs      zImageEncoder.__init__c Cs|jd}||}|||jdd}|ddd}|j|j}tj|d|jd|j|j d}||}tj ||gdd}||j |j}t |j jD]}||jvr|j|}||ddddddf|dd}tj |ddddddf||ddddddfgdd}|dkr||}|ddd}|j j||}|ddd}|d|jvrtj |ddddddf|ddd|jdddfgdd}qM||dddddf}|jdur||j}|S)a  The forward function of image encoder. Args: visual_prompts: the visual prompt, dim is [12, 1, 8, 768] x: the input data, dim is [12, 3, 224, 224] Returns: x: the output data, dim is [12, 512] rrr<)rhdevicer`N)rAr~rBpermutertorhrozerosrcatrrrrr indexrepeatr resblocksrlrmr) r2rJxrHx_1x_2i_layeri_prompt cur_layer_vpr:r:r;rUs@   @ <  zImageEncoder.forwardryr:r:r8r;r's r'crf)r+z The implementation of text encoder. This module is used to obtain the features of each word of the sentence. csvtt||j|_|j|_|j|_|j|_|j|_|j|_|j |_ d|j vs*J|j |_ |j |_ |j |j |_ dS)Nr) rr+rrtoken_embeddingrr|text_projectionrhr!rzr{ tp_token_numrr8r:r;r9szTextEncoder.__init__c Cs8|||j}|jd}|\}}t|jjD]}||jvr|j|}|j dkr^|||dddddf |dd} t j |ddddddf| |ddddddfgdd}|j dkr|||dddddf |dd} t j |ddddddf| |ddddddfgdd}|dkr||j|j}|ddd}|jj||}|ddd}|d|jvr|ddddddf} |ddd|j d|j ddf} |ddddddf} t j | | | gdd}|}q|||j}|t |jd|jdd|jf|j}|S)a The forward function of text encoder. Args: text_prompts: the text prompt, dim is 2 x [12, 4, 512] text: the input data, dim is [1, 69] Returns: x: the output data, dim is [1, 512] rrNr<r`r)rtyperhrArrrr!rrzexpandrorr{rrrr|arangeargmaxrr)r2rOrSrrH prompt_prefix prompt_suffixrrcur_layer_tp_prefixcur_layer_tp_suffixtemp_1temp_2temp_3tempr:r:r;rUHsp     6  4zTextEncoder.forwardryr:r:r8r;r+3s r+)#r/os.pathpathrrotorch.nnrrtorch.nn.functional functionalFmodelscope.metainfor'modelscope.models.base.base_torch_modelrmodelscope.models.builderrmodelscope.utils.configrmodelscope.utils.constantrrbackboner basic_utilsr r register_module vop_retrievalvop_retrieval_modelr r"r%r)r'r+r:r:r:r;s*       }"G