o ϯi>@s\ddlZddlmZddlZddlmZddlmmZddl m Z Gdddej Z dS)N)Optional)set_attributescseZdZdZ      ddejd eejd ed ed ed ededdffdd Z d ed eddfddZ de j de j de j fddZ ZS) MemoryBanka, Performs Non-Parametric Instance Discrimination for self supervised learning on video. A memory bank is built to keep and update the historical feature embedding and use them for contrastive learning. The original paper is: Unsupervised Feature Learning via Non-Parametric Instance Discrimination https://arxiv.org/pdf/1805.01978.pdf More details can be found from the memory bank part in the following paper: Momentum Contrast for Unsupervised Visual Representation Learning https://arxiv.org/pdf/1911.05722.pdf NQ?+?backbonemlpneg_size temperature bank_sizedimmmtreturncs&tt|t|||dS)a8 Args: backbone (nn.Module): backbone used to forward the input. mlp (nn.Module): multi-layer perception used in memory bank instance discrimination model. neg_size (int): size of negative samples per instance. temperature (float): temperature to use for contrastive learning. bank_size (int): size of the memory bank, expected to be the same size as the training set. dim (int): dimension of the channel. mmt (float): momentum to use. N)super__init__rlocals_init_mem_bank)selfr r r r rrr __class__S/home/ubuntu/.local/lib/python3.10/site-packages/pytorchvideo/models/memory_bank.pyrs  zMemoryBank.__init__cCsNdt|d}|dt||d|| t|j j dS)a' Given the memory bank size and the channel dimension, initialize the memory bank. Args: bank_size (int): size of the memory bank, expected to be the same size as the training set. dim (int): dimension of the channel. g?memoryN) mathsqrtregister_buffertorchrandmul_add_tonextr parametersdevice)rrrstdvrrrr5s  zMemoryBank._init_mem_bankxx_indc Cs|jd}||}|jdur||}tj|ddd}tjd|j||jdfd |j }| dd |j t|jd|d}|||jd|j}td||}t||j}tj|f|j tjd }tjj||}|jrt@t|jd|d} | |j| t |d|j| !dj"dd d !d } | | } |j#d|| Wd|S1swY|S) a Perform contrastive learning with random sampled negative instance from the memory bank. During training, update the memory bank with latest feature embedding. Args: x (torch.tensor): a batch of image with augmentation. The input tensor shape should able to be feed into the backbone. x_ind (torch.tensor): the index of the image x from the dataset. Expected shape is B. rNr)pr)sizez bkc,bc->bk)r(dtypeT)keepdimg?)$shaper r F normalizer!randintrr r%r(selectcopy_data index_selectrviewdetachreinsumdivr zeroslongnn functional cross_entropytrainingno_gradr#rr$mulpowsum index_copy_) rr*r+ batch_sizeidxweightoutgtlossposnormupdatedrrrforwardJs6       zMemoryBank.forward)Nrrrrr )__name__ __module__ __qualname____doc__r@Modulerintfloatrrr!TensorrR __classcell__rrrrr s6 $r) rtypingrr!torch.nnr@torch.nn.functionalrAr3pytorchvideo.layers.utilsrrWrrrrrs