o i@sddlmZddlZdejdedejfddZdejded ed edejf d d Z dd eejdeejded ed edeedeejfddZddejfddZej fdejdeej dejfddZ dS))OptionalNkey_padding_mask batched_dimreturncCs0|jdksJ|\}}||}t||||S)N)ndimsize_reshape_key_padding_mask)rr batch_sizesrc_len num_headsr W/home/ubuntu/.local/lib/python3.10/site-packages/xformers/components/attention/utils.pyreshape_key_padding_mask s rr r r cCs@|j||fks J||dd|d|dd||d|}|S)N)shapeviewexpandreshape)rr r r r r rr s  r att_masktgt_lencCs~|dur|}|dur=|j||fksJt||||}|dur'|d|d}|S|jtjkr4||}|S||td}|S)Nr-inf) rr rdtypetorchbool logical_and masked_fillfloat)rrr r r rr r rmaybe_merge_masks%s rF softmax_matcCstj|d|j|jd}|}|r$dttj|dd|dd}ndtjtj|ddddjddddf|dd}t |D]$}t ||}t d|d|t |d |t |d ||}qD|S) z Computing the Moore-Penrose inverse. Use an iterative method from (Razavi et al. 2014) to approximate the Moore-Penrose inverse via efficient matrix-matrix multiplications. r)devicerr)dimNg? ) reyerr"rmaxsum transposevaluesrangematmul)r!n_iterpinverse_original_initikv_kvr r riterative_pinvAs&&&   *r6maskrcCs4|jtjks Jdtj||d}td||<|S)NzEThis util is meant to convert in between bool masks and additive ones)rr)rrr zeros_liker)r7rmask_r r rbool_mask_to_additivecs r:)N)r F) typingrrTensorintrr rr6float32rr:r r r rsZ     #