o i1@s\ddlZddlmZmZddlZddlmZdejdejde fdd Z d eeej d eeej deee ee ee ffd d Z deeej dee dee ddfddZd eeej d eeej deeej ddfddZdeeej de de deej fddZdeej de de deeej fddZdeej de de deej fddZejjdd d!d"d#eej d$eej dee dee d%ee deej f d&d'Zejdd#eej d$eej dee dee d%ee deej f d(d)Zd*d+Zd,d-Zejjdeed.d eeej d eeej deeej fd/d0ZdS)1N)ListTuple)_is_triton_availabledevicedtypereturncCs:ttdds dStsdStj|}|dkrdSdS)N#XFORMERS_TILED_MATMUL_ENABLE_TRITON1F)rT)intosgetenvrtorchcudaget_device_capability)rrdevice_capabilityrM/home/ubuntu/.local/lib/python3.10/site-packages/xformers/ops/tiled_matmul.py_should_use_tritons rabc stdkrtddkrtfddDsJdtdkr6tddkr6tfddDs:Jdt}td}t|ksVJd|d ttd}fd d t|D}fd d t|D}fd d t|D}fdd t|D}t|D]U} t|D]N} | | jd|| ksJd| d|| d| | jdd| | | jd|| ksJd| d|| d| | jdd| qqt|D]W} t|D]P} | | jd|| ksJd| d|| d| | jdd| | | jd|| ks$"zcheck_inputs..zUthe first operand must be a non-empty two-dimensional regular list of lists of tenorsc3rrrrrrrr!(r"zVthe second operand must be a non-empty two-dimensional regular list of lists of tenorszYthe first operand's inner dimension must match the second operand's outer dimension, got  and cg|] }|djdqSrshapertile_mr rr 5z check_inputs..cg|] }d|jdqSrrr'rtile_nr#rrr+6r,cr-r.r'rtile_kr rrr+7r,cr%r&r'r1r#rrr+8r,the tensors on row zM of the first operand must all have the same size along the m dimension, got  at position 0 and at position the tensors on column zM of the first operand must all have the same size along the k dimension, got zN of the second operand must all have the same size along the k dimension, got zN of the second operand must all have the same size along the n dimension, got z' of the first operand and those on row zJ of the second operand must have the same size along the k dimension, got rallranger() rrm_tilesk_tilesn_tilesmsnsaksbksr*r2r0ksr)rrr check_inputs s44       rBoutr=r>c st|t|}}tdkr"tddkr"tfddDs&Jdt|ks.Jtd|ks8Jfddt|D}fddt|D}t|D]U}t|D]N}||jd||ksJd |d ||d ||jdd |||jd||ksJd |d||d ||jdd |qXqRt|D]}||||ksJd |d|d||d||qt|D]}||||ksJd |d|d||d||qdS)Nrrc3rrrrrCrrr!dr"zcheck_output..zGout must be a non-empty two-dimensional regular list of lists of tenorscr%r&r'r)rDrrr+hr,z check_output..cr-r.r'r/rDrrr+ir,r3z? of out must all have the same size along the m dimension, got r4r5r6z? of out must all have the same size along the k dimension, got z of out and those on row zI of the first operand must have the same size along the m dimension, got r$z of out and those on column zJ of the second operand must have the same size along the n dimension, got r7) rCr=r>r:r<cmscnsr*r0rrDr check_output_sb     rGc Cst||\}}}t|||t|dkr@t|dkr@t|dkr@t|ddj|ddjr@ddlm}|||||||dStt|D];}tt|D]2}t j ||d|d||||dtdt|D]} ||| ||| || |qkqNqFdS)Nrr)_launch_triton_matmulrD) rBrGrrrr_triton.tiled_matmul_kernelsrIr9rmmaddmm_) rrrCr=r>rArIr*r0r2rrrtiled_matmul_outs"     (&rMxrowscolscsPt||ksJtfdd|DsJdd|D}t||ks&J|S)Nc3|] }t|kVqdSNrrrPrrr!z_flatten..cSsg|] }|D]}|qqSrr)rrelemrrrr+z_flatten..)rr8)rNrOrPflat_xrrSr_flattens rXrWcsbt|ks Jfddtd|D}t||ks"Jtfdd|Ds/J|S)Ncsg|] }||qSrr)r row_offsetrPrWrrr+sz_unflatten..rc3rQrRrrrSrrr!rTz_unflatten..)rr9r8)rWrOrPrNrrZr _unflattens r[cCs.t|||}ddt|D}t|||}|S)NcSsg|] }dd|DqS)cSsg|]}|qSr)t)rrUrrrr+sz3_flattened_transpose...r)rcolrrrr+rVz(_flattened_transpose..)r[ziprX)rWrOrPrN transposed_xflat_transposed_xrrr_flattened_transposes  raz!xformers_python::tiled_matmul_fwdrr) mutates_args device_typesflat_aflat_brAcs^t|t|t|t|t|t}fdd|D}t||dt|t|tS)Nc g|] fddDqS)c"g|] }dd|fqSr& new_emptyrn)rmrrr+"z/tiled_matmul_fwd...rrrr>rlrr+ z$tiled_matmul_fwd..rD)r[rrMrX)rdrer=r>rArcrrortiled_matmul_fwds  rscs(fdd|D}t|t|t|S)Ncrf)crgr&rhrj)rdrlrrr+rmz4tiled_matmul_fwd_fake...rrnrdr>rprr+rqz)tiled_matmul_fwd_fake..)rXr)rdrer=r>rArrrrtrtiled_matmul_fwd_fakesrucCs,|\}}|_|_|_|jg||RdSrR)r=r>rAsave_for_backward)ctxinputsoutputrdrerrrtiled_matmul_setup_contextsrzcCst|jt|jt|jt|jt|jksJ|jdt|jt|j}|jt|j t|jd}t|t|jt|j}t|t|jt|j}t|||j|j|j}t|||j|j|j}||dddfSrR)r saved_tensorsr=rAr>rars)rw flat_grad_crdreflat_transposed_aflat_transposed_b flat_grad_a flat_grad_brrrtiled_matmul_bwds$ r) setup_contextc Cs`t||\}}}t|t|t|}t|t|t|}t|||||}t|t|t|}|S)axMultiply two matrices given as grids of tiles It performs the matmul between A and B, which are given as two-dimensional grids of tiles (i.e., blocks), represented as lists of lists of tensors. The output will itself be a matrix in such a form. Formally: out[m][n] = sum(a[m][k] @ b[k][n] for k in range(...)) with the obvious constraints needed to make it work, in terms of number of tiles and sizes of each tile. The interest of this operator is to improve performance by avoding wave quantization effects when doing independent matrix multiplications in series. Sometimes, when these matmuls have one operand in common, this can also be addressed by concatenating the other operands into a single matrix, and issuing a single matmul. However this isn't always possible (e.g., might break the checkpoint format) and it's an anti-pattern, as it obscures the logic (e.g., changing the modelling code out of performance reasons). This tiled matmul performs the same computation as if the matrices were merged, without merging them, simply through a smarter memory addressing scheme. The tiled matmul is less generic than a grouped matmul, which can also help with wave quantization, and doesn't need the matmuls to have the same lhs or rhs operand. However, a grouped matmul will write the result of each matmul to a separate output matrix, whereas the tiled matmul allows to add them together into a single output. This is needed during the backward pass of a linear layer, and it's the reason we wrote this instead of using a grouped matmul. The tiled matmul is implemented using a custom Triton kernel, which puts constraints on the strides of the tiles. All rows of A must have the same K stride, all columns of A must have the same M stride, and so on. Currently the tiled matmul supports at most three tiles on each dimension, although fewer can also be given. This is because we needed it to fuse the query, key and value weights of an attention layer. This limit can be increased if needed. This operator is differentiable. )rBrXrrsr[) rrr=r>rArdreflat_crrrrr tiled_matmul s /r)r typingrrrrrrboolrTensorr rBrGrMrXr[ralibrary custom_oprs register_fakerurzrregister_autogradrrrrrs    ,?*    *