o @Tim@s*ddlZddlmZGdddejZdS)N)nnc sbeZdZdZ  ddedededeffdd Zd ej d ej d e ej ej fd BfddZ Z S) AdaLNZeroa Adaptive Layer Normalization Zero (AdaLNZero) module. Combines LayerNorm with adaptive conditioning to produce shift, scale, and gate values. The gate is used to scale features before residual connection. Args: dim: Feature dimension condition_dim: Conditioning dimension eps: LayerNorm epsilon return_gate: If True, returns gate value for scaling. h㈵>Tdim condition_dimeps return_gatecst||_||_||_tj||dd|_|rd|nd|}tt t |||_ tj |j djtj |j djdS)NF)relementwise_affine)super__init__rrrr LayerNormnorm SequentialSiLULinearcondition_projinitzeros_weightbias)selfrrrr output_dim __class__O/home/ubuntu/.local/lib/python3.10/site-packages/linacodec/module/adaln_zero.pyrs  zAdaLNZero.__init__x conditionreturnNc Cs^||}||}|jr|jddd\}}}n |jddd\}}d}|d||}||fS)a> Args: x: Input tensor of shape (B, L, dim) condition: Conditioning tensor of shape (B, L, condition_dim) or (B, 1, condition_dim) Returns: modulated_x: Normalized and modulated features gate: Gate values for scaling (None if return_gate=False) r )rr Nr )rrrchunk) rrr x_normcondition_paramsshiftscalegate modulated_xrrrforward0s zAdaLNZero.forward)rT) __name__ __module__ __qualname____doc__intfloatboolrtorchTensortupler* __classcell__rrrrrs2r)r2rModulerrrrrs