o wiu+@sddlmZmZmZmZddlmZddlmZm Z ddl Z ddl Z ddlm Z de defd d ZGd d d ZGd ddeZdS))AnyListOptionalDict) EinopsError)ParsedExpression _ellipsisN)_productaxesreport_messagecCst|dkr t||dS)Nr)lenrformat)r r rR/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/einops/layers/_einmix.py _report_axes s rc seZdZddededeedeffdd Zdededeedefdd Zd eed ee d eed ee fddZ ddZ ddZ Z S) _EinmixMixinNpattern weight_shape bias_shape axes_lengthsc s8t||_||_||_||_|j||||ddS)a EinMix - Einstein summation with automated tensor management and axis packing/unpacking. EinMix is a combination of einops and MLP, see tutorial: https://github.com/arogozhnikov/einops/blob/main/docs/3-einmix-layer.ipynb Imagine taking einsum with two arguments, one of each input, and one - tensor with weights >>> einsum('time batch channel_in, channel_in channel_out -> time batch channel_out', input, weight) This layer manages weights for you, syntax highlights a special role of weight matrix >>> EinMix('time batch channel_in -> time batch channel_out', weight_shape='channel_in channel_out') But otherwise it is the same einsum under the hood. Plus einops-rearrange. Simple linear layer with a bias term (you have one like that in your framework) >>> EinMix('t b cin -> t b cout', weight_shape='cin cout', bias_shape='cout', cin=10, cout=20) There is no restriction to mix the last axis. Let's mix along height >>> EinMix('h w c-> hout w c', weight_shape='h hout', bias_shape='hout', h=32, hout=32) Example of channel-wise multiplication (like one used in normalizations) >>> EinMix('t b c -> t b c', weight_shape='c', c=128) Multi-head linear layer (each head is own linear layer): >>> EinMix('t b (head cin) -> t b (head cout)', weight_shape='head cin cout', ...) ... and yes, you need to specify all dimensions of weight shape/bias shape in parameters. Use cases: - when channel dimension is not last, use EinMix, not transposition - patch/segment embeddings - when need only within-group connections to reduce number of weights and computations - next-gen MLPs (follow tutorial link above to learn more!) - in general, any time you want to combine linear layer and einops.rearrange Uniform He initialization is applied to weight tensor. This accounts for the number of elements mixed and produced. Parameters :param pattern: transformation pattern, left side - dimensions of input, right side - dimensions of output :param weight_shape: axes of weight. A tensor of this shape is created, stored, and optimized in a layer If bias_shape is not specified, bias is not created. :param bias_shape: axes of bias added to output. Weights of this shape are created and stored. If `None` (the default), no bias is added. :param axes_lengths: dimensions of weight tensor )rrrrN)super__init__rrrrinitialize_einmix)selfrrrr __class__rrrs * z_EinmixMixin.__init__cs|d\}}t|}t|t|}ttjh|j|jd|jr*td|js0jrG|jr6js=td||jrGtd|t dd||fDrWtdd |vs_d |vrftd |d} d} d} t d d|j Drg|j D]} | 7q{d dDd } |d| } fdd D} t ddj Dsjrgj D]} | 7qddDd } | d|} | | | | i|jD] }|vrtd|qttth|j|jdtt|jh|jjdt|jdkrtdfdd|j D}tfdd|j D}|durt|ts9tdt|}tt|jjdtt|jtdg}d }j D];}|tkrk|ritd!q[|D]'}|tkr}|r{td!qm||jvr||d"}qm|d#qmq[nd}d$|d%}d#|d%}|||||h|jj|j}t|vr|ttt|}d&dttj|Dd't<d(tffd)d* }d+||||||_dS),Nz->z7Unrecognized identifiers on the right side of EinMix {}zKEllipsis is not supported in weight, as its shape should be fully specifiedz,Ellipsis in EinMix should be on both sides, z3Ellipsis on left side can't be in parenthesis, got css|]}|jVqdSN)has_non_unitary_anonymous_axes).0xrrr Ssz1_EinmixMixin.initialize_einmix..z2Anonymous axes (numbers) are not allowed in EinMix()z,Parenthesis is not allowed in weight shape: cs|] }t|dkVqdSNr rgrouprrrr![cSg|] }|tkr |ndqS...rrnamerrr _z2_EinmixMixin.initialize_einmix.. z-> csi|] \}}|vr||qSrr)rr0length)namesrr bsz2_EinmixMixin.initialize_einmix..csr$r%r'r(rrrr!dr*cSr+r,r.r/rrrr1hr2z ->z*Dimension {} of weight should be specifiedzAxes {} are not used in patternzWeight axes {} are redundantrzCEinMix: weight has no dimensions (means multiplication by a number)csg|]\}|qSrrraxis)rrrr1{cs g|] \}|jvr|qSr) identifiersr7)rrightrrr1}s zAbias shape should be string specifying which axes bias depends onz"Bias axes {} not present in outputz#Sizes not provided for bias axes {}Fz:all bias dimensions should go after ellipsis in the outputTr&g?cSsi|]\}}||qSrr)rletterkrrrr6r9r-r csTg}|jD]}t|tr|fdd|Dq|tksJ|dqd|S)Ncsg|]}|qSrrr7mapping2lettersrrr1szO_EinmixMixin.initialize_einmix..write_flat_remapped..r-) composition isinstancelistextendrappendjoin)r result composed_axisr?rrwrite_flat_remappeds     z;_EinmixMixin.initialize_einmix..write_flat_remappedz {},{}->{})splitrrset differencer: has_ellipsisrhas_ellipsis_parenthesizedanyrBrGitems_create_rearrange_layersrr warningswarnr rCstrrrF_create_parametersremoverDsortedzipstringascii_lowercaseeinsum_pattern)rrrrr left_pattern right_patternleftweightpre_reshape_patternpre_reshape_lengthspost_reshape_patternr)rBr8 _weight_shape_fan_inbias _bias_shapeused_non_trivial_sizer weight_bound bias_boundmapped_identifiersrJr)rr@r5r;rrCs                       z_EinmixMixin.initialize_einmixrarbrcpost_reshape_lengthscCtd)N.Should be defined in framework implementationsNotImplementedErrorrrarbrcrlrrrrRsz%_EinmixMixin._create_rearrange_layerscCrm)zShape and implementationsrnrorrrirrjrrrrVsz_EinmixMixin._create_parameterscCslt|j}|d|jd7}|jdur|d|jd7}|jD] \}}|d||7}q!d|jj|S)Nz, ''z, {}={}z{}({})) reprrrrrrQrr__name__)rparamsr8r4rrr__repr__s  z_EinmixMixin.__repr__r)ru __module__ __qualname__rUrrrdictrrrRrVrw __classcell__rrrrrs$3z  rc @sBeZdZdZdeedeedeedeefddZdd Zd S) _EinmixDebuggerzUsed only to test mixinrarbrcrlcCs||_||_||_||_dSr)rarbrcrlrqrrrrRs z(_EinmixDebugger._create_rearrange_layerscCs||_||_dSr)saved_weight_shapesaved_bias_shaperrrrrrVs z"_EinmixDebugger._create_parametersN) rurxry__doc__rrUrrRrVrrrrr|s r|)typingrrrreinopsreinops.parsingrrrSrZr rLrUrrr|rrrrs  F