o iC+@sddlZddlZddlmZmZmZmZddlmZddl m Z ddl m Z m Z dedefdd ZGd d d ZGd d d eZdS)N)AnyDictListOptional) EinopsError)_product)ParsedExpression _ellipsisaxesreport_messagecCst|dkr t||dS)Nr)lenrformat)r r rR/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/einops/layers/_einmix.py _report_axes s rc seZdZddededeedeffdd Zdededeedefdd Zd eed ee d eed ee fddZ ddZ ddZ Z S) _EinmixMixinNpattern weight_shape bias_shape axes_lengthsc s8t||_||_||_||_|j||||ddS)a EinMix - Einstein summation with automated tensor management and axis packing/unpacking. EinMix is a combination of einops and MLP, see tutorial: https://github.com/arogozhnikov/einops/blob/main/docs/3-einmix-layer.ipynb Imagine taking einsum with two arguments, one of each input, and one - tensor with weights >>> einsum('time batch channel_in, channel_in channel_out -> time batch channel_out', input, weight) This layer manages weights for you, syntax highlights a special role of weight matrix >>> EinMix('time batch channel_in -> time batch channel_out', weight_shape='channel_in channel_out') But otherwise it is the same einsum under the hood. Plus einops-rearrange. Simple linear layer with a bias term (you have one like that in your framework) >>> EinMix('t b cin -> t b cout', weight_shape='cin cout', bias_shape='cout', cin=10, cout=20) There is no restriction to mix the last axis. Let's mix along height >>> EinMix('h w c-> hout w c', weight_shape='h hout', bias_shape='hout', h=32, hout=32) Example of channel-wise multiplication (like one used in normalizations) >>> EinMix('t b c -> t b c', weight_shape='c', c=128) Multi-head linear layer (each head is own linear layer): >>> EinMix('t b (head cin) -> t b (head cout)', weight_shape='head cin cout', ...) ... and yes, you need to specify all dimensions of weight shape/bias shape in parameters. Use cases: - when channel dimension is not last, use EinMix, not transposition - patch/segment embeddings - when need only within-group connections to reduce number of weights and computations - next-gen MLPs (follow tutorial link above to learn more!) - in general, any time you want to combine linear layer and einops.rearrange Uniform He initialization is applied to weight tensor. This accounts for the number of elements mixed and produced. Parameters :param pattern: transformation pattern, left side - dimensions of input, right side - dimensions of output :param weight_shape: axes of weight. A tensor of this shape is created, stored, and optimized in a layer If bias_shape is not specified, bias is not created. :param bias_shape: axes of bias added to output. Weights of this shape are created and stored. If `None` (the default), no bias is added. :param axes_lengths: dimensions of weight tensor )rrrrN)super__init__rrrrinitialize_einmix)selfrrrr __class__rrrs * z_EinmixMixin.__init__cs|d\}}t|}t|t|}ttjh|j|jd|jr*td|js0jrG|jr6js=td||jrGtd|t dd||fDrWtdd |vs_d |vrftd |d} d} d} t d d|j Drg|j D]} | 7q{d dDd } |d| } fdd D} t ddj Dsjrgj D]} | 7qddDd } | d|} | | | | i|jD]}|vrtd|dqttth|j|jdtt|jh|jjdt|jdkrtjdddfdd|j D}tfdd|j D}|durt|tsz7Unrecognized identifiers on the right side of EinMix {}zKEllipsis is not supported in weight, as its shape should be fully specifiedz,Ellipsis in EinMix should be on both sides, z3Ellipsis on left side can't be in parenthesis, got css|]}|jVqdSN)has_non_unitary_anonymous_axes).0xrrr Ssz1_EinmixMixin.initialize_einmix..z2Anonymous axes (numbers) are not allowed in EinMix()z,Parenthesis is not allowed in weight shape: cs|] }t|dkVqdSNr rgrouprrrr [cSg|] }|tkr |ndqS...r rnamerrr _z2_EinmixMixin.initialize_einmix.. z-> csi|] \}}|vr||qSrr)rr/length)namesrr bsz2_EinmixMixin.initialize_einmix..csr#r$r&r'rrrr dr)cSr*r+r-r.rrrr0hr1z ->z Dimension z of weight should be specifiedzAxes {} are not used in patternzWeight axes {} are redundantrzCEinMix: weight has no dimensions (means multiplication by a number)) stacklevelcsg|]\}|qSrrraxis)rrrr0{cs g|] \}|jvr|qSr) identifiersr8)rrightrrr0}s zAbias shape should be string specifying which axes bias depends onz"Bias axes {} not present in outputz#Sizes not provided for bias axes {}Fz:all bias dimensions should go after ellipsis in the outputTr%g?cSsi|]\}}||qSrr)rletterkrrrr5r:r,r csTg}|jD]}t|tr|fdd|Dq|tksJ|dqd|S)Ncsg|]}|qSrrr8mapping2lettersrrr0szO_EinmixMixin.initialize_einmix..write_flat_remapped..r,) composition isinstancelistextendr appendjoin)r result composed_axisr@rrwrite_flat_remappeds     z;_EinmixMixin.initialize_einmix..write_flat_remapped,)splitrrset differencer; has_ellipsisrhas_ellipsis_parenthesizedanyrCrHitems_create_rearrange_layersr warningswarnrrDstrr rG_create_parametersremovesortedzipstringascii_lowercaseeinsum_pattern)rrrrr left_pattern right_patternleftweightpre_reshape_patternpre_reshape_lengthspost_reshape_patternr(rCr9 _weight_shape_fan_inbias _bias_shapeused_non_trivial_sizer weight_bound bias_boundmapped_identifiersrKr)rrAr4r<rrCs                    z_EinmixMixin.initialize_einmixrcrdrepost_reshape_lengthscCtd)N.Should be defined in framework implementationsNotImplementedErrorrrcrdrernrrrrTsz%_EinmixMixin._create_rearrange_layerscCro)zShape and implementationsrprqrrrkrrlrrrrXsz_EinmixMixin._create_parameterscCstt|j}|d|jd7}|jdur|d|jd7}|jD]\}}|d|d|7}q!|jjd|dS)Nz, ''z, =r!r")reprrrrrrSr__name__)rparamsr9r3rrr__repr__s  z_EinmixMixin.__repr__r)rx __module__ __qualname__rWrrrdictrrrTrXrz __classcell__rrrrrs$3x  rc @sBeZdZdZdeedeedeedeefddZdd Zd S) _EinmixDebuggerzUsed only to test mixinrcrdrerncCs||_||_||_||_dSr)rcrdrernrsrrrrTs z(_EinmixDebugger._create_rearrange_layerscCs||_||_dSr)saved_weight_shapesaved_bias_shapertrrrrXs z"_EinmixDebugger._create_parametersN) rxr{r|__doc__rrWrrTrXrrrrrs r)r\rUtypingrrrreinopsr einops.einopsreinops.parsingrr rNrWrrrrrrrs  D