o EiX+@sLddlZddlZddlZddlmZddlZddlmZm Z m Z m Z e ddej dddDZedkr?ed ej d d Zejd ZGd ddZ d!de ejjjejjjejjjfde e fddZejjjddZGdddejjZ eddZ!GdddejjZ"eddZ#GdddejjZ$edd Z%dS)"N)partial)CallableUnionOptionalAnyccs|]}t|VqdSN)int.0ir J/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/einx/nn/torch.py sr.)rrz7einx.nn.torch requires PyTorch version >= 2, but found cCs.dttvr tj|Sddlm}||S)Ncompilerr)dirtorchrallow_in_graph torch._dynamo_dynamo)funcrr r r _allow_in_graph s    rrc@sLeZdZGdddejjjZGdddejjjZGdddejj Z dS) ParamFactoryc@eZdZddZddZdS)zParamFactory.ConcretecCs||_||_dSr)paraminit)selfrrr r r __init__s zParamFactory.Concrete.__init__cCs|jt|jfSr)rrCacheKeyrrr r r to_value_and_keysz&ParamFactory.Concrete.to_value_and_keyN)__name__ __module__ __qualname__rr!r r r r Concretes r%c@s,eZdZddZddZddZddZd S) zParamFactory.CacheKeycC ||_dSrrrrr r r r# zParamFactory.CacheKey.__init__cCs t|jSr)hashrr r r r __hash__&r)zParamFactory.CacheKey.__hash__cCst|tjo |j|jkSr) isinstancerrr)rotherr r r __eq__)zParamFactory.CacheKey.__eq__cCst|j}||fSr)rTracerr)rbackend virtual_argxr r r to_tracer,s zParamFactory.CacheKey.to_tracerN)r"r#r$rr+r.r4r r r r r"s  rc@r)zParamFactory.TracercCr&rr'r(r r r r1r)zParamFactory.Tracer.__init__cs\|jdur|jn|dd}|}tj}tjj|jg|||fgd}|dur-tdt|t r|dks:|dkrDt t j jj dd}nT|dkrNt j jj}nJ|d krXt j jj}n@|d krtfd d |d D}td|d}t t j jjd|ddd}ntd|dt|ttfrt t j jj|d}tj}tjj||g|||fgd}|S)Nr)argsoutputinplace_updateszFMust specify init for tensor factory torch.nn.parameter.Uninitialized*get_at rearrangeg{Gz?)stdaddmultiplydotcsg|]}|qSr r r shaper r Msz0ParamFactory.Tracer.__call__..in_axisg?g۶%?ggg@)meanr:abz3Don't know which initializer to use for operation '')val)rgeteinxtracerTensorapply materialize ValueErrorr,strrttorchnnnormal_zeros_ones_npprodsqrt trunc_normal_rfloat constant_)rr?kwargsrr3r6fan_inr:r r>r __call__4sF     zParamFactory.Tracer.__call__N)r"r#r$rr\r r r r r00s r0N) r"r#r$rHrIinputInputr%r TensorFactoryr0r r r r rsrr3rcCs8t|tjjjtjjjfrt|tjjst ||S|S)aCreate a tensor factory for an uninitialized PyTorch parameter or buffer. If the given parameter is not initialized, this returns a tensor factory that calls the ``materialize`` method of ``uninitialized`` with the given shape and returns ``uninitialized``. Otherwise, the parameter is returned as is. Args: x: An instance of ``torch.nn.parameter.UninitializedParameter``, ``torch.nn.parameter.UninitializedBuffer`` or ``torch.nn.parameter.Parameter``. init: Initializer for the parameter. If ``None``, uses a default init method determined from the calling operation. Defaults to ``None``. Returns: A tensor factory with the given default parameters, or the parameter itself if it is already materialized. ) r,rrP parameterUninitializedParameterUninitializedBuffer _subclasses FakeTensorrr%)r3rr r r r`s  rcCs8t|tjjjtjjjfrt|tjjst| SdSr) r,rrPr`rarbrcrdrr!)r3r r r tensor_factorys  recsxeZdZdZ         ddeded ed ed ed ed ededeej efde ede ffdd Z ddZ ZS)NormaNormalization layer. Args: stats: Einstein string determining the axes along which mean and variance are computed. Will be passed to ``einx.reduce``. params: Einstein string determining the axes along which learnable parameters are applied. Will be passed to ``einx.elementwise``. Defaults to ``"b... [c]"``. mean: Whether to apply mean normalization. Defaults to ``True``. var: Whether to apply variance normalization. Defaults to ``True``. scale: Whether to apply a learnable scale according to ``params``. Defaults to ``True``. bias: Whether to apply a learnable bias according to ``params``. Defaults to ``True``. epsilon: A small float added to the variance to avoid division by zero. Defaults to ``1e-5``. fastvar: Whether to use a fast variance computation. Defaults to ``True``. dtype: Data type of the weights. Defaults to ``"float32"``. decay_rate: Decay rate for exponential moving average of mean and variance. If ``None``, no moving average is applied. Defaults to ``None``. **kwargs: Additional parameters that specify values for single axes, e.g. ``a=4``. b... [c]Th㈵>float32NstatsparamsrBvarscalebiasepsilonfastvardtype decay_raterZc  st||_||_||_||_||_||_| |_| |_ |r4| dur4| dt j j jtt | dnd|_|rN| durN| dt j j jtt | dnd|_|r_t j j jtt | dnd|_|rpt j j jtt | dnd|_d|_dS)NrBrqrlF)superrrjrkuse_meanuse_varrorprrrZregister_bufferrrPr`rbvarsrBrlrarmrn initialized) rrjrkrBrlrmrnrorprqrrrZ __class__r r rs0    z Norm.__init__c Cs*|jduo |j p |j }t||j|j|r|jr|jn|j|r%|jr%|j n|j|j dur/|j nd|j dur8|j nd|j |j |jd \}}}|jduoL|j}|rt2|duri|j|j|jd|j||dur~|j |j|j d|j|Wdn1swY|jsd|_|S)N)rBrlrmrnrorprZT)rrtrainingry_normrjrkrurBrvrlrmrnrorprZrno_gradcopy_)rr3use_emarBrl update_emar r r forwards2   ""z Norm.forward) rgTTTTrhTriN)r"r#r$__doc__rNboolrXrrrqrrrr __classcell__r r rzr rfsF    -rfc CsR|jtjj|||f||||||dd| WdS1s"wYdS)Nr)rBrlrmrnrorpr1)devicerHrPnorm) r3rjrkrBrlrmrnrorprZr r r r~s" $r~c sJeZdZdZ  d dededeejefde ffdd Z d d Z Z S) LinearaLinear layer. Args: expr: Einstein string determining the axes along which the weight matrix is multiplied. Will be passed to ``einx.dot``. bias: Whether to apply a learnable bias. Defaults to ``True``. dtype: Data type of the weights. Defaults to ``"float32"``. **kwargs: Additional parameters that specify values for single axes, e.g. ``a=4``. TriexprrnrqrZc sZttjjjtt|d|_|r"tjjjtt|d|_nd|_||_ ||_ dS)Nrs) rtrrrPr`rarxweightrnrrZ)rrrnrqrZrzr r rs  zLinear.__init__cCst||j|j|j|jSr)_linearrrrnrZrr3r r r rr/zLinear.forward)Tri) r"r#r$rrNrrrrqrrrrr r rzr rs  rcCsH|jtjj||||fddi|WdS1swYdS)Nr1r)rrHrPlinear)r3rrrnrZr r r rs$rcs6eZdZdZdededeffdd ZddZZ S) DropoutaDropout layer. Args: expr: Einstein string determining the axes along which dropout is applied. Will be passed to ``einx.elementwise``. drop_rate: Drop rate. **kwargs: Additional parameters that specify values for single axes, e.g. ``a=4``. r drop_raterZc s t||_||_||_dSr)rtrrrrZ)rrrrZrzr r r3s  zDropout.__init__cCs|jr t||j|j|jS|Sr)r}_dropoutrrrZrr r r r:szDropout.forward) r"r#r$rrNrXrrrrr r rzr r)s rcCsF|jtjj||f|dd|WdS1swYdS)Nr)rr1)rrHrPdropout)r3rrrZr r r rAs$rr)&rrHmath functoolsrnumpyrTtypingrrrrtuple __version__split_version ImportErrorrrIimport_rOrrPr`rarb Parameterrr]register_tensor_factoryreModulerfr~rrrrr r r r sD " M #  \ !