o EiR2@sddlmZddlZddlZddlmZddlmZ ddl m Z m Z m Z mZejddZGdddZ    d/de e ejfd e ed e ed e ejjd e ef d dZejjjddZGdddejZ          d0dedededededede ededed ejjd e ed efd!d"ZGd#d$d$ejZ   d1d%eded ejjd e ed ef d&d'Z Gd(d)d)ejZ! * d2d%ed+ed,ed e ed ef d-d.Z"dS)3N)partial)CallableUnionOptionalAnyz flax.linennnc@sLeZdZGdddejjjZGdddejjjZGdddejj Z dS) ParamFactoryc@eZdZddZddZdS)zParamFactory.ConcretecCs||_||_||_|durt|dr|j}nd}||_||_|dkr+|dur*tdn|dkr8|dur7tdntd|||_dS)Ndtypefloat32paramz/col is not accepted for flax.linen.Module.paramvariablez4col must be specified for flax.linen.Module.variablez)Unknown tensor factory flax.linen.Module.)modulenameinithasattrr col ValueError param_type)selfrrrr rrrI/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/einx/nn/flax.py__init__ s&  zParamFactory.Concrete.__init__cCs"|jt|j|j|j|j|jfSN)rrCacheKeyrrr rrrrrrto_value_and_key%sz&ParamFactory.Concrete.to_value_and_keyN)__name__ __module__ __qualname__rrrrrrConcrete s r c@s,eZdZddZddZddZddZd S) zParamFactory.CacheKeycC"||_||_||_||_||_dSrrrr rrrrrr rrrrrr+  zParamFactory.CacheKey.__init__cCst|j|j|j|j|jfSr)hashrrr rrrrrr__hash__2szParamFactory.CacheKey.__hash__cCsHt|tjo#|j|jko#|j|jko#|j|jko#|j|jko#|j|jkSr) isinstancerrrrr rr)rotherrrr__eq__5s      zParamFactory.CacheKey.__eq__cCs$t|j|j|j|j|j}||fSr)rTracerrrr rr)rbackend virtual_argxrrr to_tracer?szParamFactory.CacheKey.to_tracerN)rrrrr&r)r.rrrrr*s  rc@r )zParamFactory.TracercCr!rr"r#rrrrDr$zParamFactory.Tracer.__init__cCs|jdur|jn|dd}|jdur|jn|dd}|jdur$|jn|dd}|j}|dur5td|dur=tdt|tr|dksJ|dkrRtj j dd }n>|d kr\tj }n4|d krftj }n*|d krytj |d |d|d}ntd|dt|ttfrtj j||d}|jdkrtjj|j||||gtj|d}|S|jdksJtjj|j|||d||gd}tjjtj|dgtj|d}|S)Nrrr zGMust specify name for tensor factory flax.linen.Module.{param|variable}zGMust specify init for tensor factory flax.linen.Module.{param|variable}get_at rearrangeg{Gz?)stddevaddmultiplydotin_axisout_axis batch_axisz3Don't know which initializer to use for operation '')r r )argsoutputr )r9value)rgetrr rrr'strtnn initializersnormal zeros_init ones_init lecun_normalintfloatconstantreinxtracerapplyr Tensorr MemberAccess)rshapekwargsrrr rr-rrr__call__KsP     zParamFactory.Tracer.__call__N)rrrrrNrrrrr*Cs r*N) rrrrGrHinputInputr r TensorFactoryr*rrrrr srr-rrr rcCszt|dr|jtjjkr|j}d}n!t|dr$|jtjjkr$|j}d}nt|tjr/|}d}ntdt ||||||S)avCreate a tensor factory for Flax parameters. Args: x: The bound method of a Flax module, i.e. ``nn.Module.param`` or ``nn.Module.variable``, or a module instance in which case its ``param`` method is used. name: Name of the parameter. If ``None``, uses a default name determined from the calling operation. Defaults to ``None``. init: Initializer for the parameter. If ``None``, uses a default init method determined from the calling operation. Defaults to ``None``. dtype: Data type of the parameter. If ``None``, uses the ``dtype`` member of the calling module or ``float32`` if it does not exist. Defaults to ``None``. col: The collection name to use when ``bound_method`` is ``nn.Module.variable``. Returns: A tensor factory with the given default parameters. __func__r r zCx must be a bound method of a Flax module or a Flax module instance) rrRrModuler __self__r r'rrr )r-rrr rrrrrrr {s r cCsBt|tjst|dr|jtjjks|jtjjkrt|SdS)NrR)r'rrSrrRr r r)r-rrrtensor_factorys  rUc@seZdZUeed<dZeed<dZeed<dZeed<dZ eed<dZ eed<d Z e ed <d Z e ed <dZeed <dZejjed<d Zeed<ejdddZd S)_Normstatsb... [c]paramsTmeanvarscalebiasN decay_rateh㈵>epsilonfastvarr r rMc Csf|jdur |dur td|jduo| p|}tjj||j|jf|r1|jr1t |j dd|j dn|j|rB|j rBt |j dd|j dn|j |j rPt |j d|j dnd|jr]t |j d|j dnd|j|jd |jdurl|jni\}}}|jduo~|o~| }|r|jr| ddd}|j|jd |j||_|j r| ddd}|j|jd |j||_|S) Nz2training must be specified when decay_rate is usedrWrZ)rrr r[r\rr r])rZr[r\r]r`ra)r^ris_initializingrGrnormrWrYrZr r r r[r\r]r`rarMr;) rr-traininguse_emarZr[ update_emamean_emavar_emarrrrNsD z_Norm.__call__r)rrrr=__annotations__rYrZboolr[r\r]r^rEr`rar rdtypesDtyperMdictcompactrNrrrrrVs          rVrXTr_r rWrYrZr[r\r]r^r`rarMc Ks t|||||||||| | | d S)aNormalization layer. Args: stats: Einstein string determining the axes along which mean and variance are computed. Will be passed to ``einx.reduce``. params: Einstein string determining the axes along which learnable parameters are applied. Will be passed to ``einx.elementwise``. Defaults to ``"b... [c]"``. mean: Whether to apply mean normalization. Defaults to ``True``. var: Whether to apply variance normalization. Defaults to ``True``. scale: Whether to apply a learnable scale according to ``params``. Defaults to ``True``. bias: Whether to apply a learnable bias according to ``params``. Defaults to ``True``. epsilon: A small float added to the variance to avoid division by zero. Defaults to ``1e-5``. fastvar: Whether to use a fast variance computation. Defaults to ``True``. dtype: Data type of the weights. Defaults to ``"float32"``. decay_rate: Decay rate for exponential moving average of mean and variance. If ``None``, no moving average is applied. Defaults to ``None``. name: Name of the module. Defaults to ``None``. **kwargs: Additional parameters that specify values for single axes, e.g. ``a=4``. ) rYrZr[r\r]r^r`rar rrM)rV) rWrYrZr[r\r]r^r`rar rrMrrrNorms#rqc@sLeZdZUeed<dZeed<dZej j ed<dZ e ed<ej dd ZdS) _LinearexprTr]r r NrMcCsXtjj||jf|jrt|jd|jdndt|jd|jdd|jdur(|jSiS)Nr]rbweight)r]rt)rGrlinearrsr]r r rM)rr-rrrrNsz_Linear.__call__)rrrr=rkr]rlr rrmrnrMrorprNrrrrrrs   rrrscKt|||||dS)aLinear layer. Args: expr: Einstein string determining the axes along which the weight matrix is multiplied. Will be passed to ``einx.dot``. bias: Whether to apply a learnable bias. Defaults to ``True``. dtype: Data type of the weights. Defaults to ``"float32"``. name: Name of the module. Defaults to ``None``. **kwargs: Additional parameters that specify values for single axes, e.g. ``a=4``. )r]r rrM)rr)rsr]r rrMrrrLinear&srwc@sDeZdZUeed<eed<dZeed<dZeed<e j ddZ dS) _Dropoutrs drop_ratedropoutrng_collectionNrMcCsD|r tjj||jf|j||jd|jdur|jSiS|S)N)ryrng)rGrrzrsrymake_rngr{rM)rr-rfrrrrNAs z_Dropout.__call__) rrrr=rkrEr{rMrorrprNrrrrrx;s   rxrzryr{cKrv)aDropout layer. Args: expr: Einstein string determining the axes along which dropout is applied. Will be passed to ``einx.elementwise``. drop_rate: Drop rate. rng_collection: the rng collection name to use when requesting an rng key. Defaults to ``"dropout"``. name: Name of the module. Defaults to ``None``. **kwargs: Additional parameters that specify values for single axes, e.g. ``a=4``. )r{rrM)rx)rsryr{rrMrrrDropoutOsr~)NNNN) rXTTTTNr_Tr N)Tr N)rzN)# flax.linenlinenrrGflax functoolsr jax.numpynumpyjnptypingrrrrrHimport_r>rrSr=rmrnr rOregister_tensor_factoryrUrVrlrErqrrrwrxr~rrrrs   r   ( 2    3