o EiA) @sddlZddlZddlmZddlmZddlmZm Z m Z m Z ej ddZGdddZejdddfd e ejejfd e ed e ed e efd dZej jjddZGdddejZGdddejZGdddejZdS)N)partial)current_module)AnyCallableLiteralOptionalhaikuhkc@sLeZdZGdddejjjZGdddejjjZGdddejj Z dS) ParamFactoryc@eZdZddZddZdS)zParamFactory.ConcretecCs@||_||_|durt}t|dr|j}nd}||_||_dS)Ndtypefloat32)nameinitrhasattrr param_type)selfrrr rmodulerJ/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/einx/nn/haiku.py__init__ s  zParamFactory.Concrete.__init__cCsdt|j|j|j|jfSN)r CacheKeyrrr rrrrrto_value_and_keysz&ParamFactory.Concrete.to_value_and_keyN)__name__ __module__ __qualname__rrrrrrConcrete s rc@s,eZdZddZddZddZddZd S) zParamFactory.CacheKeycCs||_||_||_||_dSr)rrr r)rrrr rrrrrs zParamFactory.CacheKey.__init__cCst|j|j|j|jfSr)hashrrr rrrrr__hash__$szParamFactory.CacheKey.__hash__cCs<t|tjo|j|jko|j|jko|j|jko|j|jkSr) isinstancer rrrr r)rotherrrr__eq__'s     zParamFactory.CacheKey.__eq__cCsdt|j|j|j|j|fSr)r Tracerrrr r)rbackend virtual_argrrr to_tracer0szParamFactory.CacheKey.to_tracerN)rrrrr r#r'rrrrrs  rc@r )zParamFactory.TracercCs"||_||_||_||_||_dSr)rrr r depend_on)rrrr rr(rrrr7s  zParamFactory.Tracer.__init__cCsh|jdur|jn|dd}|jdur|jn|dd}|jdur$|jn|dd}|dur2td|dur:tdt|tr~|dvsG|dkrOtjj dd }n<|d krZtj d }n1|d kretj d }n&|dkrvtjj d dd|dd}ntd|dt|t t frtj |}|jdkrtj}n |jdkrtj}nJtjj|||||dtj||jgdS)Nrrr z=Must specify name for tensor factory hk.get_{parameter|state}z=Must specify init for tensor factory hk.get_{parameter|state}get_at rearrangeg{Gz?)stddevaddgmultiplyg?dotfan_intruncated_normalin_axis) fan_in_axesz3Don't know which initializer to use for operation '' parameterstateF)shaperr r)kwargsoutputr()rgetrr ValueErrorr!strthk initializers RandomNormalConstantVarianceScalingintfloatr get_parameter get_stateeinxtracerapplyTensorr()rr6r7rrr funcrrr__call__>s@       zParamFactory.Tracer.__call__N)rrrrrJrrrrr$6s r$N) rrrrErFinputInputrr TensorFactoryr$rrrrr sr rIrrr cCs@|tjkrd}n|tjkrd}ntd|dt||||S)aCreate a tensor factory for Haiku parameters. Args: func: Either ``hk.get_parameter`` or ``hk.get_state``. Defaults to ``hk.get_parameter``. name: Name of the parameter. If ``None``, uses a default name determined from the calling operation. Defaults to ``None``. init: Initializer for the parameter. If ``None``, uses a default init method determined from the calling operation. Defaults to ``None``. dtype: Data type of the parameter. If ``None``, uses the ``dtype`` member of the calling module or ``float32`` if it does not exist. Defaults to ``None``. Returns: A tensor factory with the given default parameters. r4r5zUnknown parameter function 'r3)r rCrDr:r r)rIrrr rrrrparamgs  rNcCs4t|ttjkst|ttjkrt|SdSr)idr rCrDrNr)xrrrtensor_factorys$ rQcszeZdZdZ          ddeded ed ed ed ed edededeedeedeffdd Z dddZ Z S)NormaNormalization layer. Args: stats: Einstein string determining the axes along which mean and variance are computed. Will be passed to ``einx.reduce``. params: Einstein string determining the axes along which learnable parameters are applied. Will be passed to ``einx.elementwise``. Defaults to ``"b... [c]"``. mean: Whether to apply mean normalization. Defaults to ``True``. var: Whether to apply variance normalization. Defaults to ``True``. scale: Whether to apply a learnable scale according to ``params``. Defaults to ``True``. bias: Whether to apply a learnable bias according to ``params``. Defaults to ``True``. epsilon: A small float added to the variance to avoid division by zero. Defaults to ``1e-5``. fastvar: Whether to use a fast variance computation. Defaults to ``True``. dtype: Data type of the weights. Defaults to ``"float32"``. decay_rate: Decay rate for exponential moving average of mean and variance. If ``None``, no moving average is applied. Defaults to ``None``. name: Name of the module. Defaults to ``None``. **kwargs: Additional parameters that specify values for single axes, e.g. ``a=4``. b... [c]Th㈵>r Nstatsparamsmeanvarscalebiasepsilonfastvarr decay_raterr7c  sTtj| d||_||_||_||_||_||_||_||_ | |_ | |_ | |_ dSNr) superrrUrVrWrXrYrZr[r\r r]r7) rrUrVrWrXrYrZr[r\r r]rr7 __class__rrrs z Norm.__init__c Cs<|jdur |dur td|jduo| pt}tjj||j|jf|r.|j r.t tj ddn|j |r<|j rtjj||jf|jrttjddndttjddd|jS)NrZr_weight)rZrp) rErelinearrorZrNr rCr7)rrPrrrrJs zLinear.__call__)Tr N) rrrrkr;rlrrrrJrmrrrarrns"rnc s@eZdZdZd dededeedeffdd Zd d Z Z S) DropoutaVDropout layer. Args: expr: Einstein string determining the axes along which dropout is applied. Will be passed to ``einx.elementwise``. drop_rate: Drop rate. name: Name of the module. Defaults to ``None``. **kwargs: Additional parameters that specify values for single axes, e.g. ``a=4``. Nro drop_raterr7c s$tj|d||_||_||_dSr^)r`rrorsr7)rrorsrr7rarrrs zDropout.__init__cCs.|rtjj||jf|jtd|jS|S)N)rsrng)rEredropoutrorsr next_rng_keyr7)rrPrhrrrrJszDropout.__call__r) rrrrkr;rBrrrrJrmrrrarrrs$ rr)rr rE functoolsrhaiku._src.basertypingrrrrrFimport_r<r rCrDr;rNrKregister_tensor_factoryrQModulerRrnrrrrrrs2  ^  R$