o ©Ì³i½ã@s6ddlmZddlZddlmZGdd„dejƒZdS)é)ÚOptionalN)Únnc s`eZdZdZde ¡dœdejdejdeejdejf‡fdd „Zd e j d e j fd d „Z ‡Z S)Ú FeedForwardaëThis class implements the feed-forward network derived from Llama2. Args: gate_proj (nn.Module): Projection from input dim to hidden dim, fed through activation and multiplied by up_proj. down_proj (nn.Module): Final projection to output dim. up_proj (Optional[nn.Module]): Projection from input dim to hidden dim, multiplied by activation(gate_proj). activation (nn.Module): Activation function to use. Default is nn.SiLU(). N)Úup_projÚ activationÚ gate_projÚ down_projrrcs&tƒ ¡||_||_||_||_dS)N)ÚsuperÚ__init__Úw1Úw2Úw3r)Úselfrrrr©Ú __class__©úR/home/ubuntu/.local/lib/python3.10/site-packages/torchtune/modules/feed_forward.pyr s  zFeedForward.__init__ÚxÚreturncCs6| | |¡¡}|jdur|| |¡}| |¡}|S)ad Args: x (torch.Tensor): input tensor with shape ``(..., in_dim)``, where ``in_dim`` is the input dimension of both ``gate_proj`` and ``up_proj``. Returns: torch.Tensor: output tensor with shape ``(..., out_dim)``, where ``out_dim`` is the output dimension of ``down_proj``. N)rr r r )rrÚhrrrÚforward(s   zFeedForward.forward) Ú__name__Ú __module__Ú __qualname__Ú__doc__rÚSiLUÚModulerr ÚtorchÚTensorrÚ __classcell__rrrrrsúýüûúr)ÚtypingrrrrrrrrrÚs