o پiwQ@sddlZddlmZddlmZddlmZmZmZddl m Z m Z ddl m Z ddlmZeeeefBZe ddddeeBdefd d Ze dddd edefd d Ze ddddededefddZe dddd edefddZe ejdddd ededeeeffddZe dddd edefddZe ejdddd ededeeeffddZe dddd edefddZe dddd ededeeeffddZe ejdddd edefddZe ejddddededefd d!Ze d"ddd#d ed$edefd%d&Ze dddd ed'edefd(d)Z e d"ddd#d ed'eded$edeeeeff d*d+Z!e ,d>dddd ed'ed-edefd.d/Z"e ,d>dddd ed'eded-edeeeeff d0d1Z#e dddd ed'edefd2d3Z$e dddd ed'ededeeeeffd4d5Z%e dddd ed'edefd6d7Z&e ejdddd ed'ededeeeeffd8d9Z'e dddd ed'edefd:d;Z(e dddd ed'ededeeeeffd|dntdf}|t|fS)Nrr+r) rrrrrr,r-r.r/)rr'r r x_posx0_posx1_posdxrrrdrelu9s "( r4cCs^tt|t rtj|td|Stj|dtdtj|dtdf}t||Sr*) rrrr,r-r.rr r!)rr r relu_xrrrrelu_sqHs, r6cCsdtt|t rt|}||}d||}||fSt|}t||}tdt||}||fS)a ReLU squared backward pass: computes gradient w.r.t. x and recomputes forward Given: relu_sq_out = max(x, 0) * x, and dout = grad w.r.t. relu_sq_out Returns: (dx, relu_sq_out) where: - dx = dout * 2 * x if x > 0, else 0 - relu_sq_out = max(x, 0) * x @)r7r7)rrrr/r r!)rr'r r r5 relu_sq_outr3rrrdrelu_sqQs   r9c Cstdtj}d|}tt|t r&d|dt|||||St||}t |||f||f}t||}t|dt|df}t |||} td| S)z gelu(x) = 0.5 * x * (1 + tanh(sqrt(2/pi) * (x + 0.044715 * x^3))) = 0.5 * x * (1 + tanh(x * (0.797885 + 0.0356774 * x * x))) Hm?r?rrr mathsqrtpirrrrr r!r") rr r sqrt_2_over_pisqrt_2_over_pi_coeffx_sq x_sq_scaledztanh_zx_tanh_zrrrgelu_tanh_approxjs    rHcCsjtdtj}d|}d|}tt|t rN||}t||||}dd|} || } d||} |||} | |d| | } || }|| fSt||}t |||f||f}t||}t|dt|df}t |dd} t|| } t ||d |d fd} t |||f||f} t| | }t||}t |d| } t|| }|| fS) a GELU tanh approximation backward pass: computes gradient w.r.t. x and recomputes forward Given: gelu_out = 0.5 * x * (1 + tanh(x * (c1 + c2 * x^2))), and dout = grad w.r.t. gelu_out Returns: (dx, gelu_out) Derivative uses the chain rule: d/dx[gelu(x)] = 0.5 * (1 + tanh(z)) + 0.5 * x * sech^2(z) * dz/dx where z = x * (c1 + c2 * x^2), dz/dx = c1 + 3 * c2 * x^2 and sech^2(z) = 1 - tanh^2(z) r:r;g@rrrrr<r<r=)rr'r r rArBsqrt_2_over_pi_coeff_3rCrFhalf_tanh_z_plus_onegelu_outsech2_zdz_dxdgelur3rDrE sech2_dz_dx x_sech2_dz_dxrrrdgelu_tanh_approxs<         rRc Cs$tt|t r$t|dk}|s"tjjttjj|dddddS|St tj }t |||f}tjj|dddtjj|dddf}t |d}tjj |dddtjj |dddf}td} t || | f} t|ddk} t|ddk} | s| dn|d| s| dfS|dfS) N4@Tfastmathr<rrrIr7)rrrrr,r>logrexplog2er r!add_packed_f32x2) rr r use_linearlog2_ex_log2ex_expx_exp_p1 log_x_exp_p1ln2 softplus_x use_linear_0 use_linear_1rrrsoftpluss. $ (   recCs2t|dk}||tjj| dd}|s|S|S)NrSTrT)rr,r>rW)r&r'r r r[r3rrrdsoftplus_from_outputs  rfF)already_halvedr r rgcCsvtt|t rt| rd|n|}|t||St| r&td|n|}t|dt|df}t|||S)z silu(x) = x * sigmoid(x) = x * (1 + tanh(x / 2)) / 2 = (0.5 * x) * tanh(0.5 * x) + (0.5 * x) This compiles down to 3 SASS instructions: FMUL to get 0.5 * x, MUFU.TANH, and FFMA. rrrrr)rrgr r r#r$rrrsilus rhycC,tt|t rt||Stt||Sr()rrrrhr r!rrir r rrrswiglus rlcCs8tt|t rCt| rt|}||}nt|}d|d}|||}||} ||||| } | |} | } ||} | | | fSt| rSt|}t||}nt|dt|df}t|dd}t|||}t||} t||d |d f|}t||| } t| |} | } t||} | | | fS)a SwiGLU backward pass: computes gradients w.r.t. x (gate) and y (up projection) Given: swiglu_out = silu(x) * y, and dout = grad w.r.t. swiglu_out Returns: (dx, dy, swiglu_out) where dx = dout * y * d_silu(x), dy = dout * silu(x) d_silu(x) = sigmoid(x) * (1 + x * (1 - sigmoid(x))) This has been optimized to use fewer instructions (i.e. we expand things out to use FFMA instead of FADD and FMUL). rrrr)rrrr%rr r!r")rrir'rgr r sigmoid_xsilu_xtanh_x silu_x_dout d_silu_x_doutr3dy swiglu_out sigmoid_x_minus_silu_x_sigmoid_xrrrdswiglus8          ruZd;?alphac Cstt|t rd|}|t|||}|||Std|}t||f|}t|dt|df}t|||}t|||S)a"The swiglu variant used in gpt-oss, which has a scaling factor on x and bias of 1 to y. https://github.com/openai/gpt-oss/blob/7be9334950053a888e24887a57dac797a17d6e00/gpt_oss/torch/model.py#L249 x * sigmoid(alpha * x) * (y + 1) Compile down to FMUL, FMUL, TANH, FFMA, FFMA rrrrr) rrirwr r r#rn alpha_x_halftanh_alpha_x_halfrrr swiglu_oaiAs   rzcCs6tt|t r=d||}ddt|}||}||} ||||||} | || } | } |||} | | | fStd|d|f|}t|dt|df}t|dd}t||}t||} t||d |d f|}t||f||}t||} t| || } | } t|||} | | | fS)au Swiglu OAI backward pass: computes gradients w.r.t. x and y Given: swiglu_oai_out = x * sigmoid(alpha * x) * (y + 1), and dout = grad w.r.t. swiglu_oai_out Returns: (dx, dy, swiglu_oai_out) Derivative of x * sigmoid(alpha * x) w.r.t. x: d/dx[x * sigmoid(alpha * x)] = sigmoid(alpha * x) + alpha * x * sigmoid(alpha * x) * (1 - sigmoid(alpha * x)) rrrrr)rrir'rwr r rxsigmoid_alpha_xrnrprqr3rrrsrysilu_x_minus_productsigmoid_plus_alpha_diffrrr dswiglu_oaiXs4         r~cCs4tt|t rt|}||St|}t||S)zGLU: Gated Linear Unit glu(x, y) = sigmoid(x) * y Using tanh to compute sigmoid: sigmoid(x) = 0.5 * (1 + tanh(x/2)) )rrrr%r r!)rrir r rmrrrglus  rc Cstt|t r!t|}||}||}|||}|} || |fSt|}t||}t||}t||} t| |}|} || |fS)a/ GLU backward pass: computes gradients w.r.t. x (gate) and y (up projection) Given: glu_out = sigmoid(x) * y, and dout = grad w.r.t. glu_out Returns: (dx, dy, glu_out) where: - dx = dout * y * sigmoid(x) * (1 - sigmoid(x)) - dy = dout * sigmoid(x) - glu_out = sigmoid(x) * y )rrrr%r r!sub_packed_f32x2) rrir'r r rmsigmoid_x_doutglu_outr3rry_minus_glu_outrrrdglus        rcCs:tt|t rtj|td|St|}t ||S)zPReGLU: ReLU Gated Linear Unit reglu(x, y) = relu(x) * y = max(x, 0) * y r+) rrrr,r-r.rr/r r!)rrir r r5rrrreglus rc Cstt|t r.t|dk}tj|td}|r||ntd}||}||} ||| fSt|ddk} t|ddk} t|}t ||} | rN| dntd| rW| dntdf}t ||}t ||} ||| fS)a! ReGLU backward pass: computes gradients w.r.t. x (gate) and y (up projection) Given: reglu_out = relu(x) * y, and dout = grad w.r.t. reglu_out Returns: (dx, dy, reglu_out) where: - dx = dout * y if x > 0, else 0 - dy = dout * relu(x) - reglu_out = relu(x) * y rr+r) rrrrr,r-r.rr/r r!) rrir'r r r0r5r3rr reglu_outr1r2dout_yrrrdreglus   (   rcCrj)zhGeGLU: GELU Gated Linear Unit geglu(x, y) = gelu(x) * y Uses the tanh approximation of GELU )rrrrHr r!rkrrrgeglus rc Cs|tt|t r t||\}}||}||}||} ||| fSt||\}}t||}t||}t||} ||| fS)a GeGLU backward pass: computes gradients w.r.t. x (gate) and y (up projection) Given: geglu_out = gelu(x) * y, and dout = grad w.r.t. geglu_out Returns: (dx, dy, geglu_out) where: - dx = dout * y * d_gelu(x) - dy = dout * gelu(x) - geglu_out = gelu(x) * y )rrrrRr r!) rrir'r r dgelu_x_doutgelu_xr3rr geglu_outrrrdgeglus      r)rv)*r>typingr cutlass.cuter,cutlassrrrcutlass.cutlass_dslrrcutlass._mlir.dialectsr quack.utilsr F32_or_F32x2floatrr%r)r/jitr4r6r9rHrRrerfboolrhrlrurzr~rrrrrrrrrrs           >""   ?   /    #