o c۷iU@s2ddlZddlmZddlmZddlmZddlmZm Z m Z ddl m Z m Z ddlmZmZeeeefBZeejjdejdZe dddd eeBd efd d Ze dddd ed efddZe ddddeded efddZe dddd ed efddZe ejdddd eded eeeffddZe dddd ed efddZe ejdddd eded eeeffddZe dddd ed efddZe dddd eded eeeffddZe ejdddd ed efd d!Z e ejddddeded efd"d#Z!e d$ddd%d ed&e"d efd'd(Z#e dddd ed)ed efd*d+Z$e d$ddd%d ed)eded&e"d eeeeff d,d-Z%e .dCdddd ed)ed/ed efd0d1Z&e .dCdddd ed)eded/ed eeeeff d2d3Z'e dddd ed)ed efd4d5Z(e dddd ed)eded eeeeffd6d7Z)e dddd ed)ed efd8d9Z*e ejdddd ed)eded eeeeffd:d;Z+e dddd ed)ed efdd?Z-de#eeed@Z.deeedAZ/e$e&e*e,e(dBZ0e%e'e+e-e)dBZ1dS)DN)Tuple)partial)Float32Boolean const_expr)T dsl_user_op)llvmnvvm)src_c calc_funclocipareturnc Cs4ttjtt|j||dgddddtjjdS)Nr ztanh.approx.f32 $0, $1;z=f,fF)has_side_effectsis_align_stack asm_dialect)rr inline_asmrf32ir_value AsmDialectAD_ATT)rrrrF/home/ubuntu/vllm_env/lib/python3.10/site-packages/quack/activation.pytanhsrxcCsZtt|t rddtd|Stjd|}t|dt|df}tj|ddS)N?rrrr isinstancetuplercutearchmul_packed_f32x2fma_packed_f32x2)rrrx_half tanh_x_halfrrrsigmoid&s r*outdoutcCs||||SNr)r+r,rrrrrdsigmoid_from_output1sr.cCsNtt|t rtj|tdStj|dtdtj|dtdfSNrr )rr"r#r$r%fmaxr)rrrrrrrelu7s,r2cCstt|t rt|dk}|r|ntdtj|tdfSt|ddk}t|ddk}|r5|dntd|r>|dntdf}|t|fS)Nrr0r ) rr"r#rrr$r%r1r2)rr,rrx_posx0_posx1_posdxrrrdrelu?s "( r7cCs`tt|t rtj|td|Stj|dtdtj|dtdf}tj||Sr/)rr"r#r$r%r1rr&)rrrrelu_xrrrrelu_sqNs,r9cCsjtt|t rt|}||}d||}||fSt|}tj||}tjdtj||}||fS)a ReLU squared backward pass: computes gradient w.r.t. x and recomputes forward Given: relu_sq_out = max(x, 0) * x, and dout = grad w.r.t. relu_sq_out Returns: (dx, relu_sq_out) where: - dx = dout * 2 * x if x > 0, else 0 - relu_sq_out = max(x, 0) * x @)r:r:)rr"r#r2r$r%r&)rr,rrr8 relu_sq_outr6rrrdrelu_sqWs  r<c Cstdtj}d|}tt|t r&d|dt|||||Stj ||}tj |||f||f}tj ||}t|dt|df}tj |||} tj d| S)z gelu(x) = 0.5 * x * (1 + tanh(sqrt(2/pi) * (x + 0.044715 * x^3))) = 0.5 * x * (1 + tanh(x * (0.797885 + 0.0356774 * x * x))) Hm?r?rr r mathsqrtpirr"r#rr$r%r&r') rrrsqrt_2_over_pisqrt_2_over_pi_coeffx_sq x_sq_scaledztanh_zx_tanh_zrrrgelu_tanh_approxps rKcCstdtj}d|}d|}tt|t rN||}t||||}dd|} || } d||} |||} | |d| | } || }|| fStj ||}tj |||f||f}tj ||}t|dt|df}tj |dd} tj || } tj ||d |d fd} tj |||f||f} tj | | }tj ||}tj |d| } tj || }|| fS) a GELU tanh approximation backward pass: computes gradient w.r.t. x and recomputes forward Given: gelu_out = 0.5 * x * (1 + tanh(x * (c1 + c2 * x^2))), and dout = grad w.r.t. gelu_out Returns: (dx, gelu_out) Derivative uses the chain rule: d/dx[gelu(x)] = 0.5 * (1 + tanh(z)) + 0.5 * x * sech^2(z) * dz/dx where z = x * (c1 + c2 * x^2), dz/dx = c1 + 3 * c2 * x^2 and sech^2(z) = 1 - tanh^2(z) r=r>g@rr rrr?r?r@)rr,rrrDrEsqrt_2_over_pi_coeff_3rFrIhalf_tanh_z_plus_onegelu_outsech2_zdz_dxdgelur6rGrH sech2_dz_dx x_sech2_dz_dxrrrdgelu_tanh_approxs<    rUc Cs*tt|t r$t|dk}|s"tjjttjj|dddddS|St tj }tj |||f}tjj|dddtjj|dddf}tj |d}tjj |dddtjj |dddf}td} tj || | f} t|ddk} t|ddk} | s| dn|d| s| dfS|dfS) N4@Tfastmathr?rr rLr:)rr"r#rr$rAlogrexplog2er%r&add_packed_f32x2) rrr use_linearlog2_ex_log2ex_expx_exp_p1 log_x_exp_p1ln2 softplus_x use_linear_0 use_linear_1rrrsoftpluss. $ (  rhcCs2t|dk}||tjj| dd}|s|S|S)NrVTrW)rr$rArZ)r+r,rrr^r6rrrdsoftplus_from_outputs  riF)already_halvedrrrjcCsztt|t rt| rd|n|}|t||St| r'tjd|n|}t|dt|df}tj|||S)z silu(x) = x * sigmoid(x) = x * (1 + tanh(x / 2)) / 2 = (0.5 * x) * tanh(0.5 * x) + (0.5 * x) This compiles down to 3 SASS instructions: FMUL to get 0.5 * x, MUFU.TANH, and FFMA. rrrr r!)rrjrrr(r)rrrsilus rkycC.tt|t rt||Stjt||Sr-)rr"r#rkr$r%r&rrlrrrrrswiglus rocCsHtt|t rCt| rt|}||}nt|}d|d}|||}||} ||||| } | |} | } ||} | | | fSt| rTt|}tj||}nt|dt|df}tj|dd}tj|||}tj||} tj||d |d f|}tj||| } tj| |} | } tj||} | | | fS)a SwiGLU backward pass: computes gradients w.r.t. x (gate) and y (up projection) Given: swiglu_out = silu(x) * y, and dout = grad w.r.t. swiglu_out Returns: (dx, dy, swiglu_out) where dx = dout * y * d_silu(x), dy = dout * silu(x) d_silu(x) = sigmoid(x) * (1 + x * (1 - sigmoid(x))) This has been optimized to use fewer instructions (i.e. we expand things out to use FFMA instead of FADD and FMUL). rrr r) rr"r#r*rr$r%r&r')rrlr,rjrr sigmoid_xsilu_xtanh_x silu_x_dout d_silu_x_doutr6dy swiglu_out sigmoid_x_minus_silu_x_sigmoid_xrrrdswiglus<       rxZd;?alphac Cstt|t rd|}|t|||}|||Stjd|}tj||f|}t|dt|df}tj|||}tj|||S)a"The swiglu variant used in gpt-oss, which has a scaling factor on x and bias of 1 to y. https://github.com/openai/gpt-oss/blob/7be9334950053a888e24887a57dac797a17d6e00/gpt_oss/torch/model.py#L249 x * sigmoid(alpha * x) * (y + 1) Compile down to FMUL, FMUL, TANH, FFMA, FFMA rrrr r!) rrlrzrrr(rq alpha_x_halftanh_alpha_x_halfrrr swiglu_oaiIs  r}cCsHtt|t r=d||}ddt|}||}||} ||||||} | || } | } |||} | | | fStjd|d|f|}t|dt|df}tj|dd}tj||}tj||} tj||d |d f|}tj||f||}tj||} tj| || } | } tj|||} | | | fS)au Swiglu OAI backward pass: computes gradients w.r.t. x and y Given: swiglu_oai_out = x * sigmoid(alpha * x) * (y + 1), and dout = grad w.r.t. swiglu_oai_out Returns: (dx, dy, swiglu_oai_out) Derivative of x * sigmoid(alpha * x) w.r.t. x: d/dx[x * sigmoid(alpha * x)] = sigmoid(alpha * x) + alpha * x * sigmoid(alpha * x) * (1 - sigmoid(alpha * x)) rrr rr!)rrlr,rzrrr{sigmoid_alpha_xrqrsrtr6rurvr|silu_x_minus_productsigmoid_plus_alpha_diffrrr dswiglu_oai`s4      rcCs6tt|t rt|}||St|}tj||S)zGLU: Gated Linear Unit glu(x, y) = sigmoid(x) * y Using tanh to compute sigmoid: sigmoid(x) = 0.5 * (1 + tanh(x/2)) )rr"r#r*r$r%r&)rrlrrrprrrglus rc Cstt|t r!t|}||}||}|||}|} || |fSt|}tj||}tj||}t||} tj| |}|} || |fS)a/ GLU backward pass: computes gradients w.r.t. x (gate) and y (up projection) Given: glu_out = sigmoid(x) * y, and dout = grad w.r.t. glu_out Returns: (dx, dy, glu_out) where: - dx = dout * y * sigmoid(x) * (1 - sigmoid(x)) - dy = dout * sigmoid(x) - glu_out = sigmoid(x) * y )rr"r#r*r$r%r&sub_packed_f32x2) rrlr,rrrpsigmoid_x_doutglu_outr6ruy_minus_glu_outrrrdglus     rcCs<tt|t rtj|td|St|}tj||S)zPReGLU: ReLU Gated Linear Unit reglu(x, y) = relu(x) * y = max(x, 0) * y r0) rr"r#r$r%r1rr2r&)rrlrrr8rrrreglusrc Cstt|t r.t|dk}tj|td}|r||ntd}||}||} ||| fSt|ddk} t|ddk} t|}tj ||} | rO| dntd| rX| dntdf}tj ||}tj ||} ||| fS)a! ReGLU backward pass: computes gradients w.r.t. x (gate) and y (up projection) Given: reglu_out = relu(x) * y, and dout = grad w.r.t. reglu_out Returns: (dx, dy, reglu_out) where: - dx = dout * y if x > 0, else 0 - dy = dout * relu(x) - reglu_out = relu(x) * y rr0r ) rr"r#rr$r%r1rr2r&) rrlr,rrr3r8r6ru reglu_outr4r5dout_yrrrdreglus  ( rcCrm)zhGeGLU: GELU Gated Linear Unit geglu(x, y) = gelu(x) * y Uses the tanh approximation of GELU )rr"r#rKr$r%r&rnrrrgeglus rc Cstt|t r t||\}}||}||}||} ||| fSt||\}}tj||}tj||}tj||} ||| fS)a GeGLU backward pass: computes gradients w.r.t. x (gate) and y (up projection) Given: geglu_out = gelu(x) * y, and dout = grad w.r.t. geglu_out Returns: (dx, dy, geglu_out) where: - dx = dout * y * d_gelu(x) - dy = dout * gelu(x) - geglu_out = gelu(x) * y )rr"r#rUr$r%r&) rrlr,rr dgelu_x_doutgelu_xr6ru geglu_outrrrdgeglus   r)Nrkr2r9rK)Nr2r9rK)ror}rrr)ry)2rAtypingr functoolsr cutlass.cuter$cutlassrrrcutlass.cutlass_dslrrcutlass._mlir.dialectsr r F32_or_F32x2r%calc_packed_f32x2_oprfloatrr*r.r2jitr7r9r<rKrUrhriboolrkrorxr}rrrrrrr act_fn_map dact_fn_map gate_fn_map dgate_fn_maprrrrsT          >""   A   /    #       "