o c۷il'@sddlmZmZddlZddlmZddlmmZddlm Z ddl m Z m Z ddl mZmZddlmZmZmZddlmZ  d%d e d e d ee de dededdede fddZ  d&d e d e d ee de dededde fddZ   d'd e d e de dedededee e e ee ee ffddZGdddejjZ   d(d e d e de dededed dede fd!d"ZGd#d$d$ej Z!dS)))OptionalLiteralN)Tensor) custom_fwd custom_bwd) cross_entropycross_entropy_fwd_out)gemmgemm_addgemm_add_inplace)linear_fwd_convert_typemeanFxweightbiastarget ignore_index reductionnonersuminplace_backwardreturncCs t|||}t|||||dS)NrrrFlinearr)rrrrrrryrP/home/ubuntu/vllm_env/lib/python3.10/site-packages/quack/linear_cross_entropy.pylinear_cross_entropy_funcs  r!cCs t|||}tj||||dS)N)rrr)rrrrrrrrrr linear_cross_entropy_func_refsr"T chunk_sizetunedc s||j\}}|j\}} |j} |d} ddksJd|ddks'Jtj|| tjd} tj|f| |jd} t|}| dkrLtj|tjdnd}d}d}ttfdd ||| |fDD]S\}\}}}}|jd}| d|}tj ||j |d |}t ||d|d||d tj |||d || dkr|}|}qc|dkrt |j |||d qct|j |||d qc| ||||fS)a4 Chunked forward pass for linear cross entropy. Splits input along batch dimension, computes matmul and cross_entropy_fwd for each chunk, stores dx for each chunk, and accumulates dw. Returns: loss: (B*L,) loss values dx: (B*L, d) gradient w.r.t. input dw: (V, d) gradient w.r.t. weight (accumulated across chunks except last) last_dlogits_chunk: (chunk_len, V) gradient of last chunk's logits (for deferred dw computation) last_x_chunk: (chunk_len, d) last chunk's input (for deferred dw computation) rz chunk_size must be multiple of 8)devicedtype)r)Nc3s|]}|VqdS)N)split).0tr$rr Qsz3chunked_linear_cross_entropy_fwd..)out)losslsedxr)r/r%r%)shaper(torchemptyfloat32r) empty_like enumeratezipmmmTrr Tr )rrrr$rr%B_LdV_r( num_chunksr0logits_chunk_preallocatedr2dwlast_dlogits_chunk last_x_chunkix_chunk target_chunk loss_chunkdx_chunk chunk_len logits_chunk dlogits_chunkrr-r chunked_linear_cross_entropy_fwd*sH      rOc@sdeZdZeedd    ddeded ed ed ed d edefddZ ee ddddZ dS)!ChunkedLinearCrossEntropyFunctioncuda) device_typer rr#Trrrrrrrr$r%cCs|j|_t||\}}|jdd}|d|jd}t||||||d\} } } } } | }|dkr4dn d||k}|| | | | |||_ ||_ ||_ ||_ |durY|S||S)zO Forward pass computes loss and stores dx and dw for backward. Nr3rg?) r) weight_dtyper r4reshaperOrfloatsave_for_backward batch_shaperrr%)ctxrrrrrr$r%rYr0r2rDrErFloss_sum loss_scalerrr forwardrs  z)ChunkedLinearCrossEntropyFunction.forwardc Cs|j\}}}}}|j}|dur||}|||jg|j|jdR}|dur6t|j||j||d}n|j|j krHt |j|||||dn t |j|||||j|d}||dddddfS)z Backward pass scales pre-computed gradients by dloss and completes the last chunk's dw computation. dloss is a scalar. NrT) out_dtypealphar%)r_betar%)r_r`r^r%) saved_tensorsr%mul_rVrYr4r r=rUr)r r )rZdlossr2rDrErFr\r%rrr backwards8   z*ChunkedLinearCrossEntropyFunction.backwardN)r rr#T) __name__ __module__ __qualname__ staticmethodrrintrboolr]rrdrrrr rPqs2rPrSc Cs0|dvr td|t|||||||}|S)a Chunked linear cross entropy with automatic differentiation support. Args: x: Input tensor of shape (B*L, d) weight: Weight tensor of shape (V, d) target: Target indices of shape (B*L,) chunk_size: Size of chunks to process ignore_index: Index to ignore in loss computation reduction: Type of reduction to apply tuned: Whether to use tuned kernels Returns: Loss tensor with specified reduction rSzInvalid reduction: ) ValueErrorrPapply)rrrr$rrr%r0rrr chunked_linear_cross_entropys rmcsreZdZ        ddededed ed ed d eed ededdffdd ZdededefddZ Z S)LinearCrossEntropyFr rNT in_features out_featuresrrrrr$rr%rc s8tj|||| | d||_||_||_||_||_dS)N)rr(r))super__init__rrr$rr%) selfrorprrrr$rr%r(r) __class__rr rrs   zLinearCrossEntropy.__init__inputrc Cs|jdurK|jrK|ddkrK|jddkrK|jddkrK|jddddkrK|jdurK|jddkrK|jdvrKt ||j ||j|j |j|j dSt ||j |j||j |j|jdS)NrTr&r'rrS)r$rrr%r)ris_cudastriderorpr4numelr$rrmrrr%r!r)rsrvrrrr r]s8    zLinearCrossEntropy.forward)Fr rNFTNN) rerfrgrirjrrrrrr] __classcell__rrrtr rns:  rn)r rF)r r)r#r T)r#r rT)"typingrrr5torch.nnnntorch.nn.functional functionalrr torch.amprrquack.cross_entropyrrquack.gemm_interfacer r r quack.linearr rirjr!r"tuplerOautogradFunctionrPrmLinearrnrrrr s      GS