o -i1@sddlmZmZddlmZddlmZmZddlZddl m m Z ddlm Z ddlmZddl mZddlmZdd lmZmZdd lmZdd lmZmZmZdd lmZdd lmZddl m!Z!m"Z"m#Z#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,m-Z-ee.Z/edZ0e1Z2ej3Z4ej5j6j7j8Z9ej5j:j;j8ZGddde=Z?Gddde-Z@dS))ABCabstractmethod)Callable)Any ParamSpecN)fx)auto_functionalized)PatternMatcherPass) Attention) VllmConfigget_layers_from_vllm_config) init_logger)QuantKey kNvfp4DynamickStaticTensorScale)current_platform)round_up) QUANT_OPS empty_bf16 empty_fp32 empty_i32)is_func)enable_fake_mode)MatcherQuantFP8)VllmInductorPassVllmPatternMatcherPassPc @seZdZdZdededejddfddZd e d e dej fd d Z d e d e dej fd dZ e deeejfdeejgdfdeeejffddZe dejjddfddZe dejjddfddZdeddfddZededdfddZdS)AttentionQuantPatternzQ The base class for Attn+Quant fusions. Should not be used directly. layer quant_keydtypereturnNcCs\||_|j|_|j|_|j|_||_|j|_||_|jtvs&Jd|jt|j|_dS)Nz unsupported quantization scheme ) r layer_name num_heads head_sizer r! quant_dtyperQUANT_OP)selfrr r!r)Y/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/vllm/compilation/fusion_attn.py__init__.s  zAttentionQuantPattern.__init__argskwargscO |jdd|}tj|i|SNcudar!device)r!torchemptyr(r,r-r)r)r*r4AzAttentionQuantPattern.emptycOr.r/)r&r3r4r5r)r)r* empty_quantEr6z!AttentionQuantPattern.empty_quanttrace_fnprocess_fx_fnscs&dtjdtjdtjffdd }|S)Nr,r-r"cs$|i|}D]}||q |SNr))r,r-gm process_fxr9r8r)r*wrappedNs z4AttentionQuantPattern.wrap_trace_fn..wrapped)rr,r-r GraphModule)r8r9r>r)r=r* wrap_trace_fnIs"z#AttentionQuantPattern.wrap_trace_fnr;cCsddlm}||dS)Nr)view_to_reshape)#torch._inductor.fx_passes.post_gradrA)r;rAr)r)r*fx_view_to_reshapeWs  z(AttentionQuantPattern.fx_view_to_reshapecCsd|jjD]+}t|tjjjjsq|jd}t ddt |Dr!q| |jd|j |qdS)Nrcss|] \}}||kVqdSr:r)).0idimr)r)r* dsz=AttentionQuantPattern.remove_noop_permutes..r) graphnodesrr3opsatenpermutedefaultr,any enumeratereplace_all_uses_with erase_node)r;nodedimsr)r)r*remove_noop_permutes]s  z*AttentionQuantPattern.remove_noop_permutespm_passcCs"|jj|jr||dSdSr:)rimplfused_output_quant_supportedr _registerr(rUr)r)r*register_if_supportedksz+AttentionQuantPattern.register_if_supportedcCstr:)NotImplementedErrorrYr)r)r*rXoszAttentionQuantPattern._register)__name__ __module__ __qualname____doc__r rr3r!r+rTensorr4r7 staticmethodrrrr?r@rCrTr rZrrXr)r)r)r*r(s8      rc sJeZdZdZ d dedejdeddffdd Zd e ddfd d Z Z S)AttentionFp8StaticQuantPatterna? Fusion for Attention+Fp8StaticQuant. Only triggers when the attention implementation returns True in `fused_output_quant_supported()`. If the pattern is found, the Fp8StaticQuant op will be removed from the graph, and its scale will be passed into Attention op as the `output_scale` argument. Trr! symmetricr"Ncs,ttt|d}t|||t||_dS)N)r!scalerc)r FP8_DTYPErsuperr+r quant_matcher)r(rr!rcr  __class__r)r*r+~s z'AttentionFp8StaticQuantPattern.__init__rUc sdtjdtjdtjdtjdtjdtjf fdd }dtjdtjdtjdtjdtjdtjf fd d }d jjd jjd jjd jjtd d g}t|||t tj tj tj |dS) Nqkv output_attnrdr"c sJtt||||jddd}t|d|jdjjg}||dS)Nquerykeyvalueoutputr# output_scaleoutput_block_scalerr)rATTN_OPr# RESHAPE_OPshaper$r%rg)rjrkrlrmrdat1 attn_out_viewr(r)r*patterns z9AttentionFp8StaticQuantPattern._register..patternc s`tjjjj|jdjjgdj|j d}t t ||||j |dd}t |ddjjgS)Nrr1rnr)r3rJrKfullrMrwr$r%r&r2rrur#rv)rjrkrlrmrdrxrzr)r* replacements"  z=AttentionFp8StaticQuantPattern._register..replacementr) r3r`r4r$r%rpmregister_replacementrr@fwd_onlyrCrTr(rUr{rinputsr)rzr*rXsVz(AttentionFp8StaticQuantPattern._register)T) r\r]r^r_r r3r!boolr+r rX __classcell__r)r)rhr*rbts  rbcsBeZdZdZdedejddffdd Zdeddfd d Z Z S) AttentionNvfp4QuantPatterna7 Fusion for Attention+Nvfp4Quant. Only triggers when the attention implementation returns True in `fused_output_quant_supported()`. If the pattern is found, the Nvfp4Quant op will be removed from the graph, and its scale will be passed into Attention op as the `output_scale` argument. rr!r"Ncst|t|dSr:)rfr+r)r(rr!rhr)r*r+sz#AttentionNvfp4QuantPattern.__init__rUcs*dtjdtjdtjdtjdtjdtjdtjdttjtjfffd d }dtjdtjdtjdtjdtjdtjdtjdttjtjfffd d }td jjtd jjtd jjtd jjd jjdtdtjjddt ddg}t |||t t jt jt j|dS)Nrjrkrlrm output_quantrs input_scaler"c srtt||||jddd}t|d|jdjjg}tj||||dd} tj j j | dt } | d| fS)NrnrrT)rrinputrsris_sf_swizzled_layout)rrur#rvrwr$r%r'r3rJrKviewr!re) rjrkrlrmrrsrrxryat2output_scale_viewrzr)r*r{s.   z5AttentionNvfp4QuantPattern._register..patternc stjjjj|jdjjdgdj|j d}tjjj |t }t t||||j||d}t|ddjjdg} | |dfS)Nrrr|r1rnrr})r3rJrKr~rMrwr$r%r&r2rr!rerrur#rv) rjrkrlrmrrsrrrrrrzr)r*rs&  z9AttentionNvfp4QuantPattern._register..replacementrrr)r3r`tuplerr$r%r7rrrrrrr@rrCrTrr)rzr*rXsn!  z$AttentionNvfp4QuantPattern._register) r\r]r^r_r r3r!r+r rXrr)r)rhr*rs rcsZeZdZdZededdffdd Zejde j j j ddfdd Z defd d ZZS) AttnFusionPassa This pass fuses post-attention quantization onto attention if supported. It uses the pattern matcher and matches each layer manually, as strings cannot be wildcarded. This also lets us check support on attention layers upon registration instead of during pattern matching. Currently, only static fp8 quant is supported, but patterns could easily be added for other quant schemes and dtypes. The bigger hurdle for wider support are attention kernels, which need to support fusing output quant. configr"Ncst|tdd|_t|t}|D])\}}t||jj }| |jt r>t tjjdr>t||jj }| |jqt|dkrJtd|||jdS)Nattn_fusion_pass) pass_namescaled_fp4_quantrzAttention + quant fusion is enabled, but no attention layers were found in CompilationConfig.static_forward_context so no fusion patterns were registered.)rfr+r patternsr r itemsrb model_configr!rZris_cudahasattrr3rJ_Crlenloggerwarning dump_patterns)r(r attn_layersr#r pattern_fp8 pattern_nvfp4rhr)r*r+Es&      zAttnFusionPass.__init__rHcCs |j||_td|jdS)Nz#Fused quant onto %s attention nodes)rapply matched_countrdebug)r(rHr)r)r*__call__aszAttnFusionPass.__call__cCst|tttSr:)r hash_sourcerrbrrzr)r)r*uuidfs zAttnFusionPass.uuid)r\r]r^r_rr r+r time_and_logr3rrHGraphrstrrrr)r)rhr*r8s r)Aabcrrcollections.abcrtypingrrr3torch._inductor.pattern_matcher _inductorpattern_matcherrr*torch._higher_order_ops.auto_functionalizerr vllm.attention.layerr vllm.configr r vllm.loggerr 9vllm.model_executor.layers.quantization.utils.quant_utilsrrrvllm.platformsrvllm.utils.math_utilsrfusionrrrrfx_utilsr inductor_passr matcher_utilsrvllm_inductor_passrrr\rr fp8_dtypereuint8 FP4_DTYPErJvllmunified_attention_with_outputrMrurKreshapervrrbrrr)r)r)r*s:             L]g