o -id+@stddlZddlmZddlZddlmZddlmZddlm Z ddl m Z ddl m Z eeZGd d d e ZdS) N)Iterable)auto_functionalized) init_logger)current_platform)is_func)VllmInductorPassc @s"eZdZdZejdejjddfddZ dejj e ejj Bddfdd Z ddejjd ejj d e eejj eBfd eejj eBd fdBddf ddZd ejj d e eejj eBfddfddZd ejj de eejj ffddZ ddejjd ejj d eejj eBd fdBddfddZdS)FixFunctionalizationPassa" This pass defunctionalizes certain nodes to avoid redundant tensor copies. After this pass, DCE (dead-code elimination) should never be run, as de-functionalized nodes may appear as dead code. To add new nodes to defunctionalize, add to the if-elif chain in __call__. graphreturnNcCsttr tddSg|_d}|jD]}t|tsq|j}|j d}|t j j j jkr|d}|d}||}t|tjrt|tjr|j d|j dkrt|j dt j jjjrtdd|Dr|j dj d} |D] } | jD]} t| t j jjjr| | || qw|| qr|||||nddd} |||| n|t j j jjkrdd d} |||| n|t j j jjkrd d d} |||| n|t j j jjkrd d d d } |||| n|t j j jjt j j j jfvrd d i} |||| nt!t j j"dr(|t j j"j#jkr(dd dddd} |||| nh|t j j j$jkr?d d i} |j||| ddnQ|t j j j%jkrVd d i} |j||| ddn:t!t j j drv|t j j j&jkrvd dd} |j||| ddn|t j j j'jkrd di} d} |j||| | dnq|d 7}q|(|dt)|j}|jD]}|*|qtd|||j+dS)NzBXPU platform does not support fix functionalizationpass currently.rquerykeycss.|]}|jD] }t|tjjjjVqqdSN)usersrtorchopsaten slice_scatterdefault).0 getitem_nodeuserrc/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/vllm/compilation/fix_functionalization.py 9s z4FixFunctionalizationPass.__call__..)rinputresidualresultscale)rrr&flashinfer_trtllm_fused_allreduce_norm allreduce_innorm_out quant_out scale_out)rrr )rrargs)rrrsilu_and_mul_nvfp4_quantresult_block_scale)rr+rinput_global_scaleqkv) r- num_heads_q num_heads_k num_heads_vhead_dimepsq_weightk_weight cos_sin_cacheis_neox position_ids) mutated_argsr)before_cleanupz,De-functionalized %s nodes, removed %s nodes),ris_xpuloggerdebugnodes_to_removenodesrrkwargsr)rr_Crotary_embeddingr getitem_usersoperatorgetitemrsplit_with_sizesallvaluesrrreplace_all_uses_with_removeinsert_defunctionalizeddefunctionalizefused_add_rms_norm#fused_add_rms_norm_static_fp8_quant rms_norm_dynamic_per_token_quantrms_normrms_norm_static_fp8_quanthasattrvllmr! silu_and_mulsilu_and_mul_quantr*fused_qk_norm_rope dump_graphlen erase_nodeclear)selfr countnoder? at_targetr r getitem_nodesmm_noderuser_of_getitemr8r) count_removedrrr__call__s                              z!FixFunctionalizationPass.__call__ node_or_nodescCs.t|tjjr|j|dS|j|dS)zM Stage a node (or nodes) for removal at the end of the pass. N) isinstancerfxNoder=appendextend)rZrcrrrrIsz FixFunctionalizationPass._remover\r8r).cCs*||||j|||d||dS)z De-functionalize a node by replacing it with a call to the original. It also replaces the getitem users with the mutated arguments. See replace_users_with_mutated_args and insert_defunctionalized. r(N)replace_users_with_mutated_argsrJrI)rZr r\r8r)rrrrKs z(FixFunctionalizationPass.defunctionalizecCsP||D]\}}||}t|tr|j|n|}||||qdS)a7 Replace all getitem users of the auto-functionalized node with the mutated arguments. :param node: The auto-functionalized node :param mutated_args: The mutated arguments, indexed by getitem index. If the value of an arg is a string, `node.kwargs[arg]` is used. N)rBitemsrdstrr?rHrI)rZr\r8idxrargrrrris    z8FixFunctionalizationPass.replace_users_with_mutated_argscCs2i}|jD]}t|tjr|jd}|||<q|S)z Returns the operator.getitem users of the auto-functionalized node, indexed by the index they are getting. r)rrrCrDr))rZr\rrrlrrrrBs   z&FixFunctionalizationPass.getitem_userscstts Jdd|5jd}|dur%|j|jdntfdd|D}|j||dWddSWddS1sJwYdS) a> Insert a new defunctionalized node into the graph before node. If one of the kwargs is 'out', provide args directly, as node.kwargs cannot be used. See https://github.com/pytorch/pytorch/blob/a00faf440888ffb724bad413f329a49e2b6388e7/torch/_inductor/lowering.py#L351 :param graph: Graph to insert the defunctionalized node into :param node: The auto-functionalized node to defunctionalize :param args: If we cannot use kwargs, specify args directly. If an arg is a string, `node.kwargs[arg]` is used. z%node must be auto-functionalized, is z insteadrN)r?c3s(|]}t|trj|n|VqdSr)rdrkr?)rrmr\rrrs zCFixFunctionalizationPass.insert_defunctionalized..r()rrinserting_beforer) call_functionr?tuple)rZr r\r)functionrrnrrJs     "z0FixFunctionalizationPass.insert_defunctionalizedr)__name__ __module__ __qualname____doc__r time_and_logrreGraphrbrfrrIdictintrkrqrKrirBrJrrrrr sH"  "r )rCcollections.abcrr*torch._higher_order_ops.auto_functionalizer vllm.loggerrvllm.platformsrfx_utilsrvllm_inductor_passrrsr;r rrrrs