o -i^9@sddlZddlmZmZddlmZddlZddlmm Z ddl m Z ddlm Z ddlmZddlmZddlmZmZddlmZdd lmZdd lmZdd lmZd d lmZd dlm Z m!Z!m"Z"d dl#m$Z$d dl%m&Z&m'Z'ee(Z)dedeej*fdedej*ffddZ+GdddZ,Gddde,Z-Gddde,Z.e/Z0Gddde,Z1Gddde,Z2Gd d!d!e'Z3dS)"N)CallableSequence)Any)PatternMatcherPass) VllmConfig)Range) get_tp_group tensor_model_parallel_all_reduce)$get_tensor_model_parallel_world_size) init_logger)kFp8StaticTensorSym)current_platform)enable_fake_mode)MatcherFusedAddRMSNormMatcherQuantFP8MatcherRMSNorm)NoOpEliminationPass)VllmInductorPassVllmPatternMatcherPassfn.returncs&tdtdtjffdd }|S)Nargsrcs |dS)Nr)rrrb/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/vllm/compilation/sequence_parallelism.pywrapper"s z&get_first_out_wrapper..wrapper) functoolswrapsrtorchTensor)rrrrrget_first_out_wrappersr!c@sreZdZdZdedejdedBddfddZd ej dej fd d Z d ej dej fd d Z d ej dej fddZ dS)_SequenceParallelPatternHelperz)Helper for sequence parallelism patterns.epsilondtypedeviceNrcCs&||_||_||_t|_t|_dSN)r#r$r%rtp_groupr tp_sizeselfr#r$r%rrr__init__,s  z'_SequenceParallelPatternHelper.__init__xcCst|Sr&)r r*r,rrr _all_reduce8sz*_SequenceParallelPatternHelper._all_reducecCtjjjj|d|j|jjdSNr)dim world_size group_name)ropsvllmreduce_scatterdefaultr(r' unique_namer-rrr_reduce_scatter; z._SequenceParallelPatternHelper._reduce_scattercCr/r0)rr4r5 all_gatherr7r(r'r8r-rrr _all_gather@r:z*_SequenceParallelPatternHelper._all_gather) __name__ __module__ __qualname____doc__floatrr$strr+r r.r9r<rrrrr")s  r"cZeZdZdedejdedBddffdd Zdeej fdd Z d e ddfd d Z Z S) FirstAllReduceRMSNormPatternr#r$r%Nrct|||t||_dSr&)superr+rrmsnorm_matcherr) __class__rrr+Gz%FirstAllReduceRMSNormPattern.__init__cCs6tjgd|j|jd}tjdg|j|jd}||gS)Nrr%r$rMremptyr%r$)r*inputarg3_1rrr get_inputsKsz'FirstAllReduceRMSNormPattern.get_inputspm_passcspdtjdtjdttjtjfffdd }dtjdtjdttjtjfffdd }t||tj|dS)NrQrRrcs|}||}||fSr&r.rG)rQrR all_reducermsnormr*rrpatternRs  z6FirstAllReduceRMSNormPattern.register..patterncs(|}||}|}||fSr&)r9rGr<)rQrRr6rWr;rXrr replacement[s   z:FirstAllReduceRMSNormPattern.register..replacementrr tuplepmregister_replacementrSfwd_onlyr*rTrYrZrrXrregisterQs"  z%FirstAllReduceRMSNormPattern.registerr=r>r?rArr$rBr+listr rSrra __classcell__rrrHrrDFs$rDcrC) MiddleAllReduceRMSNormPatternr#r$r%NrcrEr&)rFr+rrGr)rHrrr+krJz&MiddleAllReduceRMSNormPattern.__init__cCsRtjddg|j|jd}tjddg|j|jd}tjddg|j|jd}|||gS)NrMrNrO)r*mm_1residualrms_norm_weightsrrrrSosz(MiddleAllReduceRMSNormPattern.get_inputsrTc sdtjdtjdtjdttjtjfffdd }dtjdtjdtjdttjtjfffdd }t||tj|tt|t|tj|dS) Nrgrfrhrcs(|}|||}|d|dfS)NrrrU)rgrfrhrVrWrXrrrY|s z7MiddleAllReduceRMSNormPattern.register..patterncsH|}|d|ddf}|||}|d}||dfS)Nr.r)r9sizerGr<)rgrfrhr6rWr;rXrrrZs  z;MiddleAllReduceRMSNormPattern.register..replacementrr r\r]r^rSr_r!r`rrXrra{s8 z&MiddleAllReduceRMSNormPattern.registerrbrrrHrrejs$ recrC) %FirstAllReduceRMSNormStaticFP8Patternr#r$r%Nrc(t|||t||_tt|_dSr&)rFr+rrGrr quant_matcherr)rHrrr+s z.FirstAllReduceRMSNormStaticFP8Pattern.__init__cCsLtjgd|j|jd}tjdg|j|jd}tjd|jtjd}|||gS)NrKrNrMg?)rzerosr%r$rPtensorfloat32)r*rQweightscalerrrrSs z0FirstAllReduceRMSNormStaticFP8Pattern.get_inputsrTc s|dtjdtjdtjdttjtjfffdd }dtjdtjdtjdttjtjfffdd }t||tj|dS) NrQrqrrrcs.|}||}||\}}||fSr&r.rGrm)rQrqrrrVrmsquant_rXrrrYs  z?FirstAllReduceRMSNormStaticFP8Pattern.register..patterncs8|}||}||\}}|}||fSr&)r9rGrmr<)rQrqrrr6rtrurvr;rXrrrZs   zCFirstAllReduceRMSNormStaticFP8Pattern.register..replacementr[r`rrXrras*  z.FirstAllReduceRMSNormStaticFP8Pattern.registerrbrrrHrrks rkcrC) &MiddleAllReduceRMSNormStaticFP8Patternr#r$r%Nrcrlr&)rFr+rrGrr rmr)rHrrr+s z/MiddleAllReduceRMSNormStaticFP8Pattern.__init__cCsltjddg|j|jd}tjddg|j|jd}tjddg|j|jd}tjddg|jtjd}||||gS)NrMrNr)rrPr%r$rp)r*rfrgrhrrrrrrSs  z1MiddleAllReduceRMSNormStaticFP8Pattern.get_inputsrTc sdtjdtjdtjdtjdttjtjff fdd }dtjdtjdtjdtjdttjtjff fdd }t||tj|tt|t|tj|dS) Nrgrfrhrrrc s4|}|||\}}||\}}||fSr&rs) rgrfrhrrrVrt residual_outrurvrXrrrYs z@MiddleAllReduceRMSNormStaticFP8Pattern.register..patternc sT|}|d|ddf}|||\}}||\}}|} | |fS)Nr.)r9rirGrmr<) rgrfrhrrr6rtrxrurvr;rXrrrZs  zDMiddleAllReduceRMSNormStaticFP8Pattern.register..replacementrjr`rrXrras@ z/MiddleAllReduceRMSNormStaticFP8Pattern.registerrbrrrHrrws$rwcsZeZdZdZededdffdd Zdedefdd Z e j d e j ddfd d ZZS) SequenceParallelismPassa This pass enables sequence parallelism for models. It identifies patterns where an AllReduce operation is followed by an RMSNorm (or RMSNorm and then Quantization) operation. These patterns are replaced with a ReduceScatter operation, followed by a local RMSNorm/Quantization, and then an AllGather operation. The general transformation is: Input -> AllReduce -> RMSNorm -> Output becomes Input -> ReduceScatter -> RMSNorm -> AllGather -> Output While this pass itself does not directly yield performance improvements, it lays the groundwork for subsequent fusion passes, such as GEMM + ReduceScatter and AllGather + GEMM fusions. These fusions can significantly reduce communication overhead and improve overall model performance. This pass splits up the residual tensor across TP ranks and hence divides its size. Because the pattern matcher starts at the end of the graph, the replacement contains a slice that temporarily conforms the input residual to the correct size. After all patterns have been matched, we use a NoOpEliminationPass to clean up what have now become no-op slices. Note that an older version of the pass did not need this as it operated only on custom rms_norm and fused_rms_norm_add custom ops which did not complain about mismatched shapes during replacement. So this approach has the same assumption that correctness is only maintained if all rms_norm operations are split across ranks. Correctness-wise, this is approach strictly better than before - before, the graph was incorrect semantically and shape-wise during the pass. With this approach there's only semantic incorrectness during the pass. Both approaches restore a correct graph once all patterns are matched. configrNcst|t||_|jd|jj|j_tdd|_dD]2}t||j|j  |jt ||j|j  |jt ||j|j  |jt ||j|j  |jq|||jdS)N.sequence_parallelism_pass) pass_name)gh㈵>gư>)rFr+r noop_cleanupr}rpatternsrk model_dtyper%rarwrDre dump_patterns)r*rzr#rHrrr+4s0          z SequenceParallelismPass.__init__ compile_rangecCs4|jjr|jjr dSt}|o|j|dk}|S)NTr)compilation_config splitting_opsuse_inductor_graph_partitionr is_single_sizeend)r*rr(resultrrris_applicable_for_rangeUs z/SequenceParallelismPass.is_applicable_for_rangegraphcCs*|j||_td|j||dS)NzReplaced %s patterns)rapply matched_countloggerdebugr~)r*rrrr__call__nsz SequenceParallelismPass.__call__)r=r>r?r@rrr+rboolrr time_and_logfxGraphrrdrrrHrrys$ ry)4rcollections.abcrrtypingrrtorch._inductor.pattern_matcher _inductorpattern_matcherr]torch.fxrr vllm.configrvllm.config.utilsrvllm.distributedrr vllm.distributed.parallel_stater vllm.loggerr 9vllm.model_executor.layers.quantization.utils.quant_utilsr vllm.platformsr inductor_passr matcher_utilsrrrnoop_eliminationrvllm_inductor_passrrr=rr r!r"rDre fp8_dtype FP8_DTYPErkrwryrrrrs>              $7->