o c۷iH @spddlmZmZmZddlmZddlmZddlZddl m Z ddlm Z m Z m Z mZddlmZddlmZddlmZddlmZmZmZGd d d eZGd d d eZGd ddeZe jdede jde defddZeGdddeZeGdddeZ GdddZ!e jde dee e ffddZ"Gddde!Z#eGd d!d!eZ$Gd"d#d#e!Z%dS)$) NamedTupleTupleOptional) dataclass)IntEnumN)Int32Float32Boolean const_expr) FastDivmod)PipelineStateWAdvance) ArgumentsBase ParamsBasemlir_namedtuplec@seZdZdZdZdZdS)RasterOrderOptionrN)__name__ __module__ __qualname__AlongMAlongN HeuristicrrJ/home/ubuntu/vllm_env/lib/python3.10/site-packages/quack/tile_scheduler.pyrsrc@seZdZdZdZdS) RasterOrderrrN)rrrrrrrrrrsrc@seZdZdZdZdZdZdS)PersistenceModerrrN)rrrNONESTATICDYNAMICCLCrrrrrs rraster_order_optionproblem_shape_ncluster_mn group_sizereturncCsX|tjkrtjntj}|tjkr*t|d|}t|d|}||kr'tjntj}|SNrr)rrrrrcuteround_up)r"r#r$ raster_orderproblem_blocks_mproblem_blocks_nrrrget_raster_order_from_option#s  r,c@s^eZdZUeed<ejZej eed<edZ eed<dZ e e jed<dZe e jed<dS)TileSchedulerOptionsmax_active_clustersr)max_swizzle_sizeNtile_count_semaphorebatch_idx_permute)rrrr__annotations__rrr)cutlass Constexprr0r1rr'Pointerr2Tensorrrrrr-6s r-c@sveZdZUejed<ejeed<e ed<ejejed<dZ e ej ed<dZ e ejed<ejZejeed<dS) TileSchedulerArgumentsproblem_shape_ntile_mnlr)r$cluster_shape_mnkNr1r2persistence_mode)rrrr'Shaper3r4r5rrr1rr6r2r7rrr;rrrrr8?s r8c@seZdZeGdddeZddddedededed eej d ee j j d e d efd dZeddddedefddZeejdAdddd eej ddfddZeejdddd edeeeefdeeeeffddZeej   dBdddd ed eej d ee j j deeBddf ddZedddd ededeeeeffddZejddddedeeeffd d!Zejdddd"d#ed$ed%edeeeffd&d'Zej  dCdddd"d(ed)eed*eed%ede jjf d+d,Zejdddde jjfd-d.Zdddde jjfd/d0Z ejddddeeeeefBfd1d2Z!ejdddd3e jjfd4d5Z"ej dDd6ddd7deeBd8e#fd9d:Z$d;d<Z%d=d>Z&d?d@Z'dS)E TileSchedulerc@seZdZUejed<eed<eed<eed<eed<eed<eed<e ej ed<e ej ed <e j ejed <e j eed <eejd d d deddfddZd S)zTileScheduler.Paramsproblem_shape_ncluster_mnlr)num_clusters_per_problem_fddnum_groups_regulargroup_size_fddgroup_size_tail_fddnum_clusters_in_group_fddr1r2cluster_shape_mnr;Nlocipargsr%c Cs>|jddks Jttj|jddgd}tj|jddgd}t||}||jdf}t|}t|j||j }|t j krC|dn|d} |t j krP|dn|d} t |j | } | | } | | } | | }t|j tjkru|jdusuJt||t|| t| t| dkr| ndt|t|j tjkr|jnd|j||j SNrrrmode)r:r r'selectr9ceil_divsizer,r)r$rrminr;rr r1r=Paramsr r2)rHrFrGrDproblem_shape_ntile_mnr#r>num_clusters_per_problemr) ncluster_fast ncluster_slowr$group_size_tailr@num_clusters_in_grouprrrcreateYsP      zTileScheduler.Params.create)rrrr'r<r3rr rrr6r7r4r5r staticmethodjitr8rWrrrrrPKs "rPNrEcurrent_work_idxnum_tiles_executedcurrent_batch_idxnum_work_idx_before_cur_batch sched_smemscheduler_pipelinepipeline_stateparamsc  C@||_||_||_||_||_||_||_||_| |_| |_ dSN _current_work_idxr[_current_batch_idx_num_work_idx_before_cur_batch _sched_smem_scheduler_pipeline_pipeline_statera_loc_ip selfrZr[r\r]r^r_r`rarFrGrrr__init__ zTileScheduler.__init__rHr%cCtjj|||dSNrE)r=rPrWrHrFrGrrrto_underlying_argumentsz%TileScheduler.to_underlying_argumentscCsvtj|dgddks J|djd}tjtj|dWdn1s*wYtjtjdS)NrrJ Nrr/r)r'rNiteratorarch elect_one mbarrier_initmbarrier_init_fence sync_warp)r^rFrG clc_mbar_ptrrrr_init_clc_mbarriers  z TileScheduler._init_clc_mbarrier cluster_idxcCsNt|jtjtjfvrt|d}t|d}||fSt|d}d}||fSNrrr r;rrr!rrarrFrGrZ batch_idxrrr_cluster_idx_to_work_idx_batchs   z,TileScheduler._cluster_idx_to_work_idx_batchFis_scheduler_warpc Cstj|tj||d\}}d}t|jtjtj tj fvr3|dus#J|dus)Jttj |dgd}t|jtj krE|rEtj |||dt|t dt dt d||t|t dt dt d|||d S)zGis_scheduler_warp should only be true for one warp in the whole clusterrErNrrJ)r=rr'ryrr r;rrr r!rNrrr rar^r_rrFrGrZ_stagesrrrrWs6    zTileScheduler.creater.c Cst|jtjtjfvr#|jdt|jdd|jd|jdfStj|j||dt|j}tj|j||d}||}t ||}||}g|j|RS)NrrrrE) r r;rrr!rDr'rNr>r4rO) rar.rFrGnum_ctas_in_problemnum_ctas_per_clusternum_ctas_per_wavenum_persistent_ctasnum_persistent_clustersrrrget_grid_shapes  zTileScheduler.get_grid_shapecluster_id_in_problemc Cs|j}t||j\}}tdtd}}||jkr"t||j\}}nt||j\}}|ddkrF|jtj kr;|j dn|j d} | d|}||jj |} | |} } |jtj kr^|| } } | | fSNrrr) radivmodrCrr@rArBr)rrr>divisorr) rnrrFrGragroup_id id_in_groupcid_fast_in_groupcid_slowrTcid_fastcid_mcid_nrrr _swizzle_ctas"        zTileScheduler._swizzle_ctablock_zero_onlyrFrGrrrc Cs\t|r tdtdf}ntj}||jjd|d}||jjd|d}||fSr&)r rr'ryblock_in_cluster_idxrarD) rnrrrrFrGbidx_in_clusterpid_mpid_nrrr_cluster_id_to_cta_ids  z#TileScheduler._cluster_id_to_cta_idwork_idxbidzis_validcCs6|j}t|dur4t|jtjkr|jdk}nt|jtjkr,|t|j ddk}n|t|j k}t dt dt d}} } |rt|jtjtjfvrZ|} tj \} } } nt ||j\} } t|durj|} |j| ||d\}}|j|||||d\}} t|jdur| n|j| } || d| f}tj||SNrrrEr)rar r;rrr[r!r'rNr>rry block_idxrr?rrr2r4utils WorkTileInfornrrrrrFrGrarrrrrbidz_rrtile_coord_mnklrrr_delinearize_work_idx)s2      z#TileScheduler._delinearize_work_idxc sj}tdtdtdtdf\}}}}t|jtjkrnOjj fddt dD\}}}}tt |j dkrCt jt jt jjj Wdn1s_wYj t|}||d|f} tj| t|S)NrFcsg|] }j|jjfqSr)rhrjindex).0irnrr [sz2TileScheduler.get_current_work..r)rarr r r;rrri consumer_waitrjranger'rNrDryfence_view_async_sharedr}rzconsumer_releaseadvancer4rr) rnrFrGrarrrr is_valid_i32rrrrget_current_workQs$$      zTileScheduler.get_current_workcCs|j|j||dSrr)rre)rnrFrGrrrinitial_work_tile_infoksz$TileScheduler.initial_work_tile_infocCs|j}ttjd}t|jtjkr|j |St|jtj krUtd}tj dkrNt|j ddurE|t t|j d|j}n |t d|j}tj|dSt|jtjkr|jd|jjfjd}|jdjd}tjtjtjj|d||d t j||||d Wdn1swYtjtjj||jj||d tjj|||d \}} } } tj||j d| |j d| f} t!|j"|| ||d \} } | | t#| fStdS) z+should only be called by the scheduler warprrNrrrwr/rE)$rarr'rygrid_dimr r;rrrer lane_idxr>ratomic_inc_i32rNr1atomic_add_i32 shuffle_syncr!rhrjrrxr}rzmbarrier_arrive_and_expect_txissue_clc_query_nomulticast mbarrier_waitphase clc_responserrDtyperr )rnrFrGrarnext_work_linear_idxclc_response_ptr mbarrier_addrbidxbidyrvalidrrrrr_fetch_next_work_idxrsJ        z"TileScheduler._fetch_next_work_idxwork_tile_infoc CsV|j}t|jdurt|jj|jj|jj|jjdA}|j ||j d|j d|j dt |j g}tj}|t|jkr|jj}tt|jdkrhtdD] } || |j| |f<qS|j |jdS|} | |jd} | |jd} |j |j} tj| d| tj|d| |d| |d|d|jd|fj| | ddSdSdS)Nrrrrrr)smem_ptrmbar_ptrpeer_cta_rank_in_cluster)rar rhr rjrcountrrriproducer_acquiretile_idxr is_valid_tiler'ryrrNrDr4range_constexprproducer_commitproducer_get_barrierrrstore_shared_remote_x4rx)rnrrFrGrapipeline_state_producer sched_datar pipeline_idxrrrbidy_in_clusterrrrrwrite_work_tile_to_smemsJ      z%TileScheduler.write_work_tile_to_smemr) advance_countrFrGrc Cs|j}|jt|7_t|jduo|dkr|j|dt|jtjtj fvrJ|rH|j ||d|_ |j |j d||d}|j |||ddSdSt|jtjkru|rw|j ||d\|_ }}|j |j ||d||d}|j |||ddSdSdS)zis_scheduler_warp should only be true for one warp in the whole cluster. Moreover, we assume that only block zero in the cluster is calling this function. If calling with is_scheduler_warp = True, advance_count must be 1. NrrETr)rar[rr rj advance_itersr;rrr rrerrr!) rnrrrFrGrarbatchrrrradvance_to_next_works*  z"TileScheduler.advance_to_next_workcCsDt|jdur t|jj|jj|jj|jjdA}|j|dSdS)Nr) r rir rjrrrr producer_tail)rnrrrrrs zTileScheduler.producer_tailcCs\gg}|_|j|j|j|j|j|j|j|jfD]}t |}||7}|j t |q|Src) _values_posrer[rfrgrhrirjrar4extract_mlir_valuesappendlen)rnvaluesobj obj_valuesrrr__extract_mlir_values__s  z%TileScheduler.__extract_mlir_values__c Csvg}t|j|j|j|j|j|j|j|jg|j D]\}}| t ||d|||d}q|j t|d|jiS)NrF)ziprer[rfrgrhrirjrarrr4new_from_mlir_values __class__tuplerk)rnrobj_listrn_itemsrrr__new_from_mlir_values__ s   z&TileScheduler.__new_from_mlir_values__rcNNFNN)F)(rrrrrrPrrr'r7r4pipeline PipelineAsyncr rorXr8rtrYrrrboolr rWrrrrrrrrrrintrrrrrrrrr=JsG  &    &        '&/ - ! r=idxcCs<ttd|ddd}|||dd}||fS)z Convert a triangular index to 2D coordinates. This is used to convert the linear index to 2D coordinates for triangular matrices. r@?r)rceilsqrt)rrowcolrrrtriangular_idx_to_coord!s rc@s eZdZdZeGdddeZeddddedefdd Z ee j   ddddd ed e e j d e ejjdeeBddf ddZedddd ededeeeeffddZe j ddddedeeeffddZe j  dd ddddede ede ededejjf ddZdS) TriangularTileSchedulerz[We assume the tile size per cluster is square (e.g., 128 x 256 per CTA, with cluster 2 x 1)c@seZdZUejed<eed<eed<eed<eed<eed<eed<eed<e ej ed <e j ejed <e j e ed <eejd d d deddfddZd S)zTriangularTileScheduler.Paramsr>r?group_size_inv_f32r@rArBgroup_size_mul_group_size_fdd"group_size_tail_mul_group_size_fddr1rDr;NrErHr%c Cs|jddks Jttj|jddgd}tj|jddgd}t||}||jdf}|d}||dd}t|j|} || } || } t|jt j krV|j dusVJt |t|td| | t| t| dkrl| ndt| | t| dkrz| nd| t|jt j kr|j nd||j S)NrrrrJg?)r:r r'rLr9rMrOr$r;rr r1rrPr r) rHrFrGrDrQr#r> cluster_mrRr$rUr@rrrrW=s:    z%TriangularTileScheduler.Params.create)rrrr'r<r3r rrrr6r4r5rrXrYr8rWrrrrrP/s( rPNrErHr%cCrqrr)rrPrWrsrrrrtbruz/TriangularTileScheduler.to_underlying_argumentsFrar^r_rc Cstj|tj||d\}}d}t|jtjtj tj fvr3|dus#J|dus)Jttj |dgd}t|jtj krE|rEtj |||dt |tdtdtd||t|tdtdtd|||d SNrErrrJ)r=rr'ryrr r;rrr r!rNrrrr rrrrrWf6    zTriangularTileScheduler.creater.c Cs|jjdf}tddt||jD|jdf}t|jtj tj fvr&|St j |||d}t j |j||d}||}t ||} | |} g|j| RS)Nrcss|] \}}||VqdSrcr)rxyrrr sz9TriangularTileScheduler.get_grid_shape..rrE)r?rrrrDr>r r;rrr!r'rNr4rO) rar.rFrGclusters num_ctas_mnlrrrrrrrrrs  z&TriangularTileScheduler.get_grid_shapercCs|j}|jj}ttd|dd|jd}||}|||dd}||jkr/|n|jj} t dt d} } ||jkrJt ||j \} } nt ||j \} } t dt d} } || ||krjt | \} } n||jkrxt | |j\} } nt | |j\} } || }| || }||fS)Nrrrrr)rarArrrrrr@rBrrrrr)rnrrFrGrar$r cid_m_startrgroup_size_actual group_colgroup_remaindercid_m_in_groupcid_n_in_grouprrrrrrs:    z$TriangularTileScheduler._swizzle_ctarrrrrcCs|j}t|dur"t|jtjkr|jdk}n ||jj|jdk}t dt dt d}} } |rtt|jtjtj fvrH|} t j \} } } n t||j\} } t | } t|dur\|} |j| ||d\}}|j|||||d\}} | } || d| f}tj||Sr)rar r;rrr[r?rr>rr!r'ryrrrrr4rrrrrrrs4      z-TriangularTileScheduler._delinearize_work_idxrr)rrr__doc__rrrPrXr8rtr'rYrr7r4rrrr rWrrrrrrrrrrrr,sx2 %    & rc@seZdZUejed<eed<ejed<ej e ed<eed<ej ejed<ej ejed<dZ e ej ed <ejZej eed <dS) VarlenMTileSchedulerArgumentsr9total_m cu_seqlens_mr)r$ tile_shape_mnr:Nr1r;)rrrr'r<r3rr7r4r5rr1rr6rrr;rrrrrs  rc@seZdZeGdddeZddddedededed eej d ee j j d e d efd dZeddddedefddZeejdddd edeeeefdeeeeffddZeej   d-dddd ed eej d ee j j deeBddf ddZedddd ededeeeeffddZejddddededeeeffdd Zejd!ed"ed#e jedefd$d%Zej  d.dddd&d'ed(eed)eed*ede jjf d+d,ZdS)/VarlenMTileSchedulerc@seZdZUejed<eed<ejed<ej e ed<eed<e e ed<e e ed<e ed<ej ejed <e ej ed <ej ejed <ej eed <eejd d ddeddfddZd S)zVarlenMTileScheduler.Paramsr>rrr)r$rArBrCrr1rDr;NrErHr%c Cs|jddks Jttj|jddgd}tj|jddgd}dt|d|df}||jdf}t|jtjkr=t jnt j }||t jkrIdnd}||t jkrTdnd} t|durht |j |} || } n|j d} } d} t| durz| | } t|j tjkr|jdusJt||j|j|| |durt| nd| durt| dkr| ndnd| durt| nd|jt|j tjkr|jnd||j SrI)r:r r'rLr9rMr)rrrrrOr$r;rr r1rrPrrr r) rHrFrGrDrQr#r>r)rSrTr$rUrVrrrrWs^      z"VarlenMTileScheduler.Params.create)rrrr'r<r3rr7r4r5rrr r6rrXrYr8rWrrrrrPs*    rPNrErZr[r\r]r^r_r`rac  CrbrcrdrmrrrroMrpzVarlenMTileScheduler.__init__rHr%cCrqrr)rrPrWrsrrrrtfruz,VarlenMTileScheduler.to_underlying_argumentsrcCs<t|jtjtjfvrt|d}nt|d}d}||fSrrrrrrrjs  z3VarlenMTileScheduler._cluster_idx_to_work_idx_batchFrc Cstj|tj||d\}}d}t|jtjtj tj fvr3|dus#J|dus)Jttj |dgd}t|jtj krE|rEt j |||dt|tdtdtd||t|tdtdtd|||d Sr)rrr'ryrr r;rrr r!rNr=rrr rrrrrWvrzVarlenMTileScheduler.creater.c Cs|jd|jd}|jd}|j||d|}||jd}t|jtjtjfvr9|jd||jddfSt ||}g|j|RSr) rrDr>rr r;rrr!r4rO) rar.rFrG block_size num_batchtotal_clusters_m_maxtotal_clusters_maxrrrrrs  z#VarlenMTileScheduler.get_grid_shapernum_clusters_mcCsl|j}t|jdurt||j\}}|jj}n|jtjksJ|j|}||}|||}t dt d} } t|j duoA|j durd||j d} |d|| kr[t||j \} } n't||j \} } n|jtj kslJt|j|||j} || } || | } |ddkr|jtj kr|j dn|} | d| } ||j| }|| }}|jtjkr| |}}||fS)Nrrr)rar rCrrr)rrr$rrArBr>rr4rO)rnrrrFrGrarrrVrr num_clustersrrTrrrrrrrs>           z!VarlenMTileScheduler._swizzle_ctalane bidb_startrc Cst|jjd}||}td}||kr|jj|}tjj|dd}||}||kr6|tjjdkr6t||StdS)Nrrr)offset) rar>rrr'ryshuffle_sync_down WARP_SIZErM) rnr r!rrr cur_cu_seqlennext_cu_seqlenseqlenrrr_get_num_m_blockss   z&VarlenMTileScheduler._get_num_m_blocksrrr is_valid_rcCs|dusJ|j}tj}|jjd} |jd|jd} |j} |} |j} t dt dt d}}}d}t |dur>|}|r| | kr|j || | d}||jd}t ||}tj|tjjd}| |7} | | krt| tjjd7} | | krt | } | d} | | ksDnt | } | | k}t |jtjkr||jdkM}t dt d}}|j}|r| |}tjtj||| k}| |7} |dkrdntj||d}tj||}||}| |}|j||||d\}}|j|||||d\}}||d| f}| |_||_tj ||S)NrrT)r!rrrEr)rar'ryrr>rrDrfrgrr r(rwarp_prefix_sumrr$r;rrr[popcvote_ballot_syncrrr4r)rnrrr)rrFrGrarrrr next_tile_idxproblems_end_tilernum_clusters_cumulativeclusters_in_problemsrrrrr]problems_start_tilebatch_idx_in_problemsnum_clusters_prev_lanerrrrrrrrsv         z*VarlenMTileScheduler._delinearize_work_idxrr) rrrrrrPrrr'r7r4rrr rorXr8rtrYrrrr rWrrr5rr(rrrrrrrrsR     %    (  r)&typingrrr dataclassesrenumrr4 cutlass.cuter'rrr r quack.utilsrquack.fast_mathr quack.pipeliner quack.cute_dsl_utilsr rrrrrrYr<r,r-r8r=rrrrrrrrsL        Z L