o پi>1@spdZddlmZddlmZmZddlZddlZddl m Z er&ddl m Z eGdddZ Gd d d ZdS) a TeaCache: Temporal similarity-based caching for diffusion models. TeaCache accelerates diffusion inference by selectively skipping redundant computation when consecutive diffusion steps are similar enough. This is achieved by tracking the L1 distance between modulated inputs across timesteps. Key concepts: - Modulated input: The input to transformer blocks after timestep conditioning - L1 distance: Measures how different consecutive timesteps are - Threshold: When accumulated L1 distance exceeds threshold, force computation - CFG support: Separate caches for positive and negative branches References: - TeaCache: Accelerating Diffusion Models with Temporal Similarity https://arxiv.org/abs/2411.14324 ) dataclass) TYPE_CHECKINGAnyN) DiTConfig)TeaCacheParamsc@sNeZdZUdZeed<eed<eed<eed<eed<eeed<ded <d S) TeaCacheContextaCommon context extracted for TeaCache skip decision. This context is populated from the forward_batch and forward_context during each denoising step, providing all information needed to make cache decisions. Attributes: current_timestep: Current denoising timestep index (0-indexed). num_inference_steps: Total number of inference steps. do_cfg: Whether classifier-free guidance is enabled. is_cfg_negative: True if currently processing negative CFG branch. teacache_thresh: Threshold for accumulated L1 distance. coefficients: Polynomial coefficients for L1 rescaling. teacache_params: Full TeaCacheParams for model-specific access. current_timestepnum_inference_stepsdo_cfgis_cfg_negativeteacache_thresh coefficientsrteacache_paramsN) __name__ __module__ __qualname____doc__int__annotations__boolfloatlistrr`/home/ubuntu/.local/lib/python3.10/site-packages/sglang/multimodal_gen/runtime/cache/teacache.pyr s   rc @seZdZUdZhdZeeed<eed<dddZ dd d Z d e j d e ed edeeeffddZd e j ded e ed edef ddZdedBfddZde j de j ddfddZdeeefdefddZde j de j fddZdS) TeaCacheMixinar Mixin class providing TeaCache optimization functionality. TeaCache accelerates diffusion inference by selectively skipping redundant computation when consecutive diffusion steps are similar enough. This mixin should be inherited by DiT model classes that want to support TeaCache optimization. It provides: - State management for tracking L1 distances - CFG-aware caching (separate caches for positive/negative branches) - Decision logic for when to compute vs. use cache Example usage in a DiT model: class MyDiT(TeaCacheMixin, BaseDiT): def __init__(self, config, **kwargs): super().__init__(config, **kwargs) self._init_teacache_state() def forward(self, hidden_states, timestep, ...): ctx = self._get_teacache_context() if ctx is not None: # Compute modulated input (model-specific, e.g., after timestep embedding) modulated_input = self._compute_modulated_input(hidden_states, timestep) is_boundary = (ctx.current_timestep == 0 or ctx.current_timestep >= ctx.num_inference_steps - 1) should_calc = self._compute_teacache_decision( modulated_inp=modulated_input, is_boundary_step=is_boundary, coefficients=ctx.coefficients, teacache_thresh=ctx.teacache_thresh, ) if not should_calc: # Use cached residual (must implement retrieve_cached_states) return self.retrieve_cached_states(hidden_states) # Normal forward pass... output = self._transformer_forward(hidden_states, timestep, ...) # Cache states for next step if ctx is not None: self.maybe_cache_states(output, hidden_states) return output Subclass implementation notes: - `_compute_modulated_input()`: Model-specific method to compute the input after timestep conditioning (used for L1 distance calculation) - `retrieve_cached_states()`: Must be overridden to return cached output - `maybe_cache_states()`: Override to store states for cache retrieval Attributes: cnt: Counter for tracking steps. enable_teacache: Whether TeaCache is enabled. previous_modulated_input: Cached modulated input for positive branch. previous_residual: Cached residual for positive branch. accumulated_rel_l1_distance: Accumulated L1 distance for positive branch. is_cfg_negative: Whether currently processing negative CFG branch. _supports_cfg_cache: Whether this model supports CFG cache separation. CFG-specific attributes (only when _supports_cfg_cache is True): previous_modulated_input_negative: Cached input for negative branch. previous_residual_negative: Cached residual for negative branch. accumulated_rel_l1_distance_negative: L1 distance for negative branch. >wanzimagehunyuan_CFG_SUPPORTED_PREFIXESconfigreturnNcCsXd|_d|_|jj|jv|_d|_d|_d|_ d|_ |jr*d|_ d|_ d|_ dSdS)z:Initialize TeaCache state. Call this in subclass __init__.rTNF)cntenable_teacacherprefixlowerr_supports_cfg_cacheprevious_modulated_inputprevious_residualaccumulated_rel_l1_distancer !previous_modulated_input_negativeprevious_residual_negative$accumulated_rel_l1_distance_negativeselfrrr_init_teacache_states z"TeaCacheMixin._init_teacache_statecCsDd|_d|_d|_d|_d|_d|_|jr d|_d|_d|_ dSdS)z>Reset all TeaCache state at the start of each generation task.rNr!FT) r"r'r(r)r r#r&r*r+r,r-rrrreset_teacache_states z"TeaCacheMixin.reset_teacache_state modulated_inpr r c Cs|jr|jn|j}|durdS||}||}t|}|jr.|j n|j }|||}||kr=dS|dfS)ap Compute L1 distance and decide whether to calculate or use cache. Args: modulated_inp: Current timestep's modulated input. coefficients: Polynomial coefficients for L1 rescaling. teacache_thresh: Threshold for cache decision. Returns: Tuple of (new_accumulated_distance, should_calc). Nr!TF) r r*r'absmeancpuitemnppoly1dr,r)) r.r1r r prev_modulated_inpdiffrel_l1 rescale_funcr)rrr_compute_l1_and_decides"   z$TeaCacheMixin._compute_l1_and_decideis_boundary_stepcCs`|jsdS|r d\}}n |j|||d\}}|js#||_||_|S|jr.||_||_|S)a Compute cache decision for TeaCache. Args: modulated_inp: Current timestep's modulated input. is_boundary_step: True for boundary timesteps that always compute. coefficients: Polynomial coefficients for L1 rescaling. teacache_thresh: Threshold for cache decision. Returns: True if forward computation is needed, False to use cache. Tr2)r1r r ) r#r=r cloner'r)r&r*r,)r.r1r>r r new_accum should_calcrrr_compute_teacache_decisions"    z(TeaCacheMixin._compute_teacache_decisionc Csddlm}|}|j}|dus|jr|jdurdS|j}|j}|j}|j}|j}|dkr4|js4| t |||||j |j |dS)z Check TeaCache preconditions and extract common context. Returns: TeaCacheContext if TeaCache is enabled and properly configured, None if should skip TeaCache logic entirely. r)get_forward_contextN)rr r r r r r) 6sglang.multimodal_gen.runtime.managers.forward_contextrC forward_batchr#rrr do_classifier_free_guidancer r0rr r ) r.rCforward_contextrErrr r r rrr_get_teacache_contexts0  z#TeaCacheMixin._get_teacache_context hidden_statesoriginal_hidden_statescCdS)zACache states for later retrieval. Override in subclass if needed.Nr)r.rIrJrrrmaybe_cache_states0sz TeaCacheMixin.maybe_cache_stateskwargscKrK)z4Check if forward can be skipped using cached states.Fr)r.rMrrr%should_skip_forward_for_cached_states6sz3TeaCacheMixin.should_skip_forward_for_cached_statescCstd)z8Retrieve cached states. Must be implemented by subclass.z)retrieve_cached_states is not implemented)NotImplementedError)r.rIrrrretrieve_cached_states:sz$TeaCacheMixin.retrieve_cached_states)r N)rrrrrsetstrrrr/r0torchTensorrrtuplerr=rBrrHrLdictrrNrPrrrrr;sH E    / )- r)r dataclassesrtypingrrnumpyr7rS$sglang.multimodal_gen.configs.modelsr-sglang.multimodal_gen.configs.sample.teacacherrrrrrrs