o ۷i@sddlZddlZddlmZmZmZmZmZddlm Z m Z ddl m Z ddl mZddl mZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZeeZ    d4dee eej j!e fdeeeeefdeedeeeeeeeefdeedee ej j!e eff ddZ"de eBde#fddZ$de deeej j!ee%ffddZ&de eBdee%ej j!Bdeeej j!effd d!Z'd"ej j!fd#d$Z(dee eej j!ffd%d&Z)dee*ee%ffd'd(Z+de e%BeBdefd)d*Z, d5d+ee*d,ee*d-ee*de-e*fd.d/Z.    0d6d+eee*d,eee*d-ee*d1ee%de-e*f d2d3Z/dS)7N)AnyTupleListUnionOptional)DiffusionPipeline ModelMixin) CacheType) BlockAdapter)BlockAdapterRegister) CachedAdapter)BasicCacheConfig) DBCacheConfig) DBPruneConfig)CalibratorConfig)ParamsModifier)ParallelismConfig)enable_parallelism) init_loggerpipe_or_adapter cache_configcalibrator_configparams_modifiersparallelism_configreturnc KsB|dur|durtdt}ntdi}|dd}dur)|tjkr)|S|dd|dd|dd|dd|d d|d d|d d|d d|d dd }dd|D}|r}td|durv|jd-i|nt d-i|}|dur||d<|dur|dddus|dddurtdddl m } | |d|d|dd|ddd}|dur||d<|dur||d<|durt |t ttjjtfrtj|fi|}ntdt|dtd |jjd!|durt |tsJd"d#|jvrt||jd#<|jd#|_|jd$d} dur.t|| |jd$<g} t |t ratj||dud%} | durSt|d&sNJd'|j g} n!tj!| d(d)} t"| j } nt#|sntj!|d(d)}t"|j } t$| d*krtd+|St$| dkrtd,t%| D] \} }t&||| | <q|S).u) The `enable_cache` function serves as a unified caching interface designed to optimize the performance of diffusion transformer models by implementing an intelligent caching mechanism known as `DBCache`. This API is engineered to be compatible with nearly `all` diffusion transformer architectures that feature transformer blocks adhering to standard input-output patterns, eliminating the need for architecture-specific modifications. By strategically caching intermediate outputs of transformer blocks during the diffusion process, `DBCache` significantly reduces redundant computations without compromising generation quality. The caching mechanism works by tracking residual differences between consecutive steps, allowing the model to reuse previously computed features when these differences fall below a configurable threshold. This approach maintains a balance between computational efficiency and output precision. The default configuration (`F8B0, 8 warmup steps, unlimited cached steps`) is carefully tuned to provide an optimal tradeoff for most common use cases. The "F8B0" configuration indicates that the first 8 transformer blocks are used to compute stable feature differences, while no final blocks are employed for additional fusion. The warmup phase ensures the model establishes sufficient feature representation before caching begins, preventing potential degradation of output quality. This function seamlessly integrates with both standard diffusion pipelines and custom block adapters, making it versatile for various deployment scenarios—from research prototyping to production environments where inference speed is critical. By abstracting the complexity of caching logic behind a simple interface, it enables developers to enhance model performance with minimal code changes. Args: pipe_or_adapter (`DiffusionPipeline`, `BlockAdapter` or `Transformer`, *required*): The standard Diffusion Pipeline or custom BlockAdapter (from cache-dit or user-defined). For example: cache_dit.enable_cache(FluxPipeline(...)). cache_config (`BasicCacheConfig`, *required*, defaults to BasicCacheConfig()): Basic DBCache config for cache context, defaults to BasicCacheConfig(). The configurable params listed belows: Fn_compute_blocks: (`int`, *required*, defaults to 8): Specifies that `DBCache` uses the**first n**Transformer blocks to fit the information at time step t, enabling the calculation of a more stable L1 difference and delivering more accurate information to subsequent blocks. Please check https://github.com/vipshop/cache-dit/blob/main/docs/DBCache.md for more details of DBCache. Bn_compute_blocks: (`int`, *required*, defaults to 0): Further fuses approximate information in the **last n** Transformer blocks to enhance prediction accuracy. These blocks act as an auto-scaler for approximate hidden states that use residual cache. residual_diff_threshold (`float`, *required*, defaults to 0.08): the value of residual diff threshold, a higher value leads to faster performance at the cost of lower precision. max_accumulated_residual_diff_threshold (`float`, *optional*, defaults to None): The maximum accumulated relative l1 diff threshold for Cache. If set, when the accumulated relative l1 diff exceeds this threshold, the caching strategy will be disabled for current step. This is useful for some cases where the input condition changes significantly in a single step. Default None means this feature is disabled. max_warmup_steps (`int`, *required*, defaults to 8): DBCache does not apply the caching strategy when the number of running steps is less than or equal to this value, ensuring the model sufficiently learns basic features during warmup. warmup_interval (`int`, *required*, defaults to 1): Skip interval in warmup steps, e.g., when warmup_interval is 2, only 0, 2, 4, ... steps in warmup steps will be computed, others will use dynamic cache. max_cached_steps (`int`, *required*, defaults to -1): DBCache disables the caching strategy when the previous cached steps exceed this value to prevent precision degradation. max_continuous_cached_steps (`int`, *required*, defaults to -1): DBCache disables the caching strategy when the previous continous cached steps exceed this value to prevent precision degradation. enable_separate_cfg (`bool`, *required*, defaults to None): Whether to do separate cfg or not, such as Wan 2.1, Qwen-Image. For model that fused CFG and non-CFG into single forward step, should set enable_separate_cfg as False, for example: CogVideoX, HunyuanVideo, Mochi, etc. cfg_compute_first (`bool`, *required*, defaults to False): Whether to compute cfg forward first, default is False, meaning: 0, 2, 4, ..., -> non-CFG step; 1, 3, 5, ... -> CFG step. cfg_diff_compute_separate (`bool`, *required*, defaults to True): Whether to compute separate difference values for CFG and non-CFG steps, default is True. If False, we will use the computed difference from the current non-CFG transformer step for the current CFG step. num_inference_steps (`int`, *optional*, defaults to None): num_inference_steps for DiffusionPipeline, used to adjust some internal settings for better caching performance. For example, we will refresh the cache once the executed steps exceed num_inference_steps if num_inference_steps is provided. steps_computation_mask (`List[int]`, *optional*, defaults to None): This param introduce LeMiCa/EasyCache style compute mask for steps. It is a list of length num_inference_steps indicating whether to compute each step or not. 1 means must compute, 0 means use dynamic/static cache. If provided, will override other settings to decide whether to compute each step. steps_computation_policy (`str`, *optional*, defaults to "dynamic"): The computation policy for steps when using steps_computation_mask. It can be "dynamic" or "static". "dynamic" means using dynamic cache for steps marked as 0 in steps_computation_mask, while "static" means using static cache for those steps. calibrator_config (`CalibratorConfig`, *optional*, defaults to None): Config for calibrator. If calibrator_config is not None, it means the user wants to use DBCache with a specific calibrator, such as taylorseer, foca, and so on. params_modifiers ('ParamsModifier', *optional*, defaults to None): Modify cache context params for specific blocks. The configurable params listed belows: cache_config (`BasicCacheConfig`, *required*, defaults to BasicCacheConfig()): The same as 'cache_config' param in cache_dit.enable_cache() interface. calibrator_config (`CalibratorConfig`, *optional*, defaults to None): The same as 'calibrator_config' param in cache_dit.enable_cache() interface. **kwargs: (`dict`, *optional*, defaults to {}): The same as 'kwargs' param in cache_dit.enable_cache() interface. parallelism_config (`ParallelismConfig`, *optional*, defaults to None): Config for Parallelism. If parallelism_config is not None, it means the user wants to enable parallelism for cache-dit. Please check https://github.com/vipshop/cache-dit/blob/main/src/cache_dit/parallelism/parallel_config.py for more details of ParallelismConfig. backend: (`ParallelismBackend`, *required*, defaults to "ParallelismBackend.NATIVE_DIFFUSER"): Parallelism backend, currently only NATIVE_DIFFUSER and NVTIVE_PYTORCH are supported. For context parallelism, only NATIVE_DIFFUSER backend is supported, for tensor parallelism, only NATIVE_PYTORCH backend is supported. ulysses_size: (`int`, *optional*, defaults to None): The size of Ulysses cluster. If ulysses_size is not None, enable Ulysses style parallelism. This setting is only valid when backend is NATIVE_DIFFUSER. ring_size: (`int`, *optional*, defaults to None): The size of ring for ring parallelism. If ring_size is not None, enable ring attention. This setting is only valid when backend is NATIVE_DIFFUSER. tp_size: (`int`, *optional*, defaults to None): The size of tensor parallelism. If tp_size is not None, enable tensor parallelism. This setting is only valid when backend is NATIVE_PYTORCH. parallel_kwargs: (`dict`, *optional*, defaults to {}): Additional kwargs for parallelism backends. For example, for NATIVE_DIFFUSER backend, it can include `cp_plan` and `attention_backend` arguments for `Context Parallelism`. kwargs (`dict`, *optional*, defaults to {}) Other cache context kwargs, please check https://github.com/vipshop/cache-dit/blob/main/src/cache_dit/caching/cache_contexts/cache_context.py for more details. Examples: ```py >>> import cache_dit >>> from diffusers import DiffusionPipeline >>> pipe = DiffusionPipeline.from_pretrained("Qwen/Qwen-Image") # Can be any diffusion pipeline >>> cache_dit.enable_cache(pipe) # One-line code with default cache options. >>> output = pipe(...) # Just call the pipe as normal. >>> stats = cache_dit.summary(pipe) # Then, get the summary of cache acceleration stats. >>> cache_dit.disable_cache(pipe) # Disable cache and run original pipe. Nz1cache_config is None, using default DBCacheConfigzParallelism is enabled and cache_config is None. Please manually set cache_config to avoid potential compatibility issues. Otherwise, cache will not be enabled. cache_typeFn_compute_blocksBn_compute_blocksmax_warmup_stepsmax_cached_stepsmax_continuous_cached_stepsresidual_diff_thresholdenable_separate_cfgcfg_compute_firstcfg_diff_compute_separate) rrr r!r"r#r$r%r&cSsi|] \}}|dur||qSN).0kvr(r(W/home/ubuntu/vllm_env/lib/python3.10/site-packages/cache_dit/caching/cache_interface.py sz enable_cache..zManually settup DBCache context without BasicCacheConfig is deprecated and will be removed in the future, please use `cache_config` parameter instead!renable_taylorseerenable_encoder_taylorseerzManually settup TaylorSeer calibrator without TaylorSeerCalibratorConfig is deprecated and will be removed in the future, please use `calibrator_config` parameter instead!r )TaylorSeerCalibratorConfigtaylorseer_cache_typeresidualtaylorseer_order)enable_calibratorenable_encoder_calibratorcalibrator_cache_typer3rrztype: zg is not valid, Please pass DiffusionPipeline or BlockAdapterfor the 1's position param: pipe_or_adapterz.cache_config is None, skip enabling cache for .z7parallelism_config should be of type ParallelismConfig.has_controlnetextra_parallel_modules)skip_post_init transformerz_The given DiffusionPipeline does not have a 'transformer' attribute, cannot enable parallelism.F)uniquerzJNo transformer is detected in the BlockAdapter, skip enabling parallelism.zhMultiple transformers are detected in the BlockAdapter, all transfomers will be enabled for parallelism.r()'loggerinforwarninggetr NONEitemsupdatercache_contexts.calibratorsr0 isinstancerr torchnnModulerr apply ValueErrortype __class____name__rparallel_kwargs_has_controlnet_parse_extra_parallel_modulesr get_adapterhasattrr; normalizeflatten is_normalizedlen enumerater)rrrrrkwargscontext_kwargsrdeprecated_kwargsr0extra_parallel_module transformersadapterir;r(r(r, enable_caches0                       r_cCs6t|tr |j}n|}t|drt|ddurdSdS)z+Check if the given pipeline has ControlNet. controlnetNTF)rEr piperRgetattr)rrar(r(r,rOms rOracCsh|jj}t|dr|ds|dst|ddfSt|dr&t|ddfSt|dr2t|ddfSdS)Ntext_encoder_2Hunyuan Kandinskytext_encoder_3 text_encoder)NN)rLrMrR startswithrb)ra pipe_cls_namer(r(r,_parse_text_encoderxs  rjr[cCst|tr |j}n|}|sgSg}|D]@}t|tjjr"||qt||rJ|dkrAt|\}}|dur;||qt dq|t ||qt d|dq|S)NrgzAText encoder not found in the pipeline for extra parallel module.zExtra parallel module name z not found in the pipeline.) rEr rarFrGrHappendrRrjr=r?rb)rr[raparsed_extra_parallel_modulesmodule_or_namerg_r(r(r,rPs.      rPr;cKs|r>d|vr'ddlm}||dd\}}t||d<|dur&t||d<nhd}t||}|r>td |d tj |fi|dS) aRefresh cache context for the given transformer. This is useful when the users run into transformer-only case with dynamic num_inference_steps. For example, when num_inference_steps changes significantly between different requests, the cache context should be refreshed to avoid potential precision degradation. Usage: ```py >>> import cache_dit >>> from cache_dit import DBCacheConfig >>> from diffusers import DiffusionPipeline >>> # Init cache context with num_inference_steps=None (default) >>> pipe = DiffusionPipeline.from_pretrained("Qwen/Qwen-Image") >>> pipe = cache_dit.enable_cache(pipe.transformer, cache_config=DBCacheConfig(...)) >>> # Assume num_inference_steps is 28, and we want to refresh the context >>> cache_dit.refresh_context(transformer, num_inference_steps=28, verbose=True) >>> output = pipe(...) # Just call the pipe as normal. >>> stats = cache_dit.summary(pipe.transformer) # Then, get the summary >>> # Update the cache context with new num_inference_steps=50. >>> cache_dit.refresh_context(pipe.transformer, num_inference_steps=50, verbose=True) >>> output = pipe(...) # Just call the pipe as normal. >>> stats = cache_dit.summary(pipe.transformer) # Then, get the summary >>> # Update the cache context with new cache_config. >>> cache_dit.refresh_context( pipe.transformer, cache_config=DBCacheConfig( residual_diff_threshold=0.1, max_warmup_steps=10, max_cached_steps=20, max_continuous_cached_steps=4, num_inference_steps=50, ), verbose=True, ) >>> output = pipe(...) # Just call the pipe as normal. >>> stats = cache_dit.summary(pipe.transformer) # Then, get the summary ``` rr )load_cache_configT)resetNr>verboserrzIforce_refresh_kwargs contains cache_config, please put the extra kwargs: zF into cache_config directly. Ohtherwise, these kwargs will be ignored.) utilsrocopydeepcopysetkeysr=r?r maybe_refresh_context)r;force_refresh_kwargsrorr allowed_keysnot_allowed_keysr(r(r,refresh_contexts0(   r{cCs(|jj}t|td|ddS)Nz$Acceleration hooks is disabled for: r7)rLrMr maybe_release_hooksr=r?)rcls_namer(r(r, disable_caches r~cKstjdi|S)Nr()r supported_pipelines)rXr(r(r,rsrcCs t|Sr')r rQ)rar(r(r,rQs rQ compute_bins cache_bins total_stepscCsg}d}|}|}|||dur't|t||ks&Jdnt|t|}||kr`|rE|}|dg|||7}|rW|}|dg|||7}||kr\n||ks3|d|S)NrzDThe sum of compute and cache intervals must be at least total_steps.r )rsreversesumpopextend)rrrmaskstepcicair(r(r, _steps_mask s0 rmedium mask_policyc s|dur|durt|||dS|dusJdgdgdggdgdggdgd ggd gd gd }d tttdtfddd tttdtdtttffdd dtttttfdtdtttttfffdd }|dkr |D]}tt|dt|d}t |d}t |d}||krt |D]@}|d|t t|||ddd7<||krn |d|t t|||ddd7<||krnq||krn|ddd7<||krn||ksq~|||}n|dkr|dkr|||}n|dkr|dkr|dkrngdgdggd gd!ggd"gd#ggd$gd%gd }|d&krmgd'dgggd(ddgggd)dd*gggd+d*d*ggd }ngd(dgggd)ddgggd+dd*gggd,d*d*ggd }|D]}|||}qn1|dkr|d-vsJd.|d/d*dgdgg} d0dgd*gg} |d1kr| } n| } | | | | d }||vrt d2|d3t |d/||\}}t|||d} d| d<| S)4a Define a step computation mask based on compute and cache bins. Args: compute_bins (`List[int]`, *optional*, defaults to None): A list specifying the number of consecutive steps to compute. For example, [4, 2] means compute 4 steps, then 2 steps. cache_bins (`List[int]`, *optional*, defaults to None): A list specifying the number of consecutive steps to cache. For example, [2, 4] means cache 2 steps, then 4 steps. total_steps (`int`, *optional*, defaults to None): Total number of steps for which the mask is generated. If provided, the sum of compute_bins and cache_bins must be at least total_steps. mask_policy (`str`, *optional*, defaults to "medium"): Predefined mask policy. Options are "slow", "medium", "fast", "ultra". For examples, if total_steps=28, each policy corresponds to specific compute and cache bin configurations: - "slow": compute_bins=[8, 3, 3, 2, 1, 1], cache_bins=1, 2, 2, 2, 3] - "medium": compute_bins=[6, 2, 2, 2, 2, 1], cache_bins=[1, 3, 3, 3, 3] - "fast": compute_bins=[6, 1, 1, 1, 1], cache_bins=[1, 3, 4, 5, 4] - "ultra": compute_bins=[4, 1, 1, 1, 1], cache_bins=[2, 5, 6, 7] Returns: `List[int]`: A list representing the step computation mask, where 1 indicates a compute step and 0 indicates a cache step. N)rrrz?total_steps must be provided when using predefined mask_policy.)rrr r )r rrrr)rrrrr )r rrrr)rr r r r r )r rr)rr r r r )rrr)slowrfastultrapolicyrcSst|dt|dS)Nrr )r)rr(r(r, _sum_policyrszsteps_mask.._sum_policy target_stepscs|\}}||krN|r|dd8<|ddkr|||kr) ||gS|r=|dd8<|ddkr=|||krH ||gS||ks ||gS)Nr r)r)rrrr)rr(r,_truncate_policyus$       z$steps_mask.._truncate_policypoliciescs(i}|D] \}}||||<q|Sr')rB)rrtruncated_policiesnamer)rr(r,_truncate_predefined_policiessz1steps_mask.._truncate_predefined_policiesrr g?rr)rrrrr )r r r r )rrr r r )r r rr)rr r r r )r rrr)rr r r r )r rrr )rrr )rr r )rr r r)rr r )rr r )rrzaOnly total_steps=4 or 6 is supported for predefined masks while total_steps < 8. Got total_steps=r7rrz mask_policy z is not valid. Choose from )rrintdictstrvaluesminrVrsrtrangemaxrJlistrv) rrrrpredefined_policiesrr min_bins_lenr^constant_plicy_4_stepsconstant_plicy_6_stepsconstant_plicy compute_maskr()rrr, steps_mask1s! *  . .             r)NNNNr')NNNr)0rsrFtypingrrrrr diffusersrr cache_typesr block_adaptersr r cache_adaptersr cache_contextsrrrrparams_modifierr parallelismrrcache_dit.loggerrrMr=rGrHr_boolrOrrjrPr{r~rrrQrrrr(r(r(r,s               Y   % D       &