o ۷i @sddlZddlmZddlmZddlmZmZddlmZddl m Z m Z ddl m Z eeZGd d d eZ dd eejd eejdejdeddf ddZdeejddfddZGddde ZdS)N)nn) init_logger) HookRegistry ModelHook)current_omni_platform)OffloadBackend OffloadConfig)ModuleDiscoveryc@szeZdZdZdZ ddeejdej de fddZ d ejd d fd d Z d ejd d fddZ d ejd eeeffddZd S)SequentialOffloadHooka Hook for sequential offloading with mutual exclusion on encoder and DiT modules. To be used as a model-level (or "component-level") of CPU offloading method; When a module's forward is called, this hook offloads target modules to CPU and loads the current module to GPU. sequential_offloadToffload_targetsdevice pin_memorycCs||_||_||_dSNr rr)selfr rrrf/home/ubuntu/vllm_env/lib/python3.10/site-packages/vllm_omni/diffusion/offloader/sequential_backend.py__init__s zSequentialOffloadHook.__init__modulereturnNcCszt|}Wn tyYdSw|j}|jdkrdS|dtj|j rC|D]}|j jjdkrB|j sB|j |_ q.dSdS)zMove module to CPU.Ncpu) next parameters StopIterationrtypetotorchcuda empty_cacherdata is_pinned)rrparamprevious_deviceprrr_to_cpu&s"      zSequentialOffloadHook._to_cpucCsDzt|j|jkrWdSWn tyYdSw||jdS)zMove module to GPU.N)rrrrr)rrrrr_to_gpu:s zSequentialOffloadHook._to_gpucOst|jD]}||q||ttddd|jD|jj|j j d|j j t ddd||fS)Nz2Swapped: %s -> CPU, %s -> %s, free memory: %.4f GBcSsg|]}|jjqSr) __class____name__).0trrr Psz5SequentialOffloadHook.pre_forward..:i) r r&r'r synchronizeloggerdebugr(r)rrindexget_free_memory)rrargskwargstargetrrr pre_forwardEs   z!SequentialOffloadHook.pre_forwardT)r) __module__ __qualname____doc__ _HOOK_NAMElistrModulerrboolrr&r'tupledictr6rrrrr s   r T dit_modulesencoder_modulesrrrcCs|D]}t|}t|||d}|tj|td|jjq|D]}t|}t|||d}|tj|td|jjq"dS)a7Apply sequential offloading hooks to DiT and encoder modules. Registers hooks on modules to implement mutual-exclusion GPU allocation. - Before DiT runs, encoders are offloaded to CPU. - Before encoders run, DiT is offloaded to CPU. Args: dit_modules: DiT/transformer modules to register hooks on encoder_modules: Encoder modules to register hooks on device: Target GPU device for loading pin_memory: Whether to pin CPU memory for faster transfers Example: >>> apply_sequential_offload( ... dit_modules=[pipeline.transformer], ... encoder_modules=[pipeline.text_encoder, pipeline.vae], ... device=torch.device("cuda:0"), ... ) >>> # Modules of pipeline now automatically swap between CPU and GPU rzRegistered offload hook for %sN) r get_or_creater register_hookr;r/r0r(r))rArBrrdit_modregistryhookencrrrapply_sequential_offloadYs&  rImodulescCs>|D]}t|dd}|dur|tjtd|jjqdS)zRemove sequential offloading hooks from modules. Args: modules: Modules to remove hooks from Example: >>> all_modules = [*dit_modules, *encoder_modules] >>> remove_sequential_offload(all_modules) _hook_registryNzRemoved offload hook from %s)getattr remove_hookr r;r/r0r(r))rJrrFrrrremove_sequential_offloads  rNcsJeZdZdZdedejffdd Zdej ddfd d Z d d d Z Z S)ModelLevelOffloadBackendzModel-level (sequential) offloading backend. Uses SequentialOffloadHook registered via HookRegistry for automatic module swapping. configrcst||g|_dSr)superr_offload_modules)rrPrr(rrrs z!ModelLevelOffloadBackend.__init__pipelinerNc Cs|jr tddSt|}|jstddS|js#tddS|jD]}||jq&|j durXz |j j|jddWnt yW}z t d|WYd}~nd}~wwt |j|j|j|j jdg|j|j|_d|_tdd |jd |jdS) Nz(ModelLevelOffloadBackend already enabledzANo DiT/transformer modules found, skipping model-level offloadingz9No encoder modules found, skipping model-level offloadingT) non_blockingzFailed to move VAE to GPU: %s)rArBrrz %s (mutual exclusion)z, )enabledr/warningr discoverditsencodersrrvae Exceptionr0rIrPpin_cpu_memoryrRinfojoin dit_names encoder_names)rrTrJrHexcrrrenables@        zModelLevelOffloadBackend.enablecCs2|jsdSt|j|jd|_tddS)NFzModel-level offloading disabled)rVrNrRclearr/r^)rrrrdisables   z ModelLevelOffloadBackend.disable)rN) r)r8r9r:r rrrrr=rcre __classcell__rrrSrrOs 8rOr7)rr vllm.loggerrvllm_omni.diffusion.hooksrrvllm_omni.platformsrbaserr module_collectorr r)r/r r<r=rr>rIrNrOrrrrs.    L 1