o i @sxdZddlZddlZddlmZmZmZeeZ GdddZ GdddZ   dd e d e d e d ede f ddZdS)a Multi-Engine vLLM Wrapper - transparent dispatch across N engine instances. OPTIMIZATION: A 0.5B model uses ~1.3GB weights + ~5-10GB KV cache per engine. On an 80GB A100, we can comfortably run 3 engines at ~12GB each (36GB total). This triples effective prefill capacity, directly cutting streaming TTFB under concurrent load by ~3x. The wrapper is transparent to pipeline code: it exposes the same `.engine.generate()` interface, but routes calls to different underlying engines. Usage: # Creates N engines, each with their own vLLM subprocess model = create_multi_engine_model(model_path, num_engines=3, gpu_memory_per_engine=0.12) # Pipeline uses it identically to a single engine: model.engine.generate(prompt, sampling_params, request_id) # routed to engine N N)AnyAsyncGeneratorListc@sneZdZdZdefddZdeeeffddZ dedd fd d Z de fd d Z de efddZddZd S)LoadAwareEnginez Wraps multiple AsyncLLMEngine instances with least-inflight dispatch. This reduces tail latency under uneven request completion times versus strict round-robin, while preserving fair spread with a rotating cursor on ties. enginescCs*||_dd|D|_d|_t|_dS)NcSsg|]}dqS)r).0_rr8/home/ubuntu/veenaModal/veena3modal/core/multi_engine.py %sz,LoadAwareEngine.__init__..r)_engines _inflight_cursor threadingLock_lock)selfrrrr __init__#szLoadAwareEngine.__init__returncCs|jPt|j}|dkrtdt|j}|j|}|}t|D]}|||}|j||kr4|}nq!|j|d7<|d||_||j|fWdS1sVwYdS)NrzNo engines configured)rlenr RuntimeErrorminr rrange)rnmin_loadstartselectedoffsetidxrrr _acquire_engine)s"      $zLoadAwareEngine._acquire_enginerNcCs|j5d|krt|jkr0nntd|j|d|j|<WddSWddSWddS1s;wYdS)Nrr)rrr max)rrrrr _release_engine:s"zLoadAwareEngine._release_enginecsX\}z |j|i|Wn tywdtffdd }|S)zFRoute generate() to the least-loaded engine and release on completion.rc s:z2z 3dHW}|Vq6WdSwN)r")itembase_generator engine_idxrrr _wrappedHs z*LoadAwareEngine.generate.._wrapped)r generate Exceptionr"r)rargskwargsenginer(rr%r r)?s   zLoadAwareEngine.generatecCs4|j t|jWdS1swYdS)z1Current in-flight request count per engine index.N)rlistr rrrr inflight_snapshotQs$z!LoadAwareEngine.inflight_snapshotcCst|jd|S)z7Forward any other attribute access to the first engine.r)getattrr )rnamerrr __getattr__VszLoadAwareEngine.__getattr__)__name__ __module__ __qualname____doc__r.rtupleintrr r"rr)rr0r3rrrr rs rc@s6eZdZdZdefddZddZddZd d Zd S) MultiEngineModela Drop-in replacement for SparkTTSModel that wraps N engines. Exposes: .engine -> LoadAwareEngine (dispatches generate() calls) .tokenizer -> shared tokenizer (same across all engines) .model_path, .model_type, etc. modelscCsT||_|d|_|jj|_|jj|_|jj|_|jj|_dd|D}t||_dS)ze Args: models: List of SparkTTSModel instances (each has its own engine) rcSsg|]}|jqSrr-)rmrrr r tsz-MultiEngineModel.__init__..N)_models_primary tokenizer model_path model_typehf_tokenrr-)rr;rrrr res     zMultiEngineModel.__init__cCs |jSr#)r?get_model_typer/rrr rDws zMultiEngineModel.get_model_typecC|jSr#)r@r/rrr get_tokenizerzzMultiEngineModel.get_tokenizercCrEr#r<r/rrr get_engine}rGzMultiEngineModel.get_engineN) r4r5r6r7r.rrDrFrHrrrr r:[s   r:Q?rA num_enginesrCgpu_memory_per_enginerc Ksddlm}td|d|ddg}t|D]+}td|dd |d |d|||d |}||td |dd qt|} td|d| S)a Create a multi-engine model with N vLLM instances on the same GPU. Memory budget per engine (A100-80GB): - Model weights: ~1.3GB (shared via CUDA, but each engine loads independently) - KV cache at 0.12 utilization: ~8GB - Total per engine: ~10GB - 3 engines: ~30GB, leaves 50GB free Args: model_path: Path to model directory num_engines: Number of vLLM engine instances (default: 2) hf_token: HuggingFace token gpu_memory_per_engine: GPU memory fraction per engine **engine_kwargs: Additional AsyncEngineArgs overrides passed to each engine Returns: MultiEngineModel with round-robin dispatch r) SparkTTSModelzCreating multi-engine model: z engines x z.0%z GPU eachz Initializing engine r/z...)rArCgpu_memory_utilizationz Engine z readyzMulti-engine model ready: z engines, round-robin dispatchNr)veena3modal.core.model_loaderrMloggerinforappendr:) rArKrCrL engine_kwargsrMr;imodelmultirrr create_multi_engine_models"   rX)rINrJ)r7loggingrtypingrrr getLoggerr4rQrr:strr9floatrXrrrr s* @(