o i@sndZddlmZddlZddlZddlmZmZddlm Z m Z e e Z GdddZe Gdd d ZdS) a Dual vLLM Engine Router - Distributes requests across multiple engine instances. OPTIMIZATION: With gpu_memory_utilization=0.25, a 0.5B model uses ~20GB on an 80GB GPU. Running 2 engines at 0.12 each uses ~20GB total while doubling prefill parallelism. This directly halves the #1 streaming bottleneck: prefill contention under concurrency. The router is transparent to the pipeline -- it exposes the same .engine.generate() interface as a single SparkTTSModel via a RoutedEngine wrapper. ) annotationsN)AnyOptional) dataclassfieldc@s"eZdZdZd ddZddZdS) RoutedEnginea  Wraps multiple vLLM AsyncLLMEngine instances behind a single .generate() interface. Round-robins generate() calls across engines so concurrent requests spread prefill work instead of queuing on a single engine. Thread-safe via atomic counter. engineslistcCs||_d|_t|_dS)zM Args: engines: List of AsyncLLMEngine instances rN)_engines_counter threadingLock_lock)selfrr7/home/ubuntu/veenaModal/veena3modal/core/dual_engine.py__init__szRoutedEngine.__init__cKsf|j|jt|j}|jd7_Wdn1swY|j|}|jd|||d|S)z Route a generate() call to the next engine via round-robin. Returns the async generator from the selected engine. Same interface as AsyncLLMEngine.generate(). N)promptsampling_params request_idr)rr lenr generate)rrrrkwargsidxenginerrrr(s zRoutedEngine.generateN)rr )__name__ __module__ __qualname____doc__rrrrrrrs  rc@seZdZUdZdZded<dZded<dZd ed <d Zd ed <e e dZ ded<d Z ded<e dddZe ddZddZdddZd S)DualEngineRoutera Manages multiple vLLM engine instances on the same GPU. Memory Budget (A100-80GB, 2 engines at 0.12 each): - Engine A: 1.3GB model + ~8GB KV cache = ~9.3GB - Engine B: 1.3GB model + ~8GB KV cache = ~9.3GB - BiCodec decoder: ~0.6GB - Total: ~19.2GB (leaves ~61GB free) - Each engine: ~170 concurrent sequences - Combined: ~340 concurrent with 2x prefill throughput str model_pathint num_enginesgQ?floatgpu_memory_per_engineNz Optional[str]hf_token)default_factoryr _modelszOptional[RoutedEngine]_routed_enginereturnrcCs|jdur td|jS)z:The routed engine that distributes calls across instances.Nz:DualEngineRouter not initialized. Call initialize() first.)r, RuntimeErrorrrrrrRs zDualEngineRouter.enginecCs|jstd|jdjS)zDTokenizer from the first model (all share the same tokenizer/vocab).z DualEngineRouter not initializedr)r+r. tokenizerr/rrrr0Ys zDualEngineRouter.tokenizercCsddlm}td|jd|jdg}t|jD]2}td|dd|jd ||j|j|jd }|j || |j td |dd qt ||_ td |jddS)z2Initialize all vLLM engine instances sequentially.r) SparkTTSModelz Initializing z vLLM engines (gpu_memory=z each)...z Loading engine r/z...)r#r)gpu_memory_utilizationz Engine z readyzDualEngineRouter: z, engines active, round-robin routing enabledN)veena3modal.core.model_loaderr1loggerinfor&r(ranger#r)r+appendrrr,)rr1 raw_enginesimodelrrr initialize`s"     zDualEngineRouter.initializecCs|jr |jdSdS)z3Compatibility: return model type from first engine.rindic_speakers)r+get_model_typer/rrrr>yszDualEngineRouter.get_model_type)r-r)r-r")rrrrr#__annotations__r&r(r)rr r+r,propertyrr0r<r>rrrrr <s       r )r __future__rloggingr typingrr dataclassesrr getLoggerrr5rr rrrrs  &