o c i"@s:dZddlZddlmZddlmZmZGdddZdS)a' CUDA graph capture for the talker's single-token decode step, using transformers StaticCache. The talker has 28 transformer layers. Instead of reimplementing the forward pass manually, we use the model's own forward with StaticCache. The StaticCache provides fixed-size KV tensors compatible with CUDA graphs. Strategy: - Use transformers StaticCache for KV cache management - Use the model's forward method (handles mask, RoPE, attention internally) - Capture the single-token decode as a CUDA graph - Update cache_position buffer between replays N) StaticCache)create_causal_mask!create_sliding_window_causal_maskc@seZdZdZdejdfddZddZd!d ejdBfd d Z d e fd dZ ddZ e d"ddZde fddZddZd ejdejdBfddZe dejd e dejfdd ZdS)# TalkerGraphz Captures the talker's single-token decode step as a CUDA graph, using the model's own forward with transformers StaticCache. cudaicCs||_t|j}|dur|ntj}||_||_||_|j|_|j |_ ||_ t ||d|_ tjdd|j||d|_tjdd|j||d|_tjdtj|d|_tjddtj|d|_tjdddtj|d|_d|_d|_d|_d|_d|_dS)N)config max_cache_lendtypedeviceF)r torchindexrcurrent_device device_indexr max_seq_len hidden_sizenum_hidden_layers num_layersmodelr static_cachezeros input_buf output_buflongcache_positionfloat32 rope_deltas position_idsgraphcaptured attn_maskattn_mask_table _mask_key)self talker_model talker_configr r rrr(S/home/ubuntu/vllm_env/lib/python3.10/site-packages/faster_qwen3_tts/talker_graph.py__init__s(  zTalkerGraph.__init__cCsf|jj}t|d|j}t|d|j|j}tjd|d||j|jd}|j j D] }|j s0| |q&dS)zEForce lazy initialization of StaticCache layers before graph capture.num_key_value_headshead_dimr r N) rrgetattrnum_attention_headsrrrr r rlayersis_initializedlazy_initialization)r%r num_kv_headsr,dummy_klayerr(r(r)_init_cache_layers=s  zTalkerGraph._init_cache_layersNattention_maskcCstjdd|j|j|jd}|j}dg||_|jjj durt nt }t |D]}tj |g|jd}||jj||||jd}||j|<q%|jdurQ|jd|_dS|j|jddS)Nr r r )r input_embedsr6rpast_key_valuesr)rrrr r rr#rrsliding_windowrrrangetensorrr"clonecopy_)r%r6dummymax_lenmask_fniposfullr(r(r)_build_attention_masksGs"    z"TalkerGraph._build_attention_maskspositioncCs|j|j|dSN)r"r>r#)r%rFr(r(r)_set_attention_mask^szTalkerGraph._set_attention_maskcCs2|j|j|j|j|j|jdd}|j|jdS)z0Single-token decode through the model's forward.T) inputs_embedsr6r9rr use_cacheN) rrr"rrrrr>last_hidden_state)r%outr(r(r) _decode_stepaszTalkerGraph._decode_stepdr c CsPtd|d||||jd<||t|D]}|qtj tdtj |j Utj |_ tj}|tjtj|,|tj tj |j  |Wdn1sowYWdn1s~wYWdn1swYtj|tj d|_tddS)z Capture CUDA graph for single-token decode. prefill_len: simulated prefill length for warmup (graph is position-independent). zWarming up talker graph (z runs)...rz)Capturing CUDA graph for talker decode...NTzTalker CUDA graph captured!)printr5rErrHr;rMrr synchronizer r CUDAGraphr Stream wait_streamcurrent_streamstreamr!)r% prefill_len num_warmup_sr(r(r)capturems6           zTalkerGraph.capturerVcCs|jdS)zReset cache for new sequence.N)rreset)r%rVr(r(r)r[szTalkerGraph.resetcCs|jd}t|jD]1}||\}}|jd}||jkr*td|d|jdtj||j d}|j |||d|iq |S)z Copy HF DynamicCache from prefill into our StaticCache. past_key_values: DynamicCache with num_layers layers of [1, kv_heads, seq_len, head_dim] rzInput is too long: prefill has z tokens but max_seq_len=z.. Use shorter text or shorter reference audio.r7r) rr[r;rshaper RuntimeErrorraranger update)r%r9seq_lenlikv cache_posr(r(r) prefill_kvs    zTalkerGraph.prefill_kvrcCsd}d}|dur<|dkjdd}t|}tj|jd|j|j|jd}t |D]\}}|dkr;d||d|f<q+|j dusF||j krN| |||_ |durY|j dS|dkrd|d}|j |j|j j|j jddS)zCSet padding-aware attention mask and rope deltas for decode parity.Nr)dimr r )r )sumtupletolistronesr]rr r enumerater#r$rErzero_rh unsqueezer>to)r%r6rmask_keyfull_attention_mask pad_countsbpadsr(r(r)set_generation_states.    "z TalkerGraph.set_generation_stater8returncCsf|j|||jd<|||j|jd|jj}|j|d ddd|j |j S)z Run one decode step. input_embeds: [1, 1, hidden_size] position: current sequence position Returns: [1, 1, hidden_size] hidden states rr rg) rr>rrHrrpr rroexpandr replayr)r%r8rFdeltar(r(r)runs    zTalkerGraph.runrG)rNr )__name__ __module__ __qualname____doc__rbfloat16r*r5TensorrEintrHrMinference_moderZr[rfrvr{r(r(r(r)rs  "  ' r)rr transformersrtransformers.masking_utilsrrrr(r(r(r)s