o پi @sddlZddlmZddlmZmZmZddlZGdddejZ e Z de fddZ d e fd d Z ede fd d Z ddededeejjdeejjd eeeff ddZdS)N)contextmanager)AnyCallableOptionalc@seZdZddZdS)do_multi_stream_localcCs d|_dS)NF)do_multi_stream)selfr W/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/utils/multi_stream_utils.py__init__  zdo_multi_stream_local.__init__N)__name__ __module__ __qualname__r r r r r r s renablecCs |t_dSN_localr)rr r r set_do_multi_streamr rreturncCstjSrrr r r r rsrccs0tj}t|z dVWt|dSt|wr)rrr)rprev_do_multi_streamr r r with_multi_streams rfn0fn1events aux_streamcCsto|du}|rA|d|}tj||d|}|dWdn1s2wY|d||fS|}|}||fS)aUtility function to run two functions in two cuda streams in parallel. Multi-stream is only enabled when cuda graph is turned on because switch stream has extra host overhead. This design is mainly for low latency use case. It needs to be improved for max throughput use case. For simplicity, fn0 and fn1 do not support inputs. Args: fn0 (Callable): callable for the default stream fn1 (Callable): callable for the second stream, aux_stream events (list[torch.cuda.Event]): cuda events for callables aux_stream (Optional[torch.cuda.Stream]): the second cuda stream for fn1. Multi-stream is disabled when aux_stream is None. Returns: tuple[Any, Any]: the return values of fn0() and fn1() Nr)rrecordtorchcudastreamwait)rrrr multi_streamresult0result1r r r maybe_execute_in_parallel%s   r%r) threading contextlibrtypingrrrrlocalrrboolrrrlistrEventStreamtupler%r r r r s,