o پi2&@sDdZddlmZddlZGdddZddZedkr edSdS) u Smart Router: Automatically routes requests between local Ollama and remote SGLang. Uses an LLM judge to classify tasks as simple or complex, then routes accordingly: - Simple tasks → Local Ollama (fast response) - Complex tasks → Remote SGLang (powerful model) Usage: from sglang.srt.entrypoints.ollama.smart_router import SmartRouter router = SmartRouter( local_host="http://localhost:11434", remote_host="http://sglang-server:30001", ) response = router.chat("Hello!") )OptionalNc@seZdZdZdZ      dded ed ed ed eed eef ddZ d dedede eeffddZ d dedede eeffddZ    d!dedee dededede f ddZ    d!dedee dededef ddZdS)" SmartRouterzVRoutes requests between local Ollama and remote SGLang using LLM-based classification.aYou are a task classifier. Classify the following user request into one of two categories. Categories: - SIMPLE: Quick responses, greetings, factual questions, definitions, translations, basic Q&A - COMPLEX: Tasks requiring deep reasoning, multi-step analysis, long explanations, creative writing, detailed research Reply with ONLY one word: either SIMPLE or COMPLEX. User request: "{prompt}" Category:http://localhost:11434http://localhost:30001llama3.2Qwen/Qwen2.5-1.5B-InstructN local_host remote_host local_model remote_model judge_model judge_hostcCsPtj|d|_tj|d|_||_||_|p||_|p||_tj|jd|_dS)a Initialize the smart router. Args: local_host: URL of local Ollama server remote_host: URL of remote SGLang server local_model: Model name for local Ollama remote_model: Model name for remote SGLang judge_model: Model for LLM-based classification (default: same as local_model) judge_host: Host for judge model (default: same as local_host) )hostN) ollamaClient local_client remote_clientr r r r judge_client)selfrr r r r r r^/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/entrypoints/ollama/smart_router.py__init__'s  zSmartRouter.__init__Fpromptverbosereturnc Csz9|jj|ddd}|jj|jd|dgdddd }|d d }|r0td |d |vr7WdSWdStyV}z|rKtd|dWYd}~dSd}~ww)zm Use LLM to classify the prompt. Returns: Tuple of (use_remote, reason) Ni)ruserrolecontentr ) temperature num_predict)modelmessagesoptionsmessagerz[Router] LLM Judge: COMPLEX)Tz Complex task)Fz Simple taskzouter] LLM Judge failed: z, defaulting to local)Fz!Judge failed, defaulting to local) CLASSIFICATION_PROMPTformatrchatr stripupperprint Exception)rrrclassification_promptresponseresulterrr_classify_with_llmEs*   zSmartRouter._classify_with_llmcCs |||S)z Determine if the prompt should be routed to remote SGLang. Args: prompt: User's input prompt verbose: Print debug information Returns: Tuple of (should_use_remote, reason) )r2)rrrrrrshould_use_remotehs zSmartRouter.should_use_remoter# force_local force_remotec Csp|dur d|dg}|}nd}t|D]}|ddkr$|dd}nq|r,d\}} n|r3d\}} n|||\}} |rF|j} |j} d } n|j} |j} d } |rZtd | d | z| j| |d } | dd| | | dWSt y}z>|rtd| d|d|s|jn|j}|s|jn|j}|sd nd }|j||d } | dd||d| dWYd}~Sd}~ww)a Route the request and get response. Args: prompt: User's input (used if messages is None) messages: Full message history (overrides prompt if provided) verbose: Print routing decision force_local: Force use of local model force_remote: Force use of remote model Returns: Response dict with 'content', 'model', 'location', 'reason' keys NrrrrTz Forced remoteFz Forced local Remote SGLang Local Ollama [Router] -> | Model: )r"r#r%)rr"locationreasonz [Router] z failed: z, falling back...zFallback from ) reversedgetr3rr rr r,r)r-)rrr#rr4r5 check_promptmsg use_remoter>clientr"r=r/r1fallback_clientfallback_modelfallback_locationrrrr)usZ        zSmartRouter.chatccs|durd|dg}|}nd}t|D]}|ddkr%|dd}nq|r-d\}} n|r4d\}} n|||\}} |rG|j} |j} d } n|j} |j} d } |r[td | d | | j| |d dD]} | VqcdS)ze Route the request and stream response. Yields: Response chunks Nrrr6rrr7r8r9r:r;r<T)r"r#stream) r?r@r3rr rr r,r))rrr#rr4r5rArBrCr>rDr"r=chunkrrr chat_streams6     zSmartRouter.chat_stream)rrrrNN)F)NFFF)__name__ __module__ __qualname____doc__r'strrrbooltupler2r3listdictr)rJrrrrrs~   # Src CsXtdtdtdtdtdtdtdtdtdd d d d }g} zXtd}|dvr?tdWdS|sCWq*|d|dtddd dd}|j||d dD]}|didd}|rut|dd d||7}q\td|d|dWn'tytdYdSt y}ztd|dWYd}~nd}~wwq+)z%Interactive demo of the smart router.z<============================================================z,Smart Router: Local Ollama <-> Remote SGLangz Routing strategy:z8 LLM Judge classifies each request as SIMPLE or COMPLEXz' - SIMPLE tasks -> Local Ollama (fast)z- - COMPLEX tasks -> Remote SGLang (powerful)z Type 'quit' to exit rrrr)rr r r TzYou: )quitexitqzGoodbye!rrz Assistant: r6)endflush)rr#rr%r assistantz Goodbye!zError: N) r,rinputr*lowerappendrJr@KeyboardInterruptr-)routerr# user_input full_responserIrr1rrrmainsX    rb__main__)rNtypingrrrrbrKrrrrs ^3