o پizy@sddlZddlZddlmZmZddlZddlmZmZddl m Z ej GdddZ ej GdddZ ejgZd d gZd d gZe d e ddgddgZe d e dddgdde de ddgddgZe de ddde dddgdddgZe d e ddde dddgddgZe de ddde dddgddgZdejd ejd!ejfd"d#Z dRd$ejd%ejd&ejd'ejd(ejd)ejd*ed!ejfd+d,Z dSd$ejd%ejd&ejd'ejd(ejd-ejd.eejd!ejfd/d0Z  1 1 2 3dTd4eed5e d6ejd7ed8ed9eed:ed;edd?Z 1 1 2 3dUd4eed5e d6ejd7ed8ed:ed;edritemr.r?r1r<cloner@ranger3rA)r4r5r6r7r8rOrPrBtotal_output_dimrCr:rFrGrHrIrJrL slice_idxslice_start_inputslice_end_inputslice_start_outputslice_end_outputx_slicew_slicer2rrrreference_sgmv_expandsV              r]F)\(?prompts model_case torch_dtypemax_new_tokensbackendenable_lora_overlap_loadingdisable_cuda_graphdisable_radix_cachemem_fraction_statictest_tagc $Cs|j} dg} } tt|D]} | |j| | dt|j} qdd| D}td| d|jd|d|d d d|Dd |d t| |d |jdd|jD||j|j ||||d }|j |||d}Wdn1srwYt| |d |j|d}|j ||d}Wdn1swYt | |d d}|j |||d}|j ||d}Wdn1swYtt|D]/} | | }|j dur|j n|j }|j dur|j n|j }|jdur|jn|j}t|j| }t|j| }tt||}td|t|j| }t|j| }tt||}td||j| }|j| } t|g| gd}!td|!td|td| t|j| }"t|j| }#tdtt|#|tdtt|"||jddkrtt|||ksJd| d|d|d |ddd!d" |jddkrtt|||ksJd#| d|d|d |ddd!d" |!|krtd$|!d%|d&| d|d|d |ddd!d" qdS)'a_ Input a batch of prompts, and run lora tests one by one with several generate requests (each request will have bs=1). For prompt0, prompt1, ..., promptN, we will use adaptor0, adaptor1, ..., adaptorN included in model case, We will then compare the outputs of HF and SRT with and without LoRA. If number of prompts is larger than number of adaptors, the prompt i will use adaptor i % (number of adaptors). Args: prompts (List[str]): The batch of prompts to test. model_case (LoRAModelCase): The model case to test. torch_dtype (torch.dtype): The torch dtype to use. max_new_tokens (int): The maximum number of new tokens to generate. backend (str): The lora backend to use. disable_cuda_graph (bool, optional): Whether to disable CUDA graph. Defaults to False. disable_radix_cache (bool, optional): Whether to disable radix cache. Defaults to False. mem_fraction_static (float, optional): The fraction of memory to use. Defaults to 0.88. test_tag (str, optional): The tag to use for the test. Defaults to "". rrcSg|]}|jqSrr(.0adaptorrrr Bz,run_lora_test_one_by_one.. ========== Testing on base '' with backend=, dtype= --- Using prompts cSg|]}|ddqSN2rrlprrrrnF with adaptors:  --- generationcSg|] }|jdur|jqSNr(rkrrrrnM) rb model_typer lora_pathsrerr lora_backendrfrgrhrcrNrbrrrhrcrbrzMax prefill diff (HF vs SRT):zMax decode diff (HF vs SRT):ROUGE-L score: SRT output: HF output:z(Max diff (SRT base vs SRT LoRA prefill):z&Max diff (HF base vs HF LoRA prefill):dz$Prefill logprobs mismatch for base ' ', adaptor 'z ', backend ' ', prompt: 'rw...'z#Decode logprobs mismatch for base 'ROUGE-L score  below tolerance for base ')rrTrappendrprintrrrrforwardrr r r r.tensortop_input_logprobsmaxabstop_output_logprobs output_strsstriprr>allAssertionError)$r`rarbrcrdrerfrgrhri base_pathirrC adaptor_names srt_runner srt_outputssrt_no_lora_outputs hf_runner hf_outputshf_no_lora_outputsrm prefill_tol decode_tol rouge_tol hf_prefill srt_prefillmax_prefill_diff hf_decode srt_decodemax_decode_diffsrt_output_str hf_output_str rouge_scorehf_no_lora_prefillsrt_no_lora_prefillrrrrun_lora_test_one_by_ones             rc Cs|j} dg} } tt|D]} | |j| | dt|j} qdd| D} td|d|jd|d|d d d|Dd | d t| |d |jdd|jD|j|j ||||d }|j ||| d}Wdn1sqwYt| |d |j|d}|j ||d}Wdn1swYt | |d d}|j ||| d}Wdn1swYt | |d d}|j ||d}Wdn1swYtt|D]} |j | }|j | }t|g|gd}td|td|td|td|j | td|j | |j | d|j | dks9J|j | d|j | df|j | d|j | dks\J|j | d|j | dfqdS)a Run lora tests as a batch. For prompt0, prompt1, ..., promptN, we will use adaptor0, adaptor1, ..., adaptorN included in model case, We will then compare the outputs of HF and SRT with LoRA. If number of prompts is larger than number of adaptors, the prompt i will use adaptor i % (number of adaptors). Args: prompts (List[str]): The batch of prompts to test. model_case (LoRAModelCase): The model case to test. torch_dtype (torch.dtype): The torch dtype to use. max_new_tokens (int): The maximum number of new tokens to generate. backend (str): The lora backend to use. disable_cuda_graph (bool, optional): Whether to disable CUDA graph. Defaults to False. disable_radix_cache (bool, optional): Whether to disable radix cache. Defaults to False. mem_fraction_static (float, optional): The fraction of memory to use. Defaults to 0.88. test_tag (str, optional): The tag to use for the test. Defaults to "". rrcSrjrr(rkrrrrnroz*run_lora_test_by_batch..rprqrrrsrtcSrurvrrxrrrrnrzr{r|r}cSr~rr(rkrrrrnr) rbrrrrrrrfrgrhrNrrrrrrzSRT no lora output:zHF no lora output: )rrTrrrrrrrr batch_forwardrrrrr)r`rarbrcrdrfrgrhrirrrrCrrrrrrrrrrrrrrun_lora_test_by_batchs        rcCs(d}t|t|tj|dS)N*)randomseedr. manual_seedcudamanual_seed_all)rrrrensure_reproducibility"s  r)r'z ### Instruction: Write a poem about the transformers Python library. Mention the word "large language models" in that poem. ### Response: The Transformers are large language models, They're used to make predictions on text. r&z Computer science is the study ofzWrite a short story.z+What are the main components of a computer?lora_adapter_pathscCstdt|t|t|gd|d|dgft|t|t|g|d|ddgft|t|t|ggdfgS)NrrrNNN)rrchoice)r`rrrr"create_multiple_batch_test_samplesBs. r torch_nativeT model_casesuse_spec_decodingattention_backendenable_deterministic_inferencecCs$|D] }tD]}d} |j} dd|jD} t| dksJtt| } td| d|dt|s6indd dd d } t| f|d | d | dg|t| d|j d||||d | }tt | |d dd}||t | D]}\}\}}td|dd|d||j || |d}|j || |d}tddd|jDtddd|jDt|j|jD]0\}}|}|}|j}t|g|gd }||krtd|d|d| d|d|d qtd |dd!qqWdn1swYWdn 1s wYqqdS)"N cSrjrr(rlr+rrrrnroz:run_lora_multiple_batch_on_model_cases..r)z. ========== Testing multiple batches on base ' ', dtype=r|NGRAM)speculative_algorithmspeculative_num_draft_tokens'speculative_ngram_min_match_window_size'speculative_ngram_max_match_window_sizer}rrT) rbrrrerr sleep_on_idlerrrfrg)rbrpatch_model_do_sample_falsez --- Running Batch z --- prompts: z, lora_paths: rz SRT outputs:cSg|]}|qSrrrlsrrrrnz HF outputs:cSrrrrrrrrnrrrrrrr --- Batch z Comparison Passed --- ) TORCH_DTYPESrrrrTEST_MULTIPLE_BATCH_PROMPTSrrrrr enumeraterrrr@rr rr)rrrrfrrgrerarbrcrrbatches spec_argsrrrr`rrrsrt_outhf_outsrt_strhf_strrrrrr&run_lora_multiple_batch_on_model_cases}s      rcsFddtdtjffdd }|D] }tD]}|||qqdS)a Test that SRT correctly handles batch splitting with multiple LoRA adapters. When the number of distinct adapters (including None for base model) exceeds max_loras_per_batch, SRT internally splits requests into microbatches. This test validates: 1. SRT can process batches that trigger internal splitting without errors 2. Different adapters don't produce all identical outputs (i.e., at least one output differs, indicating adapters are being applied correctly) Args: model_cases: List of LoRAModelCase configurations to test attention_backend: Attention backend to use disable_cuda_graph: Whether to disable CUDA graph disable_radix_cache: Whether to disable radix cache r)rarbc sdd|jD}t|ksJddd}|j}td|d|dtd gd }|d|d |d gf||d d|d gf||d |d dgf|d|d dgf||d |d |d gf|gd fg}tt||d ||jdd f}t|D]X\}\} } td|d dtd| |j | || d} t | } t| dkrdd| j Dt fddD} | rJd|d|d d| dtd|d dq~WddS1swYdS)NcSrjrr(rrrrrnrozPrun_lora_batch_splitting_equivalence_test.._run_test..zNeed at least z adapters for this test@z- ========== Testing batch splitting on base 'rz ==========rrrr}T) rbrrrerrrrrfrgz --- Batch r|z Adapters: rr)cSsg|]}|qSr)rrrrrrn>sc3s|] }|dkVqdS)rNr)rlout all_outputsrr ?szOrun_lora_batch_splitting_equivalence_test.._run_test..zFEvery output was identical despite using different adapters for base 'z ', batch z : adapters=z). Expected at least one output to differ.rz passed ---) rrrrrrrrrrsetrr)rarbrrcrr` test_casesr batch_idx batch_promptsrrunique_adapters all_identicalrrfrgrerrr _run_tests    "z._run_testN)rr.r1r)rrrfrgrerrarbrrr)run_lora_batch_splitting_equivalence_tests P r)rr)r%NFFr^r_)r%FFr^r_)FrTFTN)rTTN)' dataclassesrtypingrrr.sglang.test.runnersrrsglang.test.test_utilsr dataclassrrfloat16rBACKENDSDEFAULT_PROMPTSCI_LORA_MODELSALL_OTHER_LORA_MODELSCI_MULTI_LORA_MODELSALL_OTHER_MULTI_LORA_MODELSLORA_MODELS_QWEN3Tensorr3r"rNr]rr1r#rrrrrrrrrrrrs         C T   p = c