o 5tip@sdZddlZddlmZddlmZmZddlZddl m Z ddl m Z ddl m Z ddlmZer8dd lmZeeZed Gd d d e ZdS) a WinML backend for lm-eval-harness with NPU/GPU/CPU support. This backend leverages Windows Machine Learning (WinML) to run models on various hardware backends including NPUs, GPUs, and CPUs. It's particularly useful for running inference on Windows devices with dedicated Neural Processing Units. Example usage: lm_eval --model winml --model_args pretrained=path/to/onnx/model.onnx --tasks hellaswag N)Path) TYPE_CHECKINGAny)tqdm)utils) TemplateLM)register_modelInstancewinmlcs<eZdZdZdZe dHdeeefdeeefdBddfddZ dId ed e dBde de ddf fdd Z dJddZ ddZ defddZdJddZdeddfddZede fddZede dBfdd Zede fd!d"Z  #dKd$ed%e dBd&edee fd'd(Zd)ee defd*d+Zd,edejfd-d.Z / dLd0eeeeefee ee fd1ed2e dBdeeeeffd3d4Z /dMd0ed5d1edeeeeffd6d7Z /dMd0ed5d1edeefd8d9Z /dMd0ed5d1edeefd:d;Z!  < = > /dNd?ed@e dAeedBdBedCedDe dEedefdFdGZ"Z#S)O WindowsMLz WindowsML backend for lm-eval-harness with NPU/GPU/CPU support. This model class provides integration with Windows Machine Learning (WindowsML) to enable evaluation on NPUs and other Windows-optimized hardware. iNarg_dictadditional_configreturncCs:i|pi}|rdd|D}|||di|S)a< Override to properly merge dictionaries and avoid duplicate keyword arguments. Args: arg_dict: Dictionary containing model arguments additional_config: Optional dictionary containing additional configuration Returns: Instance of WindowsML class cSsi|] \}}|dur||qSN).0kvrrH/home/ubuntu/.local/lib/python3.10/site-packages/lm_eval/models/winml.py <sz1WindowsML.create_from_arg_obj..Nr)itemsupdate)clsr r merged_configfiltered_additionalrrrcreate_from_arg_obj*s  zWindowsML.create_from_arg_obj@ pretrained max_length batch_sizemax_batch_sizec st|||_|p|j|_||_||_|dks |dkr2t d|d|dd|_d|_| | | | |tdt|jddS)a2 Initialize WindowsML model. Args: pretrained: Path to ONNX model file or directory containing model files max_length: Maximum sequence length batch_size: Batch size for inference max_batch_size: Maximum batch size for auto-batching rzMWindowsML backend currently only supports batch size 1. Requested batch_size=z, max_batch_size=z. Setting both to 1.zAvailable EP devices: z execution providersN)super__init___validate_dependenciesr _DEFAULT_MAX_LENGTHr!r"r# eval_loggerwarning_fix_winrt_runtime"_register_winml_providers_to_genai"_setup_winml_devices_and_providers_load_and_compile_modelinfolen ep_device_map)selfr r!r"r#kwargs __class__rrr%Cs.   zWindowsML.__init__c Cszddl}||_td|jWnty#}ztd|d}~wwzddl}||_td|jWdStyH}ztd|d}~ww)z Validate that required dependencies are available. Raises: ImportError: If required dependencies are not installed rNzONNX Runtime GenAI version: z]ONNX Runtime GenAI is required for WinML backend. Install with: pip install onnxruntime-genaizONNX Runtime version: zhONNX Runtime is also required for execution provider registration. Install with: pip install onnxruntime)onnxruntime_genaiogr(r. __version__ ImportError onnxruntimeort)r1r6er:rrrr&xs0z WindowsML._validate_dependenciescCsHddlm}tt|dd}|dd}|r"|dSdS)z This function removes the msvcp140.dll from the winrt-runtime package. So it does not cause issues with other libraries. r)metadataz winrt-runtimewinrtz msvcp140.dllN) importlibr<rstr distribution locate_fileexistsunlink)r1r<site_packages_pathdll_pathrrrr*s   zWindowsML._fix_winrt_runtimec CszYddlmmmm}ddlm}m}||jd1|j }| }|D]}| |j|j|jtd|jdq*WdWdS1sRwYWdStyt}ztd|WYd}~d Sd}~wty}ztd |WYd}~d Sd}~ww) z Register Windows ML execution providers to ONNX Runtime GenAI. Returns: True if registration was successful, False otherwise rN)InitializeOptions initialize)optionsz Registered z to ONNX Runtime GenAITzWindows ML import error: Fz&Error registering providers to GenAI: )+winui3.microsoft.windows.ai.machinelearning microsoftwindowsaimachinelearningEwinui3.microsoft.windows.applicationmodel.dynamicdependency.bootstraprGrHON_NO_MATCH_SHOW_UIExecutionProviderCatalog get_defaultfind_all_providersensure_ready_asyncgetr6#register_execution_provider_libraryname library_pathr(r.r8r) Exception)r1r rGrHcatalog providersproviderr;rrrr+s8      z,WindowsML._register_winml_providers_to_genaic Cs$zn|j}i|_|D]}|j}||jvrg|j|<|j||q td|jD]>\}}td||D]/}z|j|j j j }td|j dd|dWq;t yjtd|j ddYq;wq-Wd St y}ztd|td i|_WYd }~d Sd }~ww) zSetup execution providers using Windows ML device enumeration API. This method queries available devices and builds a mapping of execution providers. z%Available execution provider devices:zExecution Provider: z | Vendor: z<16z | Device Type: z<8z | Device Type: Unknownz&Windows ML device enumeration failed: z)Falling back to legacy provider selectionN)r:get_ep_devicesr0ep_nameappendr(r.rOrtHardwareDeviceTypedevicetyperW ep_vendorrYr))r1 ep_devicesrar^rWdevices device_typer;rrrr,sD      z,WindowsML._setup_winml_devices_and_providers model_pathc Cst|}|r|jdkr|j}n|r|}ntd|dz$td||j t ||_ |j |j |_ td||_WdSty\}z td|d|d}~ww) a! Load and optionally compile ONNX model with ONNX Runtime GenAI. Args: model_path: Path to ONNX model file or directory Raises: FileNotFoundError: If model path is not found or invalid Exception: If model loading fails z.onnxz Model path z not found or invalidz,Loading model with ONNX Runtime GenAI from: z?Model and tokenizer loaded successfully with ONNX Runtime GenAIz2Failed to load model with ONNX Runtime GenAI from z: N)ris_filesuffixparentis_dirFileNotFoundErrorr(r.r6Modelr@ genai_model Tokenizergenai_tokenizerrgrYerror)r1rginput_model_pathr;rrrr-s.  z!WindowsML._load_and_compile_modelc Csz$t|jdr#|jj}t|dr#|jr#t|jtr|jdWS|jWSWnty?}ztd|dWYd}~nd}~wwtddS) z{ Get the end-of-text token ID. Returns: End-of-text token ID from the GenAI tokenizer config eos_token_idrzError getting EOS token ID: z, using fallback value 2Nz8Could not determine EOS token ID, using fallback value 2) hasattrrnrsrt isinstancelistrYr(r))r1rsr;rrr eot_token_ids      zWindowsML.eot_token_idcCsz:t|jdr|jj}t|dr|jdur|jWSz|jd}t|dkr,|dWWSWWdSty:YWdSwtyDYdSw)z Get the prefix token ID (typically BOS token). Returns: BOS token ID if available, otherwise None rs bos_token_idNr=r)rvrnrsrzrpencoder/rY)r1rs empty_encrrrprefix_token_id5s$      zWindowsML.prefix_token_idcCsdS)z Get the maximum number of tokens to generate. Returns: Maximum generation tokens (default: 4096) rr)r1rrr max_gen_toksSszWindowsML.max_gen_toksTstringleft_truncate_lenadd_special_tokenscCs2|j|}|durt||kr|| d}|S)af Tokenize string and return token IDs. Args: string: Input string to tokenize left_truncate_len: If provided, truncate from the left to this length add_special_tokens: Whether to add special tokens (note: GenAI tokenizer handles this automatically) Returns: List of token IDs N)rpr{r/)r1rrrencodingrrr tok_encode]s zWindowsML.tok_encodetokenscCs |j|S)z Decode token IDs back to text. Args: tokens: List of token IDs to decode Returns: Decoded text string )rpdecode)r1rrrr tok_decodews zWindowsML.tok_decode input_textc Cszr|j|}t|dkrtdtjdtjdWS|j |j }|j ddd|j |j |}| ||d}tj|tjd}t|jd krO|d}nt|jd krY|}ntd |jtd |jd t|d|WSty}z td|d}~ww)a Run inference using ONNX Runtime GenAI to get full logits sequence. Args: input_text: Input text string to compute logits for Returns: Logits matrix of shape (seq_len, vocab_size) where logits[i] contains predictions for the token at position i+1 given tokens[0:i+1] Raises: Exception: If inference fails rz+No tokens to process; returning empty array)rrdtyperFr! do_samplelogitsruzUnexpected logits shape: zFull logits shape: z for z input tokenszGenAI inference failed: N)rpr{r/r(r)npemptyfloat32r6GeneratorParamsrnset_search_options Generator append_tokens get_outputarrayshape ValueErrordebugrYrq) r1r input_tokensparams generatorfull_logits_tensor logits_array logits_matrixr;rrr$_run_genai_inference_for_full_logitss2      z.WindowsML._run_genai_inference_for_full_logitsFrequests disable_tqdm override_bscCstd)a Stub implementation - not used since we override loglikelihood directly. WindowsML uses the GenAI tokenizer and overrides loglikelihood to work with text inputs directly, avoiding tokenization round-trip issues. Args: requests: List of tokenized requests disable_tqdm: Whether to disable progress bar override_bs: Optional batch size override Returns: Empty list (method not used) zUWindowsML overrides loglikelihood() directly and does not use _loglikelihood_tokens())NotImplementedError)r1rrrrrr_loglikelihood_tokensszWindowsML._loglikelihood_tokensr c Csg}t||ddD]O}|j\}}t|dkr|dq z||}|rG||}t|dkrB|jdurB|d|jkrB|dd}t|} n|jdurP|jgng}t|} ||} | | d} t| dkrn|dWq |j|j} | j ddd |j |j| } | d| }t|dkr| t j |t jd n|jdur| t j |jgt jd d }d }| D]q}| }t j |t jd }t|jd kr|ddddf}nt|jdkr|dddf}n|}t |}t ||}|t t |}|t||}|t|7}tt |t|krd}| t j t|gt jd q|||fWq tyY}ztd|ddl}t||dWYd}~q d}~ww|S)a[ Compute log-likelihood using ONNX Runtime GenAI with teacher forcing. Args: requests: List of instances containing (context, continuation) text pairs disable_tqdm: Whether to disable progress bar Returns: List of tuples containing (log_likelihood, is_greedy) for each request zComputing log-likelihoodsdisabledescr)TNrrFrrrTrruz!Failed to compute loglikelihood: )rF)rargsr/r_rr}r6rrnrrrrrint32 get_logitsrrmaxexplogsumintfloatargmaxrYr(r) tracebackr format_exc)r1rrresultsrequestcontext continuation full_text context_enc context_len full_tokenscontinuation_tokensrgencontext_tokenstotal_ll greedy_oktokrr next_logits max_logit exp_logits log_sum_explog_probr;rrrr loglikelihoods                   zWindowsML.loglikelihoodc sg}t||ddD]d}|jd}tttjtj|||j|j dd}g}|D]\}} | |} | | } | | | fq(ddl m fdd|D} |j| d d } d d| D} t| }| ||jd |f|q |S) a Compute rolling log-likelihood for perplexity using ONNX Runtime GenAI. Uses sliding windows to handle sequences longer than max_length. Args: requests: List of instances containing text sequences disable_tqdm: Whether to disable progress bar Returns: List of sum of log-likelihood values for each request z!Computing rolling log-likelihoodsrrr) token_list prefix_token max_seq_lenrr c s&g|]\}}di||fdidqS)rr) request_typedoc argumentsidxr<r)rctxcontr rr ksz3WindowsML.loglikelihood_rolling..T)rcSsg|]}|dqS)rr)rxrrrr|sloglikelihood_rolling)rrrxmaprmake_disjoint_windowget_rolling_token_windowsrr}r!rr_lm_eval.api.instancer rr cache_hook add_partial)r1rrloglikelihoodsrrrolling_token_windowswindow_requestsrr context_textcontinuation_textwindow_instances string_nll total_nllrr rr>sD        zWindowsML.loglikelihood_rollingc Cs|sgSg}t||ddD]^}|j\}}|d|j}|dg}|dd} |dd} |d d } |d d } z|j|||| | | | d } || Wq tyk}ztd||dWYd}~q d}~ww|S)a, Generate text until stopping criteria using ONNX Runtime GenAI. Args: requests: List of generation requests with context and generation kwargs disable_tqdm: Whether to disable progress bar Returns: List of generated text strings zGenerating textrr~until temperaturertop_p?top_k2rF)rrrrzGeneration failed for request: r=N) rrrUr~_run_genai_generationr_rYr(r))r1rrrrr gen_kwargsr~rrrrrgenerated_textr;rrrgenerate_untils8        zWindowsML.generate_untilrrrprompt max_tokensstop_sequencesrrrrc sz|j|j}|r!|dkr!|jt|t|t|t|ddn |jt|ddddd|j|j|} |j|} | | g} | s| | rOn;| d} t | t | } t | | kr| | }| |t | |krrn|r|j| tfdd|Drn| rF| r|j| }|r|D]}||r|dt | }|WSq|WSWd Sty}ztd |WYd}~d Sd}~ww) a2 Run text generation using ONNX Runtime GenAI. Args: prompt: Input text to generate from max_tokens: Maximum number of tokens to generate stop_sequences: List of sequences that will stop generation temperature: Sampling temperature (0 = greedy, higher = more random) top_p: Nucleus sampling threshold top_k: Top-k sampling parameter do_sample: Whether to use sampling (if False, uses greedy decoding) Returns: Generated text string rr)r!rrrrepetition_penaltyrrc3s|]}|vVqdSrr)rstop_seq current_textrr sz2WindowsML._run_genai_generation..Nr=zGenAI generation error: )r6rrnrrrrrpr{ris_donegenerate_next_token get_sequencer/r_ranyendswithrYr(rq)r1rrrrrrrrrrgenerated_tokens output_tokensbase new_tokenrrr;rrrrsj           zWindowsML._run_genai_generationr)rrr)rN)NT)FN)F)rNrrrF)$__name__ __module__ __qualname____doc__r' classmethoddictr@rrrr%r&r*boolr+r,r-propertyryr}r~rxrrrndarrayrtuplerrrrrr __classcell__rrr3rr s   5 #',   6   r J 2  r )rloggingpathlibrtypingrrnumpyrrlm_evalrlm_eval.api.modelrlm_eval.api.registryrrr getLoggerrr(r rrrrs