o Ơi+@s0ddlZddlZddlZddlZddlZddlZddlZddlZddl m Z ddl m Z m Z ddlZddlZeeZdZdZde fddZdefd d Zd edefd d ZdefddZdefddZdefddZde fddZdedede fddZde efddZ dedefddZ!defdd Z"dS)!N)Path)AnyOptionalcp310z 0.17.0.post1returncCs dtjvrtjStjddS)Nz.nvr)torch __version__splitr r I/home/ubuntu/.local/lib/python3.10/site-packages/torch_tensorrt/_utils.pysanitized_torch_versions r csDtjdr ddlddl}|}dtfdd|DSdS)Nlinuxrz&.*libnvinfer_builder_resource_win.so.*c3s|] }|VqdS)N)match).0libre target_libr r 'sz2check_cross_compile_trt_win_lib..F)sysplatform startswithrdllistany)r loaded_libsr rr check_cross_compile_trt_win_libs r min_versionc Csz= min_version, False otherwise Example: >>> if is_tensorrt_version_supported("10.8.0"): ... # Use FP4 features ... pass tensorrt_rtxTr)VersiontensorrtNr)metadataF)trt _package_namepackaging.versionrrmoduleshasattrboolr importlibr version ImportError ValueError)rrmoduler r r r is_tensorrt_version_supported+s    r,cCtjdvr dSdS)N)))r/TFrcudaget_device_capabilityr r r r is_tegra_platformIr4cCr-)N)) rTFr1r r r r is_thorOr5r7c Cst}t}t}d|vrtddS|dkr%d|vs(tr/tddSz)tj j }|dur@t dWdSt t |d \}}|d krVt d WdSWd Stys}ztd |WYd}~dSd}~ww)aS Checks if the current platform supports TensorRT-LLM plugins for the NCCL backend. Returns: bool: True if supported, False otherwise. Unsupported: - Windows platforms - Jetson/Orin/Xavier (aarch64 architecture + 'tegra' in platform release) - Thor devices - CUDA 13 not supported windowszCTensorRT-LLM plugins for NCCL backend are not supported on Windows.Faarch64tegrazfTensorRT-LLM plugins for NCCL backend are not supported on Jetson/Orin/Xavier (Tegra) or Thor devices.NzTThis pytorch build does not support CUDA, please reinstall pytorch with CUDA support. zeCUDA 13 is not currently supported for TRT-LLM plugins. Please install pytorch with CUDA 12.x supportTzFailed to detect CUDA version: )rsystemlowermachinereleaseloggerinfor7rr(r2errormapintr Exceptionwarning)r=r?r@ cuda_versionmajorminorer r r is_platform_supported_for_trtllmUs@  rLcCst}ttd|S)Ntorch_tensorrt_)getpassgetuserrtempfile gettempdir)usernamer r r _cache_rootsrSplatform_systemplatform_machinecCstdtd|d|S)Ntrtllm_)rS_TENSORRT_LLM_VERSION_)rTrUr r r _extracted_dir_trtllms rYc Cs>t}t}dtdtdtd|d|d }t|}t||}d|vr.dnd}|dd |}|r@t |S|j j d d d |j d d d |sd }||}zt d |dtj||t dWn\tjjy} zt d| jd|d| jWYd} ~ n=d} ~ wtjjy} zt d|d| jWYd} ~ nd} ~ wty} zt d| WYd} ~ nd} ~ wwzddl} Wnty} ztdd} ~ wwz%| |} | |t d|Wdn 1swYWnSty)} zt d|d| td|| d} ~ w| jyC} z t d| td| d} ~ wty\} z t d| td| d} ~ wwz|jd dt d |Wnty} zt d!|d| WYd} ~ nd} ~ ww|st d"|dSt |S)#u Returns the path to the TensorRT‑LLM shared library, downloading and extracting if necessary. Args: platform (str): Platform identifier (e.g., 'linux_x86_64') Returns: Optional[str]: Path to shared library or None if operation fails. z tensorrt_llm--rWz.whlr z!libnvinfer_plugin_tensorrt_llm.soz"libnvinfer_plugin_tensorrt_llm.dll tensorrt_llmlibsT)parentsexist_okz%https://pypi.nvidia.com/tensorrt-llm/z Downloading z ...z3Download succeeded and TRT-LLM wheel is now presentz HTTP error z when trying to download z: Nz"URL error when trying to download zLocal file write error: rz@zipfile module is required but not found. Please install zipfilezExtracted wheel to zWheel file not found at z(Failed to find downloaded wheel file at z!Invalid or corrupted wheel file: z=Downloaded wheel file is corrupted or not a valid zip archivez)Unexpected error while extracting wheel: z8Unexpected error during extraction of TensorRT-LLM wheel) missing_okzDeleted wheel file: zCould not delete wheel file z/Plugin library not found at expected location: )!rr=r>r?rX_WHL_CPYTHON_VERSIONrSrYexistsstrparentmkdirrAdebugurllibrequest urlretrieverC HTTPErrorcodereasonURLErrorOSErrorzipfiler)ZipFile extractallFileNotFoundError RuntimeError BadZipFilerFunlinkrG) rTrUwheel_filename wheel_path extract_dir lib_filenameplugin_lib_pathbase_url download_urlrKrnzip_refr r r download_and_get_plugin_lib_paths       " r}ryc Cs<zt|}td|Wn3tyB}z'dt|vr,tjd|d|d|dn tjd|d|dWYd}~d Sd}~wwztjtjg|j _ tj |j _ Wnt yl}ztjd |dWYd}~d Sd}~wwz| dd r|td Wd StdWd Sty}ztjd|dWYd}~d Sd}~ww)z Loads and initializes the TensorRT-LLM plugin from the given shared library path. Args: plugin_lib_path (str): Path to the shared TensorRT-LLM plugin library. Returns: bool: True if successful, False otherwise. z$Successfully loaded plugin library: libmpiz6Failed to load libnvinfer_plugin_tensorrt_llm.so from z , got error zf (hint: libmpi.so is a necessary dependency; ensure that OpenMPI or MPICH is installed on your system))exc_infoz;. Ensure the path is correct and the library is compatible.NFz4Unable to initialize the TensorRT-LLM plugin librarys tensorrt_llmz,TensorRT-LLM plugin successfully initializedTz4TensorRT-LLM plugin library failed in initializationzDException occurred during TensorRT-LLM plugin library initialization)ctypesCDLLrArBrmrbrGc_void_pc_char_pinitTrtLlmPluginsargtypesc_boolrestypeAttributeErrorrF)ryhandle e_os_errore_plugin_unavailablee_initialization_errorr r r !load_and_initialize_trtllm_pluginsR      rcCsXtsdStjd}|rt|Stjdddv}|s%tddSt}t|S)as Attempts to load the TensorRT-LLM plugin and initialize it. Either the env variable TRTLLM_PLUGINS_PATH can specify the path Or the user can specify USE_TRTLLM_PLUGINS as either of (1, true, yes, on) to download the TRT-LLM distribution and load it Returns: bool: True if the plugin was successfully loaded and initialized, False otherwise. FTRTLLM_PLUGINS_PATHUSE_TRTLLM_PLUGINS0)1trueyesonzNeither TRTLLM_PLUGIN_PATH is set nor is it directed to download the shared library. Please set either of the two to use TRT-LLM libraries in torchTRT) rLosenvirongetrr>rArGr})ryuse_trtllm_pluginr r r load_tensorrt_llm_for_nccl(s  r)#rrNloggingrrrrPurllib.requestrfpathlibrtypingrrrr!r getLogger__name__rAr`rXr r&rrbr,r4r7rLrSrYr}rrr r r r s4  5\5