o پi"@sddlZddlZddlZddlmZmZddlmZddlm Z m Z m Z ddl Z ddl mZddlmZeeZdde eded efd d Zd ed efd dZeGdddZeGdddZGdddeZGdddeZdS)N)ABCabstractmethod) dataclass)AnyListOptional)envs) HostKVCache token_ids prior_hashreturnc Cspt}|r|t||D]#}t|tr(|D] }||jddddqq||jddddq|S)NlittleF) byteordersigned) hashlibsha256updatebytesfromhex isinstancetupleto_bytes hexdigest)r r hashertelemrX/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/mem_cache/hicache_storage.py get_hash_strs rhash_strcCs&t|ddd}|dkr|dS|S)zConvert SHA256 hex string to signed 64-bit integer for events. Takes first 16 hex characters (64 bits) and converts to signed int64 range. Nll)int)r uint64_valrrrhash_str_to_int64"sr$c@sZeZdZUeed<eed<eed<eed<eed<eed<eeed<dZee ed <dS) HiCacheStorageConfigtp_ranktp_sizepp_rankpp_size is_mla_modelis_page_first_layout model_nameN extra_config) __name__ __module__ __qualname__r"__annotations__boolrstrr-dictrrrrr%/s  r%c@s2eZdZUdZeeeed<dZee ed<dS)HiCacheStorageExtraInfoN prefix_keysN extra_info) r.r/r0r7rrr3r1r8r4rrrrr5;s r5c @seZdZdZdefddZ d$deedej de e d ee fd d Z d$deedej de e d ee fd d Ze  d%dede ede ed ej dBfddZe  d%deede ede ed eej dBeBfddZe   d&dede ede ede ed e f ddZe   d&deede ede ede ed e f ddZeded e fddZ d$deede e d efddZd'd d!Zd"d#ZdS)(HiCacheStoragez HiCacheStorage is a class that provides a generic key-value interface for storing and retrieving KV cache. It abstracts the underlying storage mechanism, allowing different implementations to be used. mem_pool_hostcCs ||_dSr6)r:)selfr:rrrregister_mem_pool_hostI z%HiCacheStorage.register_mem_pool_hostNkeys host_indicesr8r cCdS)zx Retrieve values for multiple keys. Returns a list of booleans indicating success for each key. Nrr;r>r?r8rrr batch_get_v1L zHiCacheStorage.batch_get_v1cCr@)zu Store multiple key-value pairs. Returns a list of booleans indicating success for each key. NrrArrr batch_set_v1XrCzHiCacheStorage.batch_set_v1keytarget_location target_sizescCr@)zs Retrieve the value associated with the given key. Returns None if the key does not exist. Nr)r;rErFrGrrrgetd zHiCacheStorage.gettarget_locationscCr@)zl Retrieve values for multiple keys. Returns a list of tensors or None for each key. Nrr;r>rJrGrrr batch_getrrIzHiCacheStorage.batch_getvaluecCr@)z Store the value associated with the given key. Returns True if the operation was successful, False otherwise. Nr)r;rErMrFrGrrrset zHiCacheStorage.setvaluescCr@)zz Store multiple key-value pairs. Returns True if all operations were successful, False otherwise. Nr)r;r>rPrJrGrrr batch_setrOzHiCacheStorage.batch_setcCr@)zr Check if the key exists in the storage. Returns True if the key exists, False otherwise. Nrr;rErrrexistsszHiCacheStorage.existscCs0tt|D] }|||s|Sqt|S)z Check if the keys exist in the storage. return the number of consecutive existing keys from the start. Can be overridden by subclasses for more efficient implementation. )rangelenrS)r;r>r8irrr batch_existss zHiCacheStorage.batch_existscCdSr6rr;rrrclearzHiCacheStorage.clearcCrXr6rrYrrr get_statsr[zHiCacheStorage.get_statsr6)NNNNN)r N)r.r/r0__doc__r r<rr3torchTensorrr5r2rBrDrrrHr"rLrNrQrSrWrZr\rrrrr9As            r9c @seZdZ ddedefddZdedefdd Z dded ejd e e dejd Bfd dZ dde ede ejd e e de ejd BfddZ ddede e d e e d e e def ddZ dde ede e de e d e e def ddZdedefddZdefddZd S) HiCacheFile /tmp/hicachestorage_config file_pathcCstjp||_|j|j|j|jf\}}}}|r d| dnd}|r+d||_ n d|d|d||_ t j |jsS|dkrUt |jtd|jdSdSdS)N-/_rz)Created HiCacheFile storage directory at )r'SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIRrHrdr&r'r,r*joinsplit config_suffixospathrSmakedirsloggerinfo)r;rcrdr&r'r,r*rrr__init__s  zHiCacheFile.__init__rEr cCs ||jSr6)rlrRrrr_get_suffixed_keyr=zHiCacheFile._get_suffixed_keyNrFrGcCs||}tj|j|d}z>||}t|ddd$}t| t j  }|||kr;td|Wd|WS1sGwY|WStyatd|dYdSw)N.binrbr) bufferingzShort read for zFailed to fetch z from HiCacheFile storage.)rsrmrnrjrdnumel element_sizeopen memoryviewviewr_uint8 contiguousnumpyreadintoIOErrorFileNotFoundErrorrpwarning)r;rErFrG tensor_pathexpectedfbufrrrrHs"   zHiCacheFile.getr>rJcs&fddt||pdgt|DS)Ncsg|] \}}||qSr)rH).0rErFrYrr s z)HiCacheFile.batch_get..)ziprUrKrrYrrLs zHiCacheFile.batch_getrMc Cs||rtd|ddS||}tj|j|d}z|j t j d |WdStyO}ztd|d|WYd}~dSd}~ww) NzKey z already exists. Skipped.Trt)dtypezFailed to save tensor z: F)rSrpdebugrsrmrnrjrdr}r{r_r|r~tofile Exceptionerror)r;rErMrFrGrerrrrNs  zHiCacheFile.setrPcCs*t||D] \}}|||sdSqdS)NFT)rrN)r;r>rPrJrGrErMrrrrQs  zHiCacheFile.batch_setcCs,||}tj|j|d}tj|S)Nrt)rsrmrnrjrdrS)r;rErrrrrSs  zHiCacheFile.existsc Csz$t|jD]}tj|j|}tj|rt|qtdWdSt y?}zt d|WYd}~dSd}~ww)Nz+Cleared all entries in HiCacheFile storage.Tz%Failed to clear HiCacheFile storage: F) rmlistdirrdrnrjisfileremoverprqrr)r;filenamerdrrrrrZs   zHiCacheFile.clear)rbr6r])r.r/r0r%r3rrrsr_r`rrrHrrLr2rNrQrSrZrrrrrast       rar6)rloggingrmabcrr dataclassesrtypingrrrr_sglang.srt.environr%sglang.srt.mem_cache.memory_pool_hostr getLoggerr.rpr"r3rr$r%r5r9rarrrrs$      x