o 8wi8@sldZddlmZddlmZdeeeeefdefddZ Gdd d e Z Gd d d e Z Gd d d Z dS)a Hashing function for dataset keys using `hashlib.md5` Requirements for the hash function: - Provides a uniformly distributed hash from random space - Adequately fast speed - Working with multiple input types (in this case, `str`, `int` or `bytes`) - Should be platform independent (generates same hash on different OS and systems) The hashing function provides a unique 128-bit integer hash of the key provided. The split name is being used here as the hash salt to avoid having same hashes in different splits due to same keys )Union)insecure_hashlib hash_datareturncCsPt|ttfr |St|tr|dd}nt|trt|}nt||dS)z| Returns the input hash_data in its bytes form Args: hash_data: the hash salt/key to be converted to bytes \/zutf-8) isinstancebytes bytearraystrreplaceintInvalidKeyErrorencode)rrM/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/datasets/keyhash.py _as_bytes&s    rcs eZdZdZfddZZS)rz6Raises an error when given key is of invalid datatype.csDd|_d|dt||_d|_t|j|j|jdS)Nz7 FAILURE TO GENERATE DATASET: Invalid key type detectedz Found Key z of type z- Keys should be either str, int or bytes type)prefixtypeerr_msgsuffixsuper__init__)selfr __class__rrr@s"zInvalidKeyError.__init____name__ __module__ __qualname____doc__r __classcell__rrrrr=srcs"eZdZdZdfdd ZZS)DuplicatedKeysErrorz(Raise an error when duplicate key found.cs||_||_||_d|_t|dkrdd|d||_ndd|dddt|dd||_|rs