o -wi47@sddlZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl m Z ddl m Z mZmZmZmZddlmZeeZdddedDd Zd ed dfd d Zded eeddffddZd ed efddZded eejddffddZGdddZ Gddde Z!deded efddZ"ej# d4dededed ed ee ddff d!d"Z$deded efd#d$Z%d%e d&e d dfd'd(Z&d%e d&e d dfd)d*Z'd5d%ed&ed,e(d dfd-d.Z)ded eefd/d0Z*d5ded1e(d efd2d3Z+dS)6N)Path)IOAnyBinaryIO GeneratorOptional)StrPathccs|]}t|VqdSN)chr).0irU/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/wandb/sdk/lib/filesystem.py sr z:"*<>?|dir_namereturnc Csdz tj|ddWdSty}zt|d|d}~wty1}zt|d|d}~ww)zCreate `dir_name` and any parent directories if they don't exist. Raises: FileExistsError: if `dir_name` exists and is not a directory. PermissionError: if `dir_name` is not writable. Texist_okz exists and is not a directoryNz is not writable)osmakedirsFileExistsErrorPermissionError)rerrrmkdir_exists_oksrpathccs\t|}tj|\}}tj||VtD]}||vr+||d}tj||VqdS)zYield variations of `path` that may exist on the filesystem. Return a sequence of paths that should be checked in order for existence or create-ability. Essentially, keep replacing "suspect" characters until we run out. -N)strrr splitdrivejoinPROBLEMATIC_PATH_CHARSreplace)rroottailcharrrrpath_fallbacks%s r&c Cst|D]P}z,tj|ddt|t|kr"td|d|dt|tr.t|WS|WSttfy=Yqt yT}z |j dkrJWYd}~qd}~wwt d|d) zCreate `dir_name`, removing invalid path characters if necessary. Returns: The path to the created directory, which may not be the original path. Trz Creating 'z' instead of ''NzUnable to create directory ') r&rrrloggerwarning isinstance ValueErrorNotADirectoryErrorOSErrorerrno)rnew_namerrrrmkdir_allow_fallback4s "  r1ccsFtj|s dSt|D]}|rt|jEdHq|VqdS)zEYield a directory entry for each file under a given path (recursive).N)rrisdirscandiris_dirfiles_in)rentryrrrr5Is r5c@s6eZdZdZdeddfddZd ddZd d d ZdS) WriteSerializingFilez1Wrapper for a file object that serializes writes.frNcCst|_||_dSr ) threadingLocklockr8selfr8rrr__init__Ws  zWriteSerializingFile.__init__cOsD|jz|jj|i||jW|jdS|jwr )r;acquirer8writeflushrelease)r=argskargsrrrr@[s  zWriteSerializingFile.writecCs2|jz |jW|jdS|jwr )r;r?r8closerBr=rrrrEcs  zWriteSerializingFile.closerN)__name__ __module__ __qualname____doc__rr>r@rErrrrr7Ts  r7csBeZdZdeddffdd Zd fdd Zd fdd ZZS) CRDedupedFiler8rNcstj|dd|_dS)N)r8)superr>_buffr< __class__rrr>ls zCRDedupedFile.__init__cstd|}g}|D]%}|dddkr!|r|n|jr!d|_|dd}|r/||q |jr:|d|j|rA||_td|ddS)Ns |  rMr ) resplitpoprOappendinsertrNr@r )r=datalinesretlinerPrrr@ps"    zCRDedupedFile.writecs"|jr t|jtdSr )rOrNr@rErFrPrrrEszCRDedupedFile.closerG)rHrIrJrr>r@rE __classcell__rrrPrrLksrL source_path target_pathc Cst|}t|dd}tj| pt|jt|jk}t|j}|rntj|\}}tj t ||}zt ||Wn)t ygzt||t ||Wnt yd}zt d|d}~wwYnwt||||S)aCopy source_path to target_path, unless it already exists with the same mtime. We liberally add write permissions to deal with the case of multiple users needing to share the same cache or run directory. Args: source_path: The path to the file to copy. target_path: The path to copy the file to. Returns: The path to the copied file (which may be different from target_path). T)warnz%Unable to overwrite '{target_path!s}'N)typesystem_preferred_pathrrisfilestatst_mtimest_moderWr r1shutilcopy2rchmod)r`ra return_type need_copypermissions_plus_writer file_namerrrrcopy_or_overwrite_changeds.      rprmoderCkwargsc ost|}|jjdddd|vr|rt|dd|vrHd|vrH|j|g|Ri| }|VWddS1sAwYdStj|jdi}t||j }d|vs_d |vri|rit |||j|g|Ri|}|V| t |Wdn1swYd|vrt ||t |n ||WddSWddS1swYdS) a/Open a file, ensuring any changes only apply atomically after close. This context manager ensures that even unsuccessful writes will not leave a "dirty" file or overwrite good data, and that all temp data is cleaned up. The semantics and behavior are intended to be nearly identical to the built-in open() function. Differences: - It creates any parent directories that don't exist, rather than raising. - In 'x' mode, it checks at the beginning AND end of the write and fails if the file exists either time. Tparentsrx already existsrq+Ndira)rresolveparentmkdirexistsropentempfileTemporaryDirectorynamerirjrArfsyncfilenolinkunlinkr")rrrrCrsr8tmp_dirtmp_pathrrr safe_opens8      "rcCs~t|}|jjdddtj|jd }t|t|jd}t ||| |Wd|S1s8wY|S)aCopy a file atomically. Copying is not usually atomic, and on operating systems that allow multiple writers to the same file, the result can get corrupted. If two writers copy to the same file, the contents can become interleaved. We mitigate the issue somewhat by copying to a temporary file first and then renaming. Renaming is atomic: if process 1 renames file A to X and process 2 renames file B to X, then X will either contain the contents of A or the contents of B, not some mixture of both. Trtryz.tmpN) rr|r}r~rrr with_suffixrirjr")r`ra output_pathrrrrr safe_copys    r existing_pathnew_pathc Csddl}d}t|d1}t|d}||||Wdn1s'wYWddSWddS1s?wYdS)z;Create a reflink to `existing_path` at `new_path` on Linux.rNi @rbzwb+)fcntlrioctlr)rrrFICLONEt_fl_frrr_reflink_linuxs Prc Csz tjddd}Wn(ttfy1}zttjkr t|ts tjddd}WYd}~nd}~wwz|j}Wn t yDttj dwtj tj tj f|_ tj |_|t|t|t drnt}t|t||dS)Nz libc.dylibT) use_errnoz/usr/lib/libSystem.dylibz+'clonefile' is not available on this systemr)ctypesCDLLFileNotFoundErrorr. get_errnor/ENOENTr+ clonefileAttributeErrorENOTSUPc_char_pc_intargtypesrestyperfsencodestrerror)rrclibrrerrrrr_reflink_macoss&   rF overwritec Cstdkr t}ntdkrt}n ttjdtt|}t|}| rC|s6t |dt d|d| |jjdddz|||WdSty}zfd |d |d}|jtjtjfvrutd |||jtjkrtd |||jtjkrtd |||jtjkrtd|||jtjtjfvrttjd|||jtjkrtd||d}~ww)a9Create a reflink to `existing_path` at `new_path`. A reflink (reflective link) is a copy-on-write reference to a file. Once linked, the file and link are both "real" files (not symbolic or hard links) and each can be modified independently without affecting the other; however, they share the same underlying data blocks on disk so until one is modified they are "zero-cost" copies. Reflinks have all the functionality of copies, so we should use them wherever they are supported if we would otherwise copy a file. (This is not particularly radical-- GNU `cp` defaults to `reflink=auto`, using it whenever available) However, support for them is limited to a small number of filesystems. They should work on: - Linux with a Btrfs or XFS filesystem (NOT ext4) - macOS 10.13 or later with an APFS filesystem (called clone files) Reflinks are also supported on Solaris and Windows with ReFSv2, but we haven't implemented support for them. Like hard links, a reflink can only be created on the same filesystem as the target. LinuxDarwinzreflinks are not supported on rwzOverwriting existing file .Trtzfailed to create reflink from z to zInsufficient permissions; zFile not found; z Cannot link across filesystems; zCannot reflink a directory; z&Filesystem does not support reflinks; zCannot link file ranges; N)platformsystemrrr.r/rrr|rrr)r*rr}r~EPERMEACCESrrrEXDEVr,EISDIRIsADirectoryError EOPNOTSUPPEINVAL)rrrlink_fnrbase_msgrrrreflinksN        rcCs<t|D]}tj|rt|trt|S|SqdS)aLook for variations of `path` and return the first found. This exists to support former behavior around system-dependent paths; we used to use ':' in Artifact paths unless we were on Windows, but this has issues when e.g. a Linux machine is accessing an NTFS filesystem; we might need to look for the alternate path. This checks all the possible directories we would consider creating. N)r&rrrr+r)rdestrrr check_existsYs  rrbcCsdtdkr|Stj|\}}|rd|vrtd|d||dd}t|t r0t |S|S)zReplace ':' with '-' in paths on Windows. Args: path: The path to convert. warn: Whether to warn if ':' is replaced. Windows:zReplacing ':' in z with '-'r) rrrrrr)r*r"r+r)rrbheadr$rrrrrdgs  rd)rq)F), contextlibrr/loggingrrrVrirr9pathlibrtypingrrrrrwandb.sdk.lib.pathsr getLoggerrHr)r ranger!rrr&r1DirEntryr5r7rLrpcontextmanagerrrrrboolrrrdrrrrsR    *  2 >