o ٷi>@sdZddlZddlZddlZddlZddlZddlZddlZddlm Z ddl m Z ddl m Z GdddejZGd d d eZGd d d eZGd ddZdS)z Post-processing hooks N)ZipFile)TarFile) get_loggerc@sPeZdZdZd ddZeejddZejddZ ejd d Z d d Z dS)ExtractorProcessora Abstract base class for extractions from compressed archives. Subclasses can be used with :meth:`pooch.Pooch.fetch` and :func:`pooch.retrieve` to unzip a downloaded data file into a folder in the local data store. :meth:`~pooch.Pooch.fetch` will return a list with the names of the extracted files instead of the archive. Parameters ---------- members : list or None If None, will unpack all files in the archive. Otherwise, *members* must be a list of file names to unpack from the archive. Only these files will be unpacked. extract_dir : str or None If None, files will be unpacked to the default location (a folder in the same location as the downloaded zip file, with a suffix added). Otherwise, files will be unpacked to ``extract_dir``, which is interpreted as a *relative path* (relative to the cache location provided by :func:`pooch.retrieve` or :meth:`pooch.Pooch.fetch`). NcC||_||_dSN)members extract_dir)selfr r r D/home/ubuntu/.local/lib/python3.10/site-packages/pooch/processors.py__init__1 zExtractorProcessor.__init__cCdS)z String appended to unpacked archive folder name. Only used if extract_dir is None. MUST BE IMPLEMENTED BY CHILD CLASSES. Nr r r r r suffix5zExtractorProcessor.suffixcCr)zf Return all the members in the archive. MUST BE IMPLEMENTED BY CHILD CLASSES. Nr )r fnamer r r _all_members>rzExtractorProcessor._all_memberscCr)z This method receives an argument for the archive to extract and the destination path. MUST BE IMPLEMENTED BY CHILD CLASSES. Nr )r rr r r r _extract_fileErz ExtractorProcessor._extract_filec s.jdur |j_n|jtjjddd}tj|j_jdus(js.|}nj}|dvsGtj jrGt fdd|DsVtj jdd  |jg}t jD]6\}}} | D].} tjtjtj|j| jdustfd djDr|tj|| qeq^|S) a Extract all files from the given archive. Parameters ---------- fname : str Full path of the zipped file in local storage. action : str Indicates what action was taken by :meth:`pooch.Pooch.fetch` or :func:`pooch.retrieve`: * ``"download"``: File didn't exist locally and was downloaded * ``"update"``: Local file was outdated and was re-download * ``"fetch"``: File exists and is updated so it wasn't downloaded pooch : :class:`pooch.Pooch` The instance of :class:`pooch.Pooch` that is calling this. Returns ------- fnames : list of str A list of the full path to all files in the extracted archive. Nr)maxsplitrupdatedownloadc3s(|]}tjtjj|VqdSr)ospathexistsjoinr .0mrr r us z.ExtractorProcessor.__call__..T)exist_okc3s"|] }tj|VqdSr) startswithrrnormpathr)relpathr r r"s )r rrsplitrrseprr rrallmakedirsrwalkr%r&anyappend) r ractionpooch archive_dirr fnamesr_filesfilenamer )r&r r __call__Ms:     zExtractorProcessor.__call__)NN) __name__ __module__ __qualname____doc__rpropertyabcabstractmethodrrrr5r r r r rs     rc@,eZdZdZeddZddZddZdS) Unzipa Processor that unpacks a zip archive and returns a list of all files. Use with :meth:`pooch.Pooch.fetch` or :func:`pooch.retrieve` to unzip a downloaded data file into a folder in the local data store. The method/function will return a list with the names of the unzipped files instead of the zip archive. The output folder is ``{fname}.unzip``. Parameters ---------- members : list or None If None, will unpack all files in the zip archive. Otherwise, *members* must be a list of file names to unpack from the archive. Only these files will be unpacked. extract_dir : str or None If None, files will be unpacked to the default location (a folder in the same location as the downloaded zip file, with the suffix ``.unzip`` added). Otherwise, files will be unpacked to ``extract_dir``, which is interpreted as a *relative path* (relative to the cache location provided by :func:`pooch.retrieve` or :meth:`pooch.Pooch.fetch`). cCr)l String appended to unpacked archive folder name. Only used if extract_dir is None. z.unzipr rr r r rz Unzip.suffixcCs6t|d }|WdS1swYdS)(Return all members from a given archive.rN)rnamelist)r rzip_filer r r rs $zUnzip._all_memberscst|dF}|jdurtd|||j|dn!|jD]%td||fdd|D}|j||dqWddSWddS1sNwYdS) o This method receives an argument for the archive to extract and the destination path. rBNz"Unzipping contents of '%s' to '%s')r!Extracting '%s' from '%s' to '%s'cs*g|]}tj|tjr|qSr )rrr%r$)r namememberr r s z'Unzip._extract_file..r r)rr rinfo extractallrC)r rr rDsubdir_membersr rHr rs$    "zUnzip._extract_fileNr6r7r8r9r:rrrr r r r r>   r>c@r=) Untara Processor that unpacks a tar archive and returns a list of all files. Use with :meth:`pooch.Pooch.fetch` or :func:`pooch.retrieve` to untar a downloaded data file into a folder in the local data store. The method/function will return a list with the names of the extracted files instead of the archive. The output folder is ``{fname}.untar``. Parameters ---------- members : list or None If None, will unpack all files in the archive. Otherwise, *members* must be a list of file names to unpack from the archive. Only these files will be unpacked. extract_dir : str or None If None, files will be unpacked to the default location (a folder in the same location as the downloaded tar file, with the suffix ``.untar`` added). Otherwise, files will be unpacked to ``extract_dir``, which is interpreted as a *relative path* (relative to the cache location provided by :func:`pooch.retrieve` or :meth:`pooch.Pooch.fetch`). cCr)r?z.untarr rr r r rr@z Untar.suffixcCsBt|d}dd|DWdS1swYdS)rArBcSsg|]}|jqSr )rGr rLr r r rJsz&Untar._all_members..N)ropen getmembers)r rtar_filer r r rs$zUntar._all_memberscstjdkrinddi}t|dN}|jdur*td|||jd d|i|n%|jD])td||fd d |D}|jd ||d |q-WddSWddS1sbwYdS) rE) filterdatarBNz"Untarring contents of '%s' to '%s'rrFcs,g|]}tj|jtjr|qSr )rrr%rGr$rRrHr r rJs z'Untar._extract_file..rKr ) sys version_inforrSr rrLrMrT)r rr filter_kwargrUrNr rHr rs.   "zUntar._extract_fileNrOr r r r rQrPrQc@sFeZdZdZdeeeedZddddZdd d Z d d Z d dZ dS) Decompressa3 Processor that decompress a file and returns the decompressed version. Use with :meth:`pooch.Pooch.fetch` or :func:`pooch.retrieve` to decompress a downloaded data file so that it can be easily opened. Useful for data files that take a long time to decompress (exchanging disk space for speed). Supported decompression methods are LZMA (``.xz``), bzip2 (``.bz2``), and gzip (``.gz``). File names with the standard extensions (see above) can use ``method="auto"`` to automatically determine the compression method. This can be overwritten by setting the *method* argument. .. note:: To unpack zip and tar archives with one or more files, use :class:`pooch.Unzip` and :class:`pooch.Untar` instead. The output file is ``{fname}.decomp`` by default but it can be changed by setting the ``name`` parameter. .. warning:: Passing in ``name`` can cause existing data to be lost! For example, if a file already exists with the specified name it will be overwritten with the new decompressed file content. **Use this option with caution.** Parameters ---------- method : str Name of the compression method. Can be "auto", "lzma", "xz", "bzip2", or "gzip". name : None or str Defines the decompressed file name. The file name will be ``{fname}.decomp`` if ``None`` (default) or the given name otherwise. Note that the name should **not** include the full (or relative) path, it should be just the file name itself. N)autolzmaxzgzipbzip2r_rarb)z.xzz.gzz.bz2r^cCrr)methodrG)r rcrGr r r rSrzDecompress.__init__c Cs|jdur |d}n tjtj||j}|dvs tj|shtd|||j| |}t |d,}| |}t ||Wdn1sKwYWd|SWd|S1scwY|S)aK Decompress the given file. The output file will be either ``{fname}.decomp`` or the given *name* class attribute. Parameters ---------- fname : str Full path of the compressed file in local storage. action : str Indicates what action was taken by :meth:`pooch.Pooch.fetch` or :func:`pooch.retrieve`: - ``"download"``: File didn't exist locally and was downloaded - ``"update"``: Local file was outdated and was re-download - ``"fetch"``: File exists and is updated so it wasn't downloaded pooch : :class:`pooch.Pooch` The instance of :class:`pooch.Pooch` that is calling this. Returns ------- fname : str The full path to the decompressed file. Nz.decomprz-Decompressing '%s' to '%s' using method '%s'.zw+b) rGrrrdirnamerrrLrc_compression_modulerSshutil copyfileobj)r rr.r/ decompressedmoduleoutput compressedr r r r5Ws,       zDecompress.__call__cCsd}|j|jvr'd|jdt|jd}|jdvr#d||g}t||jdkr^tj|d}||j vrVd |dt|j d}|d vrRd||g}t||j|j |S|j|jS) a Get the Python module compatible with fname and the chosen method. If the *method* attribute is "auto", will select a method based on the extension. If no recognized extension is in the file name, will raise a ValueError. z:To unpack zip/tar archives, use pooch.Unzip/Untar instead.zInvalid compression method 'z'. Must be one of 'z'.>tarzip r^zUnrecognized file extension '>.tar.zip) rcmoduleslistkeysr ValueErrorrrsplitext extensions)r rerror_archivesmessageextr r r res*        zDecompress._compression_module)r^N) r6r7r8r9r_rabz2rrrwrr5rer r r r r]$s+   ,r])r9r;rr{rar_rfrZzipfilertarfilerutilsrABCrr>rQr]r r r r s   tHO