o poi @sddlZddlZddlZddlZddlZdededefddZdeededeefd d Zd ed ed ee eefddfddZ    dd ededededdf ddZ dS)Nfile_urlfolderreturncCstj|}tj||}tj|rtd|d|tj|dd}t |d}| |j Wd|S1s6sz._replace_remote_with_local..rrNz..w) rr dirnamereplacelensplitseprr r r) rr(r) relt_pathdepthfopenr%urlfpathpath_upfwrrr_replace_remote_with_local*s     "r; docs/sourcefetched-s3-assets*.rst5https?://[-a-zA-Z0-9_]+\.s3\.[-a-zA-Z0-9()_\\+.\\/=]+ assets_folder file_patternretrieve_patternc Cstjtj|d|dd}|std|d|ddSt||d}|s2td |d |dStj||}tj|dd g}t t |D]%\}} td |d t |d| t | |} | | tj|| fqH|D]} t| ||qpdS)aSearch all URL in docs, download these files locally and replace online with local version. Args: docs_folder: the location of docs related to the project root assets_folder: a folder inside ``docs_folder`` to be created and saving online assets file_pattern: what kind of files shall be scanned retrieve_pattern: pattern for reg. expression to search URL/S3 resources z**T) recursivez no files were listed in folder "z" and pattern ""N)rz"no resources/assets were match in z for )exist_okz >> downloading (/z): )globrr r r rr'infomakedirs enumeratesetr1rappendr;) r(r@rArBrurls target_folderpairs_url_fileir7rr8rrrfetch_external_assetsBs$   rQ)r<r=r>r?) rGr rr!rstrrlistr'tupler;rQrrrrs. &