o
Ni, @ sj d Z ddlZddlmZ ddlZddlmZmZmZm Z m
Z
ddl Z ddl
m mZ ddlZdZdZdZdgZeeeee
e f f Zeee eeee eeeef f f f f ZG d d
d
eZ d ed eeeef fd
dZ dee d efddZ deded efddZ ej j!" d efddZ#G dd deZ$ej j!" dd Z%d d Z&d"d d Z'd"d d!Z(dS )#zyUtil to generate the dataset documentation content.
Used by tensorflow_datasets/scripts/documentation/build_catalog.py
N)futures)DictListTupleUnionSet 2 zFhttps://github.com/tensorflow/datasets/tree/master/tensorflow_datasets
wmt_translatec @ sN e Zd ZdZejdZdZdd Z dd Z
dejj d e fd
d Z
d d
ZdS )VisualizationDocUtilz?Small util which generate the path/urls for the visualizations.zvisualization/z7https://storage.googleapis.com/tfds-data/visualization/c C s |j jddd S )N/-z.png)info full_namereplaceselfbuilder r o/home/ubuntu/.local/lib/python3.10/site-packages/tensorflow_datasets/scripts/documentation/document_datasets.py _get_name9 s z VisualizationDocUtil._get_namec C s | j | | S N)BASE_URLr r r r r get_url< s z VisualizationDocUtil.get_urlr returnc C s | |}d| dS )zReturns the ![]()
html tag.z
)r )r r urlr r r get_html_tag? s
z!VisualizationDocUtil.get_html_tagc C s$ t j| j| |}tjj|S r ) ospathjoin BASE_PATHr tfiogfileexists)r r filepathr r r has_visualizationD s z&VisualizationDocUtil.has_visualizationN)__name__
__module__ __qualname____doc__tfdscoregcs_pathr r r r DatasetBuilderstrr r& r r r r r 3 s r r r c C s^ t jj| std| d| d^}}}t|dks J |r(tt|nd}|||fS )z=Extracts the `(ds name, config, version)` from the full_name.z Parsing builder name string zd failed.The builder name string must be of the following format:`dataset_name[/config_name]/version`r ) r+ r,
registered is_full_name
ValueErrorsplitlennextiter)r ds_nameoptional_configversionconfigr r r _split_full_nameI s
r=
full_namesc C s< t dd }| D ]}t|\}}}|| | | q |S )z5Creates the dict `d['dataset']['config']['version']`.c S
t tS r ) collections defaultdictsetr r r r Y
z%_full_names_to_dict..)r@ rA r= add)r> full_names_dictr r9 r< r; r r r _full_names_to_dictV s rG
registered_dsstable_version_dsc C s t dd }| D ]A}||v rF| | D ]1}||| v r>| | | D ]}||| | v r4d|| | |< q!d|| | |< q!qd|| |< qq d||< q |S )z>Computes the nightly dict from the registered and stable dict.c S t dd S )Nc S r? r )r@ rA boolr r r r rC g rD z7_build_nightly_dict....r@ rA r r r r rC f z%_build_nightly_dict..FTrL )rH rI
nightly_dsdatasetr< r; r r r _build_nightly_dict` s
rP c C sl t jjd} tjj| d}| }W d n1 s w Y t
t jj }t
|}t
||S )z$Loads (and caches) the nightly dict.zstable_versions.txtrN)r+ r, utils
get_tfds_pathr! r" r# GFileread
splitlinesrG r2 list_full_namesrP ) version_pathfstable_versionsrH rI r r r _load_nightly_dict~ s
r[ c @ s e Zd ZdZdd Zdeejje f de
fddZ dejjde
fdd Z dejjd
e de
fd d Z
dejjde
fd
dZdZdS )NightlyDocUtilz Small util to format the doc.c C s t | _d S r )r[
_nightly_dict)r r r r __init__ s zNightlyDocUtil.__init__r r c C s( t |tjjr |j}n|}| j| du S )z%Returns `True` if the builder is new.T)
isinstancer+ r, r. namer] )r r builder_namer r r is_builder_nightly s z!NightlyDocUtil.is_builder_nightlyc C s2 t |jj\}}}| |rdS | j| | du S )z$Returns `True` if the config is new.FT)r= r r rb r] )r r r9 r< _r r r is_config_nightly s
z NightlyDocUtil.is_config_nightlyr; c C s@ t |jj\}}}| |s| |rdS | j| | | du S )z%Returns `True` if the version is new.FT)r= r r rb rd r] )r r r; r9 r< rc r r r is_version_nightly s z!NightlyDocUtil.is_version_nightlyc s fdd | j |j S )z9Returns True if any of the builder/config/version is new.c s@ t | tr| S t | trt fdd| D S td| )Nc 3 s | ]} |V qd S r r ).0xreducer r s z=NightlyDocUtil.has_nightly..reduce..z Invalid nightly_dict value: )r_ rK dictanyvaluesAssertionError)valuerh r r ri s
z*NightlyDocUtil.has_nightly..reduce)r] r` r r rh r has_nightly s zNightlyDocUtil.has_nightlyzbnights_stayN)r' r( r) r* r^ r r+ r, r. r/ rK rb rd re rp iconr r r r r\ s&
r\ c C s^ t jjd| }tjj|d }| }W d n1 s w Y t j
j |ddgdS )zReturns mako.lookup.Template object to use to render documentation.
Args:
tmpl_name: string, name of template to load.
Returns:
mako 'Template' instance that can be rendered.
z*scripts/documentation/templates/%s.mako.mdrQ Nr/ trim)default_filters) r+ r, rR rS r! r" r# rT rU makolookupTemplate) tmpl_name tmpl_pathtmpl_f tmpl_contentr r r get_mako_template s
r{ c s t d j fdd}g } jr1tjtd}t|| j}W d n1 s,w Y t d}t
}|j ||t d
}t d}|j ||d
}|d
| }|S ) z9Doc string for a single builder, with or without configs.zDocument builder %s...c s t j j| dS )Nr< )r+ r r` r| r r r rC s z)document_single_builder.. max_workersNrO )r config_builders
visu_doc_utilnightly_doc_util
schema_org)r r r
)printr` builder_configsr ThreadPoolExecutorWORKER_COUNT_CONFIGSlistmapBUILDER_CONFIGSr{ r render_unicoder\ strip) r get_config_builderr tpooltmplr out_strschema_org_tmplschema_org_out_strr r} r document_single_builder s: r c C s t dd }| sdd t D } tdt| tjtd}| tj
| }W d n1 s2w Y td |D ] }|j j }|
d }d
|v rMq=|}|D ]}|| }qQ|| q=|d }|S ) z5Get all builders organized by module in nested dicts.c S rJ )Nc S r? r )r@ rA r r r r r rC rD z?make_module_to_builder_dict....rL r r r r rC rM z-make_module_to_builder_dict..c S s g | ]}|t vr|qS r )BUILDER_BLACKLIST)rf r` r r r
s z/make_module_to_builder_dict..z0Creating the vanilla builders for %s datasets...r~ Nz>Vanilla builders built, constructing module_to_builder dict....testingtensorflow_datasets)r@ rA r+
list_buildersr r6 r r WORKER_COUNT_DATASETSr r __class__r( r5 append) datasetsmodule_to_builderr buildersr module_namemodulescurrent_mod_ctrmodr r r make_module_to_builder_dict s.
r c C s t d t| }tt| }tt}|D ]@}tj || }t|dd d}t
d}t j t
d}|t|}W d n1 sDw Y dd t||D }|||< qt
d
} | }
|
|gS ) a Create dataset documentation string for given datasets.
Args:
datasets: list of datasets for which to create documentation.
If None, then all available datasets will be used.
Returns:
- overview document
- a dictionary of sections. Each dataset in a section is represented by a
tuple (dataset_name, is_manual_dataset, string describing the datasets
(in the MarkDown format))
z"Retrieving the list of builders...c S s | j S r )r` )br r r rC % s z"dataset_docs_str..)keyrO r~ Nc S s g | ] \}}|j |j|fqS r )r` MANUAL_DOWNLOAD_INSTRUCTIONS)rf r builder_docr r r r ) s
z$dataset_docs_str..catalog_overview)r r sortedr keysr@ rA r! nestflattenr{ r r r r r zipr lstrip) r r sections section_docssectionr unused_r builder_docsr r r r r dataset_docs_str s$
r r ))r* r@
concurrentr r typingr r r r r mako.lookuprt tensorflow.compat.v2compatv2r! r r+ r r r r r/
FullNamesDictrK NightlyDictobjectr r= rG rP r, rR memoizer[ r\ r{ r r r r r r r s@ ,
4
!