o xi{+@sdZddlZddlZddlZddlZddlmZddlZ ddl m Z ddl Z ddl m Z ddlmZddlmZdd Zd d Zd d ZddZddZddZdS)a:Prodigy integration for W&B. User can upload Prodigy annotated datasets directly from the local database to W&B in Tables format. Example usage: ```python import wandb from wandb.integration.prodigy import upload_dataset run = wandb.init(project="prodigy") upload_dataset("name_of_dataset") wandb.finish() ``` N)deepcopy)Image)util) test_missing) telemetrycCsNtjddd}tjdddt|dr%|jj|dddd d }t|}|Sd S) zCreate a named entity visualization. Taken from https://github.com/wandb/wandb/blob/main/wandb/plots/named_entity.py. spacyzKpart_of_speech requires the spacy library, install with `pip install spacy`requireden_core_web_mdhpart_of_speech requires `en_core_web_md` library, install with `python -m spacy download en_core_web_md`docsentTF)stylepageminifyjupyterN)r get_modulerdisplacyrenderwandbHtml)r rhtml wandb_htmlrU/home/ubuntu/.local/lib/python3.10/site-packages/wandb/integration/prodigy/prodigy.py named_entity!s   rcCsTt|}|D]\}}t|tjjrt||i|||<qt||||<q|S)z@Return a new dictionary by merging two dictionaries recursively.)ritems isinstance collectionsabcMappingmergeget)dict1dict2resultkeyvaluerrrr"9s r"cCst|D]\}}|D]\}}||vrst|trWt|dkr-t|dtr-t|||<q t|dkrCt|dttfsCt|||<q ||i||<t||||||<q t|trli||<t|g|||||<q t|||<q ||}t|trt|dkrt|dtr|durt|||<q t|dkrt|dttfs|durt|||<q ||i||<t||||||<t |||||<q t|tri||<t|g|||||<t |||||<q |durt|||<q q|S)z7Get a schema of the dataset's structure and data types.rN) enumeraterrlistlentypedictappend get_schemar")list_data_dictstructarray_dict_types_iitemkv cur_structrrrr/FsT       7r/cCs |D]\}}||vr6t|tr"||vr"i||<t||||qt|tr0||vr0d||<q|||<qt||trtt||dkoMt||dt oct||dkobt||dttf }|rs||D]}t|||qjqt||trt||||qdS)zStandardize all rows/entries in dataset to fit the schema. Will look for missing values and fill it in so all rows have the same items and structure. Nr)rrr- standardizer*r+)r4 structurer2r5r6 conditionsub_itemrrrr8s,  "& r8c Cst|}t|j}d|jvrd|jvr|dd|vr"|dtj|d}|jdd}tj d d d }|j d gd }t |D]\}}d|vrd|vrd|d<||d} g} d|vr|ddur|dD] } d| vrd| vrd| vr| | d| d| d} | | qg| | _ t| d|d<d|vr8d|d<d|vr8|ddur8tj|djdv} d|dvod|dv}| rzttj|d}t||d<Wnctjjytd|ddd|d<YnJw|r/|ddd}zt|}t|}t|}t||d<Wn$tjj y.td|ddd|d<Yn wt|d|d<t|!}|j"|q@|S)zCreate a W&B Table. - Create/decode images from URL/Base64 - Uses spacy to translate NER span data to visualizations. spanstext spans_visualimage image_visual)columnsrecords)orientr r rner)disableNstartendlabelr )httphttpszdata:z;base64z Image URL z is invalid.zbase64,zBase64 string )#pd DataFramer*rAr.rTableto_dictrrloadr) char_spanentsrurllibparseurlparseschemeropenrequesturlopenerrorURLErrortermwarnsplitbase64 b64decodeioBytesIObinasciiErrorvaluesadd_data)datatable_dfrA main_tablematrixr nlpr3documentdocrRspancharspanisurlisbase64imimgb64msgbuf values_listrrr create_tablest                   rvc Cstjdur tdtjtjd }d|j_Wdn1swYtjddd}| }| |}g}t |i|}t |D] \}}t ||||q@t|} t|| itd|d dS) zUpload dataset from local database to Weights & Biases. Args: dataset_name: The name of the dataset in the Prodigy database. Nz2You must call wandb.init() before upload_dataset())runTzprodigy.components.dbzY`prodigy` library is required but not installed. Please see https://prodi.gy/docs/installrzProdigy dataset `z ` uploaded.)rrw ValueError wb_telemetrycontextfeatureprodigyrrconnect get_datasetr/r)r8rvlogtermlog) dataset_nametel prodigy_dbdatabaserfr2schemai_dtablerrrupload_datasets$    r)__doc__r^collections.abcrr`rScopyrpandasrLPILrrrwandb.plot.utilsr wandb.sdk.librryrr"r/r8rvrrrrrs$      @& T