o -wi"@sXddlZddlmZmZmZmZddlmZddlZddl m Z Gdddeej Z dS)N)AnyDictListOptional)Callback) telemetryc sHeZdZdZdeedeedededdf fdd Zd&d ee ee fddfd d Z d&d e d ee ee fddfddZ ejd&d ee ee fddfddZej d&d e d ee ee fddfddZdeeddfddZdeeddfddZ d'dedededdfdd Z ! " d(deded#eeeddfd$d%ZZS))WandbEvalCallbacka Abstract base class to build Keras callbacks for model prediction visualization. You can build callbacks for visualizing model predictions `on_epoch_end` that can be passed to `model.fit()` for classification, object detection, segmentation, etc. tasks. To use this, inherit from this base callback class and implement the `add_ground_truth` and `add_model_prediction` methods. The base class will take care of the following: - Initialize `data_table` for logging the ground truth and `pred_table` for predictions. - The data uploaded to `data_table` is used as a reference for the `pred_table`. This is to reduce the memory footprint. The `data_table_ref` is a list that can be used to access the referenced data. Check out the example below to see how it's done. - Log the tables to W&B as W&B Artifacts. - Each new `pred_table` is logged as a new version with aliases. Example: ```python class WandbClfEvalCallback(WandbEvalCallback): def __init__(self, validation_data, data_table_columns, pred_table_columns): super().__init__(data_table_columns, pred_table_columns) self.x = validation_data[0] self.y = validation_data[1] def add_ground_truth(self): for idx, (image, label) in enumerate(zip(self.x, self.y)): self.data_table.add_data(idx, wandb.Image(image), label) def add_model_predictions(self, epoch): preds = self.model.predict(self.x, verbose=0) preds = tf.argmax(preds, axis=-1) data_table_ref = self.data_table_ref table_idxs = data_table_ref.get_index() for idx in table_idxs: pred = preds[idx] self.pred_table.add_data( epoch, data_table_ref.data[idx][0], data_table_ref.data[idx][1], data_table_ref.data[idx][2], pred, ) model.fit( x, y, epochs=2, validation_data=(x, y), callbacks=[ WandbClfEvalCallback( validation_data=(x, y), data_table_columns=["idx", "image", "label"], pred_table_columns=["epoch", "idx", "image", "label", "pred"], ) ], ) ``` To have more fine-grained control, you can override the `on_train_begin` and `on_epoch_end` methods. If you want to log the samples after N batched, you can implement `on_train_batch_end` method. data_table_columnspred_table_columnsargskwargsreturnNcsltj|i|tjdurtdtjtjd }d|j_Wdn1s)wY||_ ||_ dS)Nz>You must call `wandb.init()` first before using this callback.)runT) super__init__wandbrErrorrcontextfeaturekeras_wandb_eval_callbackr r )selfr r r r tel __class__m/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/wandb/integration/keras/callbacks/tables_builder.pyrQs   zWandbEvalCallback.__init__logscCs$|j|jd|||dSN) column_names)init_data_tabler add_ground_truthlog_data_tablerrrrron_train_begines  z WandbEvalCallback.on_train_beginepochcCs&|j|jd||||dSr)init_pred_tabler add_model_predictionslog_pred_tablerr$rrrr on_epoch_endms  zWandbEvalCallback.on_epoch_endcCt|jjd)aAdd ground truth data to `data_table`. Use this method to write the logic for adding validation/training data to `data_table` initialized using `init_data_table` method. Example: ```python for idx, data in enumerate(dataloader): self.data_table.add_data(idx, data) ``` This method is called once `on_train_begin` or equivalent hook. z.add_ground_truthNotImplementedErrorr__name__r"rrrr usz"WandbEvalCallback.add_ground_truthcCr*)aAdd a prediction from a model to `pred_table`. Use this method to write the logic for adding model prediction for validation/ training data to `pred_table` initialized using `init_pred_table` method. Example: ```python # Assuming the dataloader is not shuffling the samples. for idx, data in enumerate(dataloader): preds = model.predict(data) self.pred_table.add_data( self.data_table_ref.data[idx][0], self.data_table_ref.data[idx][1], preds, ) ``` This method is called `on_epoch_end` or equivalent hook. z.add_model_predictionsr+r(rrrr&sz'WandbEvalCallback.add_model_predictionsrcCstj|dd|_dS)a Initialize the W&B Tables for validation data. Call this method `on_train_begin` or equivalent hook. This is followed by adding data to the table row or column wise. Args: column_names: (list) Column names for W&B Tables. T)columnsallow_mixed_typesN)rTable data_tablerrrrrrs z!WandbEvalCallback.init_data_tablecCstj|d|_dS)a Initialize the W&B Tables for model evaluation. Call this method `on_epoch_end` or equivalent hook. This is followed by adding data to the table row or column wise. Args: column_names: (list) Column names for W&B Tables. )r.N)rr0 pred_tabler2rrrr%s z!WandbEvalCallback.init_pred_tablevaldatasetval_datanametype table_namecCsNtj||d}||j|tjdusJtj|||||_dS)aLog the `data_table` as W&B artifact and call `use_artifact` on it. This lets the evaluation table use the reference of already uploaded data (images, text, scalar, etc.) without re-uploading. Args: name: (str) A human-readable name for this artifact, which is how you can identify this artifact in the UI or reference it in use_artifact calls. (default is 'val') type: (str) The type of the artifact, which is used to organize and differentiate artifacts. (default is 'dataset') table_name: (str) The name of the table as will be displayed in the UI. (default is 'val_data'). r8N) rArtifactaddr1r use_artifactwaitgetdata_table_ref)rr7r8r9 data_artifactrrrr!s  z WandbEvalCallback.log_data_table evaluation eval_dataaliasescCsPtjdusJtjdtjjd|d}||j|tjj||p#dgddS)a+Log the W&B Tables for model evaluation. The table will be logged multiple times creating new version. Use this to compare models at different intervals interactively. Args: type: (str) The type of the artifact, which is used to organize and differentiate artifacts. (default is 'evaluation') table_name: (str) The name of the table as will be displayed in the UI. (default is 'eval_data') aliases: (List[str]) List of aliases for the prediction table. Nrun__predr:latest)rD)rrr;idr<r3 log_artifact)rr8r9rD pred_artifactrrrr'sz WandbEvalCallback.log_pred_table)N)r4r5r6)rBrCN)r- __module__ __qualname____doc__rstrrrrrfloatr#intr)abcabstractmethodr r&rr%r!r' __classcell__rrrrr sdF $"     r) rQtypingrrrrtensorflow.keras.callbacksrr wandb.sdk.librABCrrrrrs