Model analysis in Object Detection #312
Conversation
… - can draw conf_mat"
jyrainer
requested review from
ljj727,
echoyunjin,
ZeroAct,
dustjr1565 and
Jiyong-Jeon
waffle_hub/hub/hub.py
Outdated
| ) | ||
|
|
||
| else: | ||
| pass |
There was a problem hiding this comment.
i think it is better to print a Warning or Info message instead of pass.
There was a problem hiding this comment.
I'll change it to display a warning message using 'warnings'.
waffle_hub/temp_utils/image/io.py
Outdated
| @@ -21,6 +21,18 @@ def save_image(output_path: Union[str, Path], image: Mat, create_directory: bool | |||
| with open(str(output_path), mode="w+b") as f: | |||
| img_arr.tofile(f) | |||
|
|
|||
| def save_images(output_path: Union[str, Path], images: list[Mat], create_directory: bool = False) -> None: | |||
There was a problem hiding this comment.
it seems better to change the function name. Functionally, it appears to merge multiple images into one and save them as a single image. However, based on the function name alone, it looks like it saves each image individually into separate directories.
There was a problem hiding this comment.
I would consider naming it batch_save_images. Thank you for participating in the review!
There was a problem hiding this comment.
I think..
I'm also confused about batch_save_images naming.
How about naming it save_concat_image?
waffle_hub/hub/hub.py
Outdated
| import matplotlib.pyplot as plt | ||
| import numpy as np | ||
| import pandas as pd | ||
| import seaborn as sn |
waffle_hub/hub/hub.py
Outdated
| @@ -1364,23 +1391,86 @@ def evaluating( | |||
| metrics = evaluate_function( | |||
| preds, labels, self.task, len(self.categories), image_size=cfg.image_size | |||
| ) | |||
| # TODO: Confusion matrix visualization functions other than 'OBJECT_DETECTION' and 'CLASSIFICATION' are required. | |||
| if self.task == "OBJECT_DETECTION" or self.task == "CLASSIFICATION": | |||
waffle_hub/hub/hub.py
Outdated
| if value == None: | ||
| continue | ||
| elif isinstance(value, list): | ||
| if len(self.get_category_names()) == len(value) - 1: | ||
| """for object detection confusion matrix""" | ||
| values = [ | ||
| { | ||
| "class_name": cat, | ||
| "value": cat_value, | ||
| } | ||
| for cat, cat_value in zip(self.get_category_names(), value) | ||
| ] | ||
| values.append({"class_name": "background", "value": value[-1]}) | ||
| else: | ||
| values = [ | ||
| { | ||
| "class_name": cat, | ||
| "value": cat_value, | ||
| } | ||
| for cat, cat_value in zip(self.get_category_names(), value) | ||
| ] | ||
| elif isinstance(value, set): | ||
| """It can load the misprediction image info.""" | ||
| values = [] | ||
| pred_list = list(value) | ||
| set_file = io.load_json(getattr(dataset, f"{cfg.set_name}_set_file")) | ||
| for pred in pred_list: | ||
| image_info = dataset.image_dict[set_file[pred]] | ||
| values.append({pred: image_info.file_name}) |
There was a problem hiding this comment.
these if-elif condition looks ambiguous
|
@ZeroAct |
There was a problem hiding this comment.
I think we should also save the Evaluate dataset information. (such as name, root_dir etc..)
| tpfpfn_table: list[float] = None | ||
| fp_images_set: set = None | ||
| fn_images_set: set = None |
There was a problem hiding this comment.
Is it necessary for evaluation.json? In particular, I seems that images set is not needed or saved as separate files.
There was a problem hiding this comment.
I think the same. I think it would be better to separate it into files according to the image index...
There was a problem hiding this comment.
I think getf1, getConfusionMatric can be confused with classification.
You divided the folder structure for object detection, but isn't it necessary in common?
I'll think about this part at the same time as I verify it.
When evaluate in Object Detection task
drawoption. To analyze mispredictions, You can compare the prediction and label through images. Left: Pred / Right: LabelWhen evaluate in Classification task,
Add hub cached properties :
confusionmatrix_file,mis_predict_dir,fp_dirandfn_dir.Add
save_imagesfunction, in utils.image.io.Ref : https://www.notion.so/snuailab/0a2a29ce014f42fb84520057a2eb5500