Tensorflow "layer" that categorizes good and bad predictions into a HTML document
git clone https://github.com/sabhiram/tf-eval-reporter.git
cd tf-eval-reporter
python setup.py installThis assumes that you have access to a data provider to get the inputs to process
and the expected outputs, as well as a model to evaluate the inputs against to get
a batch-wide set of results (The ModelFunction below is such an abstraction).
from eval_reporter import EvalReporter
[image, label] = provider.get(['image', 'label'])
input, expected = tf.train.batch(
[image, label],
batch_size=32)
output = ModelFunction(input)
# Create a reporter based on the input, output and expectation tensors.
reporter = EvalReporter(input, output, expected)
# Or more commonly
# reporter = EvalReporter(images, predictions, labels)
# Register the reporter op if you want to enable the report building.
# This is usually done by passing this op in addition to the rest of the
# ops that you want to process during evaluation.
eval_ops = [reporter.get_op()]
slim.evaluation.evaluate_once(
master=...,
checkpoint_path=...,
logdir=...,
num_evals=...,
eval_op=eval_ops,
vairables_to_restore=...)
# Finally, to get out a HTML document of this run...
reporter.write_html_file("eval.html")The above usage is mostly broken and serves as a gentle guide to integrating this into your own deep learning evaluation flow. The idea here is not to have this tied down to a single framework, but rather allow it to accept input of the format [image tensor, prediction tensor, expectation tensor].
- Verify all input tensors are of same width.
- Make the HTML document prettier and easier to filter based on classes.
- Add ability for reporter to map the items in the expected and predicted labels to string labels.