-
Notifications
You must be signed in to change notification settings - Fork 29.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
benchmark: add csv and plotting script #777
Conversation
This commit adds an `OUTPUT_FORMAT` environment variable option for all benchmark tests that allow either 'csv' or 'default' output. Default output has been left unchanged, and csv output prints out the csv headers along with the csv formatted per-test output, each test also seperated by a newline. It can be used like the following: $ OUTPUT_FORMAT=csv iojs benchmark/common.js http Not specifying the OUTPUT_FORMAT env var will default to 'default'. Specifying a bad value will throw an error.
If anyone wants more examples, here is one for |
I'd love to see performance graphs made public ala http://arewefastyet.com/ and its siblings. |
console.log('%s: %s', heading, value.toFixed(5)); | ||
else if (outputFormat == 'csv') | ||
console.log('%s,%s', heading, value.toFixed(5)); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does the combination silent === true && outputFormat === 'csv'
make sense? If not, you could write this as:
if (!silent && outputFormat == 'default')
// ...
else if (outputFormat == 'csv')
// ...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks - I ended up taking your suggestion about the silent mode though, so this'll be changed anyways in the upcoming commit
Suggestion: replace NODE_BENCH_SILENT with OUTPUT_FORMAT=silent. |
LGTM with some suggestions and one style nit. |
This commit adds an `OUTPUT_FORMAT` environment variable option for all benchmark tests that allow either 'csv' or 'default' output. Default output has been left unchanged, and csv output prints out the csv headers along with the csv formatted per-test output, each test also seperated by a newline. It can be used like the following: $ OUTPUT_FORMAT=csv iojs benchmark/common.js http Not specifying the OUTPUT_FORMAT env var will default to 'default'. Specifying a bad value will throw an error.
394735d
to
2383a6c
Compare
Made the modifications and took your suggestion, @bnoordhuis. I'd like at least one more person to look at the R code though before this is merged since I'm not an R wizard. |
This commit adds a graphing script (in R) for graphing the CSV output of a benchmark. It can be run like this: ``` $ OUTPUT_FORMAT=csv iojs benchmark/http/client-request-body.js > data.csv $ ./benchmark/plot_csv.R data.csv graph.png bytes type ``` This will graph the output to `graph.png`, using the output's `bytes` value as X and the result value for each as Y. Output will be grouped by `type`. Running as the example yields a beautiful graph like this: http://pbrd.co/1vBhUfy.
2383a6c
to
6389c58
Compare
Got a user from the #R IRC channel to give me some tips (thanks ernst). Could you please take another look @bnoordhuis? And I'm unsure whether the IRC user's details should go on the signed commit message. Do you have a verdict on that? |
@iojs/collaborators |
What is the use case for the Also this is for a separate issue, but it would be neat to have JSON output too. |
@mscdex Unsure why anyone would use that, but I didn't want to remove it in an unrelated PR. I just kept it in (with backwards compat) and put it under the new option. JSON output would be awesome. |
@brendanashworth Sorry, missed your earlier comment. LGTM. |
I shalt merge tonight if nobody else comments. |
This commit adds a graphing script (in R) for graphing the CSV output of a benchmark. It can be run like this: ``` $ OUTPUT_FORMAT=csv iojs benchmark/http/client-request-body.js > data.csv $ ./benchmark/plot_csv.R data.csv graph.png bytes type ``` This will graph the output to `graph.png`, using the output's `bytes` value as X and the result value for each as Y. Output will be grouped by `type`. Running as the example yields a beautiful graph like this: http://pbrd.co/1vBhUfy. PR-URL: #777 Reviewed-By: Ben Noordhuis <[email protected]>
This commit adds an `OUTPUT_FORMAT` environment variable option for all benchmark tests that allow either 'csv' or 'default' output. Default output has been left unchanged, and csv output prints out the csv headers along with the csv formatted per-test output, each test also seperated by a newline. It can be used like the following: $ OUTPUT_FORMAT=csv iojs benchmark/common.js http Not specifying the OUTPUT_FORMAT env var will default to 'default'. Specifying a bad value will throw an error. PR-URL: #777 Reviewed-By: Ben Noordhuis <[email protected]>
This pull request consists of two commits:
For each commit, longer and better explanations are in each commit description.
The reasoning behind the new graphing script is that it makes it easier to visualize where the improvements can be made when optimizing, allows for more visual speed comparisons between commits, and that the already-present
plot.R
script hasn't been touched for a long time and doesn't do the same functionality.