I feel like this article really wanted seven reasons because there are a few that feel like duplicates and most of them distil down to ‘they’re signs of real problems and not fixing problems will waste time and money elsewhere’ (which I agree with).
The thing missing from here is a cost-benefit analysis. By definition, a flaky test is one that is hard to debug: the failure does not deterministically appear and so you can’t reproduce it locally very easily. That means that the cost of fixing them is high. Is it higher than the cost of dealing with all of the things that this article? If so, you should mark your flaky tests as flaky and not make them block PRs (and possibly remove them: they might be flaky because they’re testing the wrong thing). The important thing is that this cost-benefit will change over time and the cost of having flaky tests will go up (especially the reputational damage if customers see the flaky behaviour in production).
These justifications ring true and sometimes one is in a position of having to defend investing time in improving test suites - this provides some ammo. The marketing at the end for tools that will solve your problems I would skip over - I suppose inevitable that this is marketing their products.
One thing implicit in the title and justifications is that you can delete tests that aren’t providing value. Flaky tests can be a strong evidence that you can consider this. There might be value in writing a similar test, and my experience is that you may as well write that new test from scratch instead of carrying forward code and strategies and assumptions baked in to a flaky test.
Flaky tests are just bad tests and more often than not they test nothing or at least not what they seem to test. Just set them to pending and create a ticket. Also flaky Test can be an indication of bad architecture
I feel like this article really wanted seven reasons because there are a few that feel like duplicates and most of them distil down to ‘they’re signs of real problems and not fixing problems will waste time and money elsewhere’ (which I agree with).
The thing missing from here is a cost-benefit analysis. By definition, a flaky test is one that is hard to debug: the failure does not deterministically appear and so you can’t reproduce it locally very easily. That means that the cost of fixing them is high. Is it higher than the cost of dealing with all of the things that this article? If so, you should mark your flaky tests as flaky and not make them block PRs (and possibly remove them: they might be flaky because they’re testing the wrong thing). The important thing is that this cost-benefit will change over time and the cost of having flaky tests will go up (especially the reputational damage if customers see the flaky behaviour in production).
Ah yes. The seven reasons I don’t like Selenium.
These justifications ring true and sometimes one is in a position of having to defend investing time in improving test suites - this provides some ammo. The marketing at the end for tools that will solve your problems I would skip over - I suppose inevitable that this is marketing their products.
One thing implicit in the title and justifications is that you can delete tests that aren’t providing value. Flaky tests can be a strong evidence that you can consider this. There might be value in writing a similar test, and my experience is that you may as well write that new test from scratch instead of carrying forward code and strategies and assumptions baked in to a flaky test.
ANYWAY. I rate this article 2/5 😉
Woops, I left out an important part. I was trying to say that it’s implied that you misy fix the tests. You don’t! You can delete them.
On this topic there is a nice video (https://www.youtube.com/watch?v=H4iNuufAe_8) in which one of the rr developers explains how rr can be used to find such bugs.
Flaky tests are just bad tests and more often than not they test nothing or at least not what they seem to test. Just set them to pending and create a ticket. Also flaky Test can be an indication of bad architecture