-
Notifications
You must be signed in to change notification settings - Fork 49
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor tests using Theory #108
base: master
Are you sure you want to change the base?
Refactor tests using Theory #108
Conversation
73b71d4
to
4fd8ff7
Compare
You don't need to switch to Theory to do what you are doing. It is very doable with XUnit to pass arguments to functions for various tests. The problem with this change is about setup and teardown (ctor + dispose). If you separate the tests into multiple 1 liners, you run setup and teardown for each and every one of those, because that's the whole point, and that can add significant slowdowns to test running. In this set of tests, perhaps it won't be too noticeable, but there is a real chance that these changes could 3x-10x test runtime. So, with these changes I would be concerned about showing things down annoyingly, and also unnecessarily dropping XUnit. If you can address/justify those concerns, I would be open to this continuing. |
But with calling methods you won't get individual errors, since the test still fails at the first error.
Sure, as I already stated we should not parse entire files for each individual test for example.
Does this address the concerns so far? |
Small addition: I found some explanation about SharedContext in X-Unit, which could solve performance issues: https://xunit.net/docs/shared-context I need to read through, but my first impression ist, that this can address all performance issues. |
Actually it turns out what you are doing is the XUnit way of doing this, I think I was thinking of nunit's approach, or some other framework. So I support the path you are on. For sure this makes things easier to isolate for failures. And we can check performance at the end. Thanks! |
4fd8ff7
to
b6dd9da
Compare
b6dd9da
to
f1ac796
Compare
Splitted to single commits for each changed method now, that way a review should be possible. Updating the other methods will follow soon. |
Ready with ParserTests, but searching for a better solution for ParseDotenvFile()-refactoring. But at least the refactoring of this class can be reviewed, then I can use that feedback for refactoring the next classes. |
Found the solution for ParseDotenvFileShouldParseContents: Using MemberData combined with TheoryData removes all these dictionaryRefs. It is still quite messy due to the long multiline strings, but that's also true for the old solution. |
While refactoring There is no good solution for showing which InlineData or TheoryData is under tests (no link back to the code-line). While that is acceptable for most of the tests, there are cases (mainly with TheoryData I think) where this is quite inconvenient. That's why I added a workaround, adding a new Type But still it reduces the code so much, that this tradeoff should be okay imho. |
31534ed
to
83e777e
Compare
83e777e
to
eb5810a
Compare
I just had a look through the other TestClasses now: HelperTests and EnvReaderTests seem hard to rewrite, and 'm not sure how valuable this is.
I would postpone EnvTests and EnvConfigurationTests, because there are potentially huge simplifications when working on #105 (decoupling Parsers and EnvVars) first. What do you think so far? |
Hey sorry for the delay, I did not realize you were waiting on me for next steps. Honestly, this is hard for me to confirm that you retained all existing tests correctly, but if this all passes, it looks broadly correct, and I do like the improvements. I would in fact encourage not doing more than this at this time and instead stop here and call it done, if indeed you are done with these changes. So, I am fine with merging as-is if you are complete on this set of changes. |
While doing major changes (eg changing from Sprache to Superpower) I had some trouble with testing, because the current Fact-Methods do fail at the first failing case. That way it is hard to check whether you fixed something and just broke a different case, or if it does not fix the intended case at all.
Additionally you always have to care about whether the testcases within a method are really "the same", or if for example some check "AtEnd" and some do not.
That's where Theories come into action.
They reduce many of the test-methods to one-liners, which prevents you from merging different aspects of testing.
Instead it enforces to have a precise test-statements which should be tested with different inputs.
Additionally it reports each single failing input instead of just failing at the first of them.
Is that something you would support?
It is not finished, and of course some scenarios have to be handled differently, we should not parse an entire file for each single test for example.
It's not really possible to check that request via git-compare at the moment, too many changes which are not recognized in the correct order.
If there is an interest in having this, I'd split this commit into single method-change-commits to enable a proper review.
And it is of course still a draft, it is not even fully finished for ParserTests.