Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose CancellationToken as passed-in argument to tests #3069

Open
julealgon opened this issue Nov 29, 2024 · 5 comments
Open

Expose CancellationToken as passed-in argument to tests #3069

julealgon opened this issue Nov 29, 2024 · 5 comments

Comments

@julealgon
Copy link

I've started a migration towards xunit.v3 and immediately ran into the new xUnit1051 warning about propagating the built-in TestContext.Current.CancellationToken value to underlying calls for test cancellation.

This is all well and good, I like the new capability, but I'd like to propose that the engine also supports propagation of this token as an argument on the test method itself.

I'd like to go from this:

[Fact]
public async Task MyTestAsync()
{
    await myService.DoSomethingAsync(TestContext.Current.CancellationToken);
}

To this:

[Fact]
public async Task MyTestAsync(CancellationToken cancellationToken)
{
    await myService.DoSomethingAsync(cancellationToken);
}

I strongly believe the latter is much more natural and idiomatic in the .NET ecosystem than fetching the token from an ambient object. This approach also leads to simpler overall code as there is a lot less boilerplate involved.

I would assume such a feature should be straightforward to implement by checking the signature of the method right before performing the call: if it has an argument of type CancellationToken, pass the TestContext.Current.CancellationToken through that value.

In case the method also takes in inline data (for [Theory] methods), the token could still be provided using a similar strategy that, say, ASP.NET Core uses for binding the token parameter to controller actions: as long as there is one argument of type CancellationToken anywhere in the arguments list, it passes the token through that, regardless of parameter order.

This argument should of course be ignored in discovery processes (so it wouldn't show up in, say, Visual Studio Test Explorer).

@bradwilson
Copy link
Member

I'd like to propose that the engine also supports propagation of this token as an argument on the test method itself
[...]
This approach also leads to simpler overall code as there is a lot less boilerplate involved.

We're going to have to agree to disagree.

Adding features like this increases both the complexity of the implementation, and the need for users to understand which method parameters are fulfilled from which sources and why. You could make an argument for passing every value from TestContext in as constructor arguments, or as test method arguments, and in my opinion, this way lies madness.

I argue that it's simpler to let users know about TestContext and then they can explore the availability of that information via Intellisense, rather than trying to guess which 17 things could be magically added to a method or constructor instead.

@julealgon
Copy link
Author

I'd like to propose that the engine also supports propagation of this token as an argument on the test method itself
[...]
This approach also leads to simpler overall code as there is a lot less boilerplate involved.

We're going to have to agree to disagree.

Interesting. I would've thought that passing cancellation token via a parameter would be universally seen as the more standard thing to do whenever cancellation is involved in any operation, so I wasn't expecting pushback on that idea. Fair enough though.

Adding features like this increases both the complexity of the implementation,

Absolutely. Not denying that.

...and the need for users to understand which method parameters are fulfilled from which sources and why.

Again, if you consider that every other common framework in the ecosystem can provide a CancellationToken argument if you add one (WebAPI, gRPC, SignalR, etc) then I would argue that people actually expect it to work for tests as well, and it not working is the surprising factor. So in a sense, not supporting passing the token in the method would be a principle of least astonishment violation.

You could make an argument for passing every value from TestContext in as constructor arguments, or as test method arguments, and in my opinion, this way lies madness.

I would not argue in favor of that, no. I could see maybe passing the whole TestContext object as (another) parameter would be interesting, so you could have, if you wanted to:

[Fact]
public async Task MyTestAsync(TestContext testContext, CancellationToken cancellationToken)
{
    await myService.DoSomethingAsync(cancellationToken);
}

But in that scenario I would still advocate for the CancellationToken parameter to be passed separately.

I argue that it's simpler to let users know about TestContext and then they can explore the availability of that information via Intellisense, rather than trying to guess which 17 things could be magically added to a method or constructor instead.

Now you are making an argument against something I did not propose at all. I've not mentioned 17 things, I mentioned just 1, super idiomatic one, that basically every other framework in the entire .NET ecosystem already leverages.


On a related note, I checked some of the extension interfaces and was wondering how one could make this work via a custom extension, if so desired. I saw that some of the interfaces that expose test method calling arguments rely on arrays, meaning you cannot "add arguments" to the call in the extension.

Is that really the case, or is there an extension point in the framework that would allow one to pass CancellationToken as a direct argument to all test methods? If so, I'd like to explore that idea. If not, I would be fine creating a separate issue to make that a possibility by making whatever extension hooks there are more flexible in that front.

@bradwilson
Copy link
Member

Now you are making an argument against something I did not propose at all.

You're arguing for your request.

I'm arguing against the number of requests that would come after it. "Why is CancellationToken so special? Why can't you just add this one more thing?"

We have never supported anything coming to a method other than theory data. A simple followup question is: What if your theory data includes a CancellationToken? Which CancellationToken should get injected? Whichever answer you give is likely to conflict with someone else who would expect the opposite.

Keeping things easier to reason about is better for everybody.

On a related note, I checked some of the extension interfaces and was wondering how one could make this work via a custom extension, if so desired.

The method arguments are given to the test case, so it's the test case discoverer which is responsible for supplying them (or, in the case of XunitDelayEnumeratedTheoryTestCase, the method arguments are determined at run time).

The real problem you have here is that the test context is contextual. The value for the cancellation token at any given point in the pipeline is different, so grabbing it during (for example) test case discovery (where test method arguments are usually determined) would be useless anyways. The only time you could effectively grab it is immediately before invoking the test method. That means the only real override point is by overriding TestRunner.InvokeTest and replacing this code with code that is able to resolve the extra method argument(s):

var parameterCount = ctxt.TestMethod.GetParameters().Length;
var valueCount = ctxt.TestMethodArguments is null ? 0 : ctxt.TestMethodArguments.Length;
if (parameterCount != valueCount)
{
ctxt.Aggregator.Add(
new InvalidOperationException(
string.Format(
CultureInfo.CurrentCulture,
"The test method expected {0} parameter value{1}, but {2} parameter value{3} {4} provided.",
parameterCount,
parameterCount == 1 ? "" : "s",
valueCount,
valueCount == 1 ? "" : "s",
valueCount == 1 ? "was" : "were"
)
)
);
}
else
{
var logEnabled = TestEventSource.Log.IsEnabled();
if (logEnabled)
TestEventSource.Log.TestStart(ctxt.Test.TestDisplayName);
try
{
var result = ctxt.TestMethod.Invoke(testClassInstance, ctxt.TestMethodArguments);
var valueTask = AsyncUtility.TryConvertToValueTask(result);
if (valueTask.HasValue)
await valueTask.Value;
}
catch (TaskCanceledException) { }
finally
{
if (logEnabled)
TestEventSource.Log.TestStop(ctxt.Test.TestDisplayName);
}
}

@julealgon
Copy link
Author

Now you are making an argument against something I did not propose at all.

You're arguing for your request.

I'm arguing against the number of requests that would come after it. "Why is CancellationToken so special? Why can't you just add this one more thing?"

The answer is precisely that "yes, CancellationToken is special", the same way it is special everywhere else in the ecosystem.

People seeing CancellationToken being passed and concluding from that that passing anything else the same way would be a sensible request are probably not very used to .NET at this point.

We have never supported anything coming to a method other than theory data.

That's fair, I understand it. You also didn't ever support cancellation, now you do. New things are added to frameworks.

A simple followup question is: What if your theory data includes a CancellationToken? Which CancellationToken should get injected? Whichever answer you give is likely to conflict with someone else who would expect the opposite.

This is a fair question, even though passing a cancellation token via TheoryData would be super weird to me. I don't have the answer to how one would disambiguate it, but I'm sure there would be many possibilities, the simplest probably being "consider the last CancellationToken in the method the one to pass the operation cancellation on", after TheoryData arguments have been bound.


Thanks for the code sample. I'll take a look at that.

@bradwilson
Copy link
Member

I'm afraid we're at the end of this discussion, because we simply disagree, and neither is convincing the other.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants