Xunit has the option to mark tests with data attributes. The CodeRush test runner seems to work fine with this unless you are having multiple of these turning a single test method into multiple tests.
The one supplied by Xunit is called InlineData and works fine, but if you do your own attributes working similiar, the test runner has some problems.
Let´s start with a simple test class:
C# public class Test
{
public Test(ITestOutputHelper output)
{
Output = output;
}
private ITestOutputHelper Output { get; }
}
If I add a simple test using Xunit´s InlineDataAttribute everything works like expected:
C#[Theory]
[InlineData(null)]
[InlineData("text1")]
[InlineData("text2")]
public void Test1(string text)
{
Output.WriteLine(text);
Assert.NotNull(text);
}
The Test Runner shows 3 tests (Test1(null), Test1("text1") and Test1("text2") and you can trigger each of them individually or run all 3 of them with the one having the null as data failing.
I could provide some code for a custom attribute, but for you to easily reproduce the bug let us use the InlineAutoDataAttribute from the NuGet package AutoFixture.Xunit2:
C#[Theory]
[InlineAutoData(null)]
[InlineAutoData("text1")]
[InlineAutoData("text2")]
public void Test2(string text, string autogeneratedText)
{
Output.WriteLine($"{text}: {autogeneratedText}");
Assert.NotNull(text);
}
The Test Runner now only shows one Test (Test2). This seems sadly a limitation of the framework as other test runners do the same.
Of course it would be nice to have the same 3 tests generated in the list as it works for the inbuild attribute. I would be happy if you could also do this or lead me to some way I can customize my attributes for this functionality. But that would be more a nice to have.
The main problem I have is that the Test Runner now runs Test2 as a batch of these 3 data attributes, but passes depending on the last run in this batch.
So depending on the order of the 3 attributes the test passes or fails as the test should be red if any of these fails.
Thank you for the code snippets. I have reproduced the problem. We will try to fix it as soon as possible. Once we make any progress, we will let you know.
We have fixed the problem and prepared a daily build with the fix - DevExpress.CodeRush-19.1.4.19156.vsix. After you install the build, the test runner will correctly run tests with custom xunit attributes (a Xunit.Sdk.DataAttribute descendant).
However for tests with AutoFixture.Xunit2.InlineAutoDataAttribute, we cannot correctly discover all test alternatives (3 tests in your code snippets), because "AutoFixture.Xunit2.InlineAutoDataAttribute" has the "[DataDiscoverer("AutoFixture.Xunit2.NoPreDiscoveryDataDiscoverer", "AutoFixture.Xunit2")]" attribute, which prevents to discover a single test as several. In this case, your test will be run as one test, but the result will be built with the help of each "InlineAutoDataAttribute" alternative.
Thus if at least one alternative (for InlineAutoDataAttribute) fails, the whole test will fail. However if all alternatives are passed, the whole test will be passed. In this case, the test runner will log information about all alternatives.