Bug Report T745917
Visible to All Users

Test Runner incorrectly runs tests with custom xunit attributes

created 6 years ago

Xunit has the option to mark tests with data attributes. The CodeRush test runner seems to work fine with this unless you are having multiple of these turning a single test method into multiple tests.

The one supplied by Xunit is called InlineData and works fine, but if you do your own attributes working similiar, the test runner has some problems.

Let´s start with a simple test class:

C#
public class Test { public Test(ITestOutputHelper output) { Output = output; } private ITestOutputHelper Output { get; } }

If I add a simple test using Xunit´s InlineDataAttribute everything works like expected:

C#
[Theory] [InlineData(null)] [InlineData("text1")] [InlineData("text2")] public void Test1(string text) { Output.WriteLine(text); Assert.NotNull(text); }

The Test Runner shows 3 tests (Test1(null), Test1("text1") and Test1("text2") and you can trigger each of them individually or run all 3 of them with the one having the null as data failing.

I could provide some code for a custom attribute, but for you to easily reproduce the bug let us use the InlineAutoDataAttribute from the NuGet package AutoFixture.Xunit2:

C#
[Theory] [InlineAutoData(null)] [InlineAutoData("text1")] [InlineAutoData("text2")] public void Test2(string text, string autogeneratedText) { Output.WriteLine($"{text}: {autogeneratedText}"); Assert.NotNull(text); }

The Test Runner now only shows one Test (Test2). This seems sadly a limitation of the framework as other test runners do the same.

Of course it would be nice to have the same 3 tests generated in the list as it works for the inbuild attribute. I would be happy if you could also do this or lead me to some way I can customize my attributes for this functionality. But that would be more a nice to have.

The main problem I have is that the Test Runner now runs Test2 as a batch of these 3 data attributes, but passes depending on the last run in this batch.

So depending on the order of the 3 attributes the test passes or fails as the test should be red if any of these fails.

Comments (2)
DevExpress Support Team 6 years ago

    Thank you for the code snippets. I have reproduced the problem. We will try to fix it as soon as possible. Once we make any progress, we will let you know.

    DevExpress Support Team 6 years ago

      We have fixed the problem and prepared a daily build with the fix - DevExpress.CodeRush-19.1.4.19156.vsix. After you install the build, the test runner will correctly run tests with custom xunit attributes (a Xunit.Sdk.DataAttribute descendant).
      However for tests with AutoFixture.Xunit2.InlineAutoDataAttribute, we cannot correctly discover all test alternatives (3 tests in your code snippets), because  "AutoFixture.Xunit2.InlineAutoDataAttribute" has the "[DataDiscoverer("AutoFixture.Xunit2.NoPreDiscoveryDataDiscoverer", "AutoFixture.Xunit2")]" attribute, which prevents to discover a single test as several. In this case, your test will be run as one test, but the result will be built with the help of each "InlineAutoDataAttribute" alternative.
      Thus if at least one alternative (for InlineAutoDataAttribute) fails, the whole test will fail. However if all alternatives are passed, the whole test will be passed. In this case, the test runner will log information about all alternatives.

      Answers approved by DevExpress Support

      created 6 years ago (modified 6 years ago)

      We have fixed the issue described in this ticket and will include the fix in our next maintenance update. To apply this solution before the official update, request a hotfix by clicking the corresponding link for product versions you require.

      Note: Hotfixes may be unavailable for beta versions and updates that are about to be released.

        Comments (2)
        GM GM
        Gerald Mahlmeister 6 years ago

          Thx, works like a charm.

          As said the display as multiple tests would have been a nice to have. As no other test runner does this, I had not high hopes you could solve the impossible ;)

          Greetings
            Gerald

          DevExpress Support Team 6 years ago

            You are welcome. At the moment, the current test discovering version is the only possible one.
            However, we will be monitoring the framework changes in order to find a way to improve the custom test discovering in the CodeRush test runner.

            Disclaimer: The information provided on DevExpress.com and affiliated web properties (including the DevExpress Support Center) is provided "as is" without warranty of any kind. Developer Express Inc disclaims all warranties, either express or implied, including the warranties of merchantability and fitness for a particular purpose. Please refer to the DevExpress.com Website Terms of Use for more information in this regard.

            Confidential Information: Developer Express Inc does not wish to receive, will not act to procure, nor will it solicit, confidential or proprietary materials and information from you through the DevExpress Support Center or its web properties. Any and all materials or information divulged during chats, email communications, online discussions, Support Center tickets, or made available to Developer Express Inc in any manner will be deemed NOT to be confidential by Developer Express Inc. Please refer to the DevExpress.com Website Terms of Use for more information in this regard.