Bug Report T588089
Visible to All Users

Unit Testing - xunit tests with classes as a data source are not shown in the TestRunner Window

created 7 years ago

Hi,

With xunit, when you have Th eory tests that have a MemberData or ClassData attribute that specifies data from a class, only a single test is shown.

Example:

C#
using System; using System.Collections.Generic; using Xunit; namespace TestRunner.b { public class MemberData { [Theory] [MemberData(nameof(AllValues))] public void Test1(Data data) { Assert.True(data.Value % 2 == 0); } public static IEnumerable<object[]> AllValues() { return new[] { new object[]{ new Data(1) }, new object[]{ new Data(2) }, new object[]{ new Data(3) }, new object[]{ new Data(4) } }; } } }

Observed behavior:
* Only one Test is show in the Test Runner ("Test1") and it's green
* In the test runner output window everything looks fine: "Summary: Passed: 2, Failed: 2, Ignored: 0"

Expected behavior:
* Number of tests in the test runner should be correct (4)
* If any of those fail it should be red
* The output from all tests should be available in the detail pane
* Showing all tests in the tree is not something I'm expecting (4 entries instead of just one) but maybe it should be consistent with the other Theory tests.

Additional issues that may be related:
* Double clicking "Test1" doesn't go to the source code after you run the tests
* Sometimes "Test1" ghost tests appear even in scenarios that work (check the MemberDataWithSimpleTypes class in the attached project)

Attached is a project that exemplifies the problem with both MemberData and ClassData, and an example that works with simple types (int).

Show previous comments (1)
DevExpress Support Team 7 years ago

    Hello,

    Additional research showed that working on this issue is quite a resource consuming task, so we need some additional time to fix this issue.
    We will notify you when we make any progress.

      It's not urgent. Happy Holidays!

      DevExpress Support Team 7 years ago

        Thank you for the update and understanding. Happy Holidays!

        Answers approved by DevExpress Support

        created 7 years ago (modified 7 years ago)

        We have fixed the issue described in this ticket and will include the fix in our next maintenance update. To apply this solution before the official update, request a hotfix by clicking the corresponding link for product versions you require.

        Note: Hotfixes may be unavailable for beta versions and updates that are about to be released.

        Additional information:

        Hi Jorge,

        We have fixed this issue but the final behavior has several differences from your expectations.

        > * Number of tests in the test runner should be correct (4)
        > * Showing all tests in the tree is not something I'm expecting (4 entries instead of just one) but maybe it should be consistent with the other Theory tests.

        It could not be realized because the xUnit engine recognizes the Theory test in your project as a single test with some results. You can check it by opening the Test Explorer window and running tests via the standard Visual Studio test runner:

        In everything else CodeRush Test Runner works on your project the way you expect:

        Also, you can add the implementation of the IXunitSerializable interface in the Data class:

        C#
        public class Data : IXunitSerializable { public Data() { } public Data(int value) { this.Value = value; } public int Value { get; set; } public void Deserialize(IXunitSerializationInfo info) { Value = info.GetValue<int>(nameof(Value)); } public void Serialize(IXunitSerializationInfo info) { info.AddValue(nameof(Value), Value); } public override string ToString() { return "TEST " + this.Value.ToString(); } }

        In this case, the xUnit engine recognizes the Theory test as some tests with different parameters:

          Disclaimer: The information provided on DevExpress.com and affiliated web properties (including the DevExpress Support Center) is provided "as is" without warranty of any kind. Developer Express Inc disclaims all warranties, either express or implied, including the warranties of merchantability and fitness for a particular purpose. Please refer to the DevExpress.com Website Terms of Use for more information in this regard.

          Confidential Information: Developer Express Inc does not wish to receive, will not act to procure, nor will it solicit, confidential or proprietary materials and information from you through the DevExpress Support Center or its web properties. Any and all materials or information divulged during chats, email communications, online discussions, Support Center tickets, or made available to Developer Express Inc in any manner will be deemed NOT to be confidential by Developer Express Inc. Please refer to the DevExpress.com Website Terms of Use for more information in this regard.