Bug Report T1188968
Visible to All Users

Test Runner (and Test Window) do not show all entries for data driven tests that use collections/arrays as members

created a year ago (modified a year ago)

Most test frameworks support data driven tests, one way or another. This means that a data provider (often a property or a method; sometimes even external files or database connections) returns a set of test data; and the framework calls a single parameterized test method with each of them in turn.

I'm currently working on a proof-of-concept that ports over portions of a PHP project to C# (mostly to compare its performance characteristics,) and part of that is porting over tests to see if we made any mistakes.

Those tests are purely data driven; they usually consist of data rows that contain:

  • An expected target (string)
  • Whether the result should be positive or negative (bool) and
  • An array of additional things that might be necessary to reach the target (string[])

Many of those tests use the same target and result with varying sets of additional inputs to prove that multiple paths to the target exist.

The test runner shows those as
TestMethod(Target, True, System.String[])
…which means a lot of tests don't show up properly.

According to the test runner, the project has 855 tests (at the time of writing.) Visual Studios built-in Test Explorer shows a total of 4067 tests.
Note that Test Explorer still shows TestMethod(Target, True, System.String[]), but it shows multiple copies. While suboptimal for running specific tests, it is a feasible way to fix this. Another way could be resolving collection types recursively (not just arrays but also lists/dictionaries etc.) but I can see this become a complex problem as the depth increases.

At the moment, this is using MSTest (because it felt most convenient,) but I assume this also happens with NUnit/xUnit and maybe others.

The project is available at https://github.com/BhaaLseN/alttp_vt_randomizer/tree/csharp (specifically the branch named csharp, with a solution file at app/app.sln and the test project at tests/cs/RandomizerTests/RandomizerTests.csproj)
For a particular example that shows this issue, look at the very top level PalaceOfDarkness.cs test that starts off with two TestLogic(Palace of Darkness - Big Chest, False, System.String[]) entries followed by six more TestLogic(Palace of Darkness - Big Chest, True, System.String[]) entries. Test Explorer shows those as a total of eight (8) tests while Test Runner only shows two (2.)
Pretty much every parameterized test in there shows the same behavior though, so you can pick any of them really.

Comments (3)
PW PW
Przemyslaw Wlodarczak (DevExpress) a year ago

    Emanuel,

    Thank you for the report. I reproduced this issue. VS Test Explorer shows 111 tests in total for the PalaceOfDarkness.cs file, whereas CodeRush Test Runner shows only 28 of them. We will investigate why this happens and follow up once we have information to share.

    EW EW
    Emanuel Wlaschitz a year ago

      My guess is that the string representation (which is the same for most of them due to string[].ToArray() being a static "System.String[]") is used as a key or distinction criteria somewhere, leading to the first (or last) one remaining.

      I tried a few things in the mean time, such as wrapping them in a collection that overrides ToString(), but so far all I managed to get was broken test detection in both Test Explorer and the Test Window (only showing a single test instead of a dynamic number of data tests.) The upside is that all tests run that way, but the significant downside is not being able to run and debug specific cases (which at least somewhat works with the regular array; except that I don't know which one of the tests it is running until I'm in the debugger to inspect the array parameter.)

      EW EW
      Emanuel Wlaschitz a year ago

        Just a side-note; I tried making MSTest produce more readable test names by using DynamicDataAttribute.DynamicDataDisplayName, but in the end this also only runs a subset of tests when the string representation before the display name method is the same. I didn't push those changes (because they don't help a whole lot at this point,) but it just reinforces that this is likely the internal storage for the Test Runner that keeps all data rows at hand.

        Answers approved by DevExpress Support

        created a year ago

        We have addressed the issue described in this ticket and will include a fix in our next maintenance update. Should you need to apply our fix prior to official release, you can request a hotfix here.

        Important Notes:

        • Hotfixes may be unavailable for Early Access/Beta builds or updates set for release within a week.
        • .NET only: in the NuGet Package Manager, use your personal NuGet feed and check the "Include prerelease" option to view the hotfix package in the "Updates" tab.
          Comments (3)
          PW PW
          Przemyslaw Wlodarczak (DevExpress) a year ago

            Emanuel,

            Would you mind if we make this ticket public to include it in our list of fixed issues for the next CodeRush release?

            EW EW
            Emanuel Wlaschitz a year ago

              Sure, go ahead. I don't recall marking it as private ticket in first place anyways.

              PW PW
              Przemyslaw Wlodarczak (DevExpress) a year ago

                Thank you!

                Disclaimer: The information provided on DevExpress.com and affiliated web properties (including the DevExpress Support Center) is provided "as is" without warranty of any kind. Developer Express Inc disclaims all warranties, either express or implied, including the warranties of merchantability and fitness for a particular purpose. Please refer to the DevExpress.com Website Terms of Use for more information in this regard.

                Confidential Information: Developer Express Inc does not wish to receive, will not act to procure, nor will it solicit, confidential or proprietary materials and information from you through the DevExpress Support Center or its web properties. Any and all materials or information divulged during chats, email communications, online discussions, Support Center tickets, or made available to Developer Express Inc in any manner will be deemed NOT to be confidential by Developer Express Inc. Please refer to the DevExpress.com Website Terms of Use for more information in this regard.