I've got a Universal Windows Test Project using MSTest, built straight from the Visual Studio 2019 UWP Test Project Template. When I run using the Visual Studio Test Runner, it works fine, and is green. However, when I use the Code Rush Test Runner, it fails before starting to execute any of my test code. I have also tried with other test frameworks (xUnit, nUnit), and they also fail under Code Rush. I have all options for Supported Frameworks selected, except for MSpec.
The specific error I'm getting is: "Microsoft.VisualStudio.TestPlatform.ObjectModel.TestPlatformException: No suitable test runtime provider found for this run." (Full output below)
Any suggestions on how to proceed? Code Rush is my desired tooling for tests - switching back and forth is super frustrating! I should note that this has been going on for quite some time, I think more than a year, so it's not a new break in 20.2.7. I am current on all tools involved.
Thanks much!
-Eric.
Test Runner Output:
Test started…
Error! Microsoft.VisualStudio.TestPlatform.ObjectModel.TestPlatformException: No suitable test runtime provider found for this run.
at Microsoft.VisualStudio.TestPlatform.Client.TestPlatform.ThrowExceptionIfTestHostManagerIsNull(ITestRuntimeProvider testHostManager, String settingXml)
at Microsoft.VisualStudio.TestPlatform.Client.TestPlatform.CreateDiscoveryRequest(IRequestData requestData, DiscoveryCriteria discoveryCriteria, TestPlatformOptions options)
at Microsoft.VisualStudio.TestPlatform.CommandLine.TestPlatformHelpers.TestRequestManager.DiscoverTests(DiscoveryRequestPayload discoveryPayload, ITestDiscoveryEventsRegistrar discoveryEventsRegistrar, ProtocolConfig protocolConfig)
at Microsoft.VisualStudio.TestPlatform.Client.DesignMode.DesignModeClient.<>c__DisplayClass24_0.<StartDiscovery>b__0()
Failed: The test runner process has been unexpectedly closed. It is possible when an unhandled exception (like StackOverflowException, MemoryOverflowException, etc.) was thrown.
Summary: Passed: 0, Failed: 0, Ignored: 0
Duration: 0.57 seconds
Hi Eric,
Thank you for your report. I reproduced this behavior locally. We will research the possibility to fix this issue in the future.
Thanks. I was trying a different approach, and received an error I thought might help. Basically, I was using xunit.runners.devices as documented here: https://xunit.net/docs/getting-started/uwp/devices-runner
After following their setup, I received the following error from DevExpress.CodeRush.TestRunner.WCF.TestRunnerClientBase.DiscoverTests. Hope it helps!
System.IO.FileNotFoundException: Could not load file or assembly 'System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e' or one of its dependencies. The system cannot find the file specified.
File name: 'System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e'
at System.ModuleHandle.ResolveMethod(RuntimeModule module, Int32 methodToken, IntPtr* typeInstArgs, Int32 typeInstCount, IntPtr* methodInstArgs, Int32 methodInstCount)
at System.ModuleHandle.ResolveMethodHandleInternalCore(RuntimeModule module, Int32 methodToken, IntPtr[] typeInstantiationContext, Int32 typeInstCount, IntPtr[] methodInstantiationContext, Int32 methodInstCount)
at System.ModuleHandle.ResolveMethodHandleInternal(RuntimeModule module, Int32 methodToken, RuntimeTypeHandle[] typeInstantiationContext, RuntimeTypeHandle[] methodInstantiationContext)
at System.Reflection.CustomAttributeData…ctor(RuntimeModule scope, CustomAttributeRecord caRecord)
at System.Reflection.CustomAttributeData.GetCustomAttributes(RuntimeModule module, Int32 tkTarget)
at System.Reflection.CustomAttributeData.GetCustomAttributesInternal(RuntimeAssembly target)
at Xunit.Sdk.ReflectionAssemblyInfo.GetCustomAttributes(String assemblyQualifiedAttributeTypeName)
at Xunit.Sdk.XunitTestFrameworkDiscoverer…ctor(IAssemblyInfo assemblyInfo, ISourceInformationProvider sourceProvider, IMessageSink diagnosticMessageSink, IXunitTestCollectionFactory collectionFactory)
at Xunit.Sdk.XunitTestFramework.CreateDiscoverer(IAssemblyInfo assemblyInfo)
at Xunit.Sdk.TestFramework.GetDiscoverer(IAssemblyInfo assemblyInfo)
at Xunit.Xunit2Discoverer…ctor(AppDomainSupport appDomainSupport, ISourceInformationProvider sourceInformationProvider, IAssemblyInfo assemblyInfo, String assemblyFileName, String xunitExecutionAssemblyPath, String configFileName, Boolean shadowCopy, String shadowCopyFolder, IMessageSink diagnosticMessageSink, Boolean verifyAssembliesOnDisk)
at Xunit.XunitFrontController.CreateInnerController()
at Xunit.XunitFrontController.get_InnerController()
at Xunit.XunitFrontController.Find(Boolean includeSourceInformation, IMessageSink messageSink, ITestFrameworkDiscoveryOptions discoveryOptions)
at DevExpress.CodeRush.TestRunner.xUnit21.Executor.LoadTests()
WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].
Exception was thrown:
Type: System.IO.FileNotFoundException
Message: Could not load file or assembly 'System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e' or one of its dependencies. The system cannot find the file specified.
Stack trace:
at DevExpress.CodeRush.TestRunner.xUnit21.Executor.LoadTests()
at DevExpress.CodeRush.TestRunner.xUnit21.xUnitWCFServiceCallback.LoadTests(RequestBase request, Executor& executor)
at DevExpress.CodeRush.TestRunner.xUnit21.xUnitWCFServiceCallback.DiscoverTestsInternal(TestDiscoveryRequest request)
at DevExpress.CodeRush.TestRunner.WCF.TestRunnerClientBase.DiscoverTests(TestDiscoveryRequest request)
Thank you for the additional information. We will research it.
Hi Eric,
We have fixed this issue for the MSTest framework only. Please try installing this build and let me know your results.
The fix for other frameworks is more complex and needs more time. We will notify you when the fix is ready.
I am looking forward to your reply.
I saw no change in behavior using v20.2.7.21048 (linked above) – still broken for me.
However, the specific error is different:
Error! System.ArgumentNullException: Value cannot be null. Parameter name: path at System.IO.Directory.EnumerateFiles(String path, String searchPattern, SearchOption searchOption) at Microsoft.VisualStudio.TestPlatform.Utilities.Helpers.FileHelper.EnumerateFiles(String directory, SearchOption searchOption, String[] endsWithSearchPatterns) at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Hosting.DefaultTestHostManager.<GetTestPlatformExtensions>b__38_0(String s) at System.Linq.Enumerable.<SelectManyIterator>d__17`2.MoveNext() at System.Linq.Enumerable.<ConcatIterator>d__59`1.MoveNext() at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Hosting.DefaultTestHostManager.FilterExtensionsBasedOnVersion(IEnumerable`1 extensions) at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Hosting.DefaultTestHostManager.GetTestPlatformExtensions(IEnumerable`1 sources, IEnumerable`1 extensions) at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyExecutionManager.InitializeExtensions(IEnumerable`1 sources) at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyExecutionManager.StartTestRun(TestRunCriteria testRunCriteria, ITestRunEventsHandler eventHandler) Failed: The test runner process has been unexpectedly closed. It is possible when an unhandled exception (like StackOverflowException, MemoryOverflowException, etc.) was thrown.
I've requested the hotfix (below) and will test with that once it is available to me.
As a side question - will this ticket be updated when Xunit support is also available, or is there a separate ticket I can watch for the Xunit specific fixes?
Thanks!
Hi Eric,
I created a separate ticket on your behalf (T974893: Test Runner - Fails to run tests from the Universal Windows Test Project for the xUnit framework). It is in our processing queue, and we will address it shortly.
I attached a video showing how the fix works on my side.
Could you please send us a small sample where this issue occurs?
Note: Test debugging is not working yet, and this is a known issue. We are working on a solution.
I'm looking forward to your reply.
Thanks for the Xunit ticket, that's super helpful.
I don't have a small sample easily available at this time, but I'll see what I can do… I replicated the success from your basic sample, and it looked good. I also updated to the most recent Uwp sdk (which worked), and ran it against a x64 build (which also worked). However, my main solutions Uwp tests still fail to run under CodeRush.
I'll try to get a simple sample to send you, but it might take me a little time. In the mean time, are there any diagnostics I could collect to help you make sense out of that failure stack trace I sent in my last comment?
Thanks,
-Eric.
I got it. Using your example, if I change the UnitTest.cs file to contain 10 or more [TestMethod]'s, it will fail. 1-9 test methods works fine, but the addition of the 10th one gives me the error shown above. It does not matter if you run one, a sub-set of all, or all tests, the execution will fail internally. This only happens when running under CodeRush - if I run the same tests using the visual studio test runner, they execute as expected.
Here's my literal test code, using your example with the minimum and target uwp version set to 19041, and behaves the same under x86 and x64.
I should also note that I get the uwp test window to pop up and disappear repeatedly from time to time when NOT running tests. My last build caused it to pop up and disappear repeatedly until I ran tests again.
using System; using System.Threading.Tasks; using Microsoft.VisualStudio.TestTools.UnitTesting; namespace UnitTestProject2 { [TestClass] public class UnitTest1 { [TestMethod] public void TestMethod1() { } [TestMethod] public void TestMethod2() { } [TestMethod] public void TestMethod3() { } [TestMethod] public void TestMethod4() { } [TestMethod] public void TestMethod5() { } [TestMethod] public void TestMethod6() { } [TestMethod] public void TestMethod7() { } [TestMethod] public void TestMethod8() { } [TestMethod] public void TestMethod9() { } [TestMethod] public void TestMethod10() { } } }
Hi Eric,
Thank you for the additional information. I reproduced this behavior locally. We need some time to research it.
Hi Eric,
We have fixed the problem with running more than ten tests. Please try to install this build and let me know your results.
The debug is not supported yet. We are working on this issue.
As for the window that appears outside of testing, this happens after the project is built. The CodeRush thus searches for tests from assemblies. You can turn off this feature on the options page:
This worked perfectly! Thanks much,
-Eric.
Hi Eric,
You're welcome! Thank you for informing us. We will continue working to fix issues with debugging.
Hi Eric,
We fixed the issue with debugging. Please try to install the build from the link below and let me know your results.
DevExpress.CodeRush-20.2.8.21077.vsix