WSL2 remote testing and MSTest seems broken on Visual Studio?

[markm77]

Hi there,

Something seems to have gone wrong with Visual Studio/MS Test tooling which means that remote tests on WSL2 that used to work now don't. I don't believe it's my code as I have tried tests that previously worked with earlier versions of Visual Studio a couple of months ago or less. I hoped rolling back Visual Studio (which I believe is linked to a .NET SDK version) would fix the issue but it doesn't (and it doesn't seem easy to roll back further than the previous version). So I'm reporting the issue here in the hope of getting guidance in isolating it further.

I've created a minimal repro (see below) to demo the issue so perhaps that can help in working out where to go from here.

I appreciate any help as this has completely broken my test system.

BR,
Mark

Steps To Reproduce

Create a simple test project in latest version of Visual Studio (17.12.3) as follows.

Project file:

<Project Sdk="MSTest.Sdk/3.7.0">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>

</Project>

testEnvironments.json file (configures remote testing):

{
  "version": "1",
  "environments": [
    {
      "name": "WSL-Ubuntu",
      "type": "wsl",
      "wslDistribution": "Ubuntu"
    }
  ]
}

Test method file:

namespace RemoteTestRepro;

public class TestData
{
    public required string MyValue { get; init; }

    public override string ToString() => MyValue;
}

[TestClass]
public sealed class Test1
{
    public static IEnumerable<object[]> GetNewTestCases()
    {
        return
        [
            [new TestData { MyValue = "1" }], [new TestData { MyValue = "2" }], [new TestData { MyValue = "3" }]
        ];
    }

    [DataTestMethod]
    [DynamicData(nameof(GetNewTestCases), DynamicDataSourceType.Method)]
    [DoNotParallelize]
    public Task TestMethod1(TestData data) => Task.CompletedTask;

}

Expected behaviour

There are three trivial tests and they should pass in both Windows and WSL2 (remote testing) environments.

Actual behaviour

Tests pass fine in Windows envrionment.

Tests will not run in WSL2 environment and errors like the following turn up in the output from Tests:

JsonRpc Error: 10 : 
Exception thrown from request "(not specified)" for method testing/testUpdates/tests: System.AggregateException: One or more errors occurred. (Path C:\Repos\RemoteTestRepro\RemoteTestRepro\bin\Debug\net8.0\RemoteTestRepro.dll must begin with the input root /mnt/c/Repos/RemoteTestRepro/.)
 ---> System.ArgumentException: Path C:\Repos\RemoteTestRepro\RemoteTestRepro\bin\Debug\net8.0\RemoteTestRepro.dll must begin with the input root /mnt/c/Repos/RemoteTestRepro/.
   at Microsoft.VisualStudio.TestWindow.Internal.RemoteAgent.TestAgentBridge.PathTranslator.StripRoot(String root, String path, Boolean caseInsensitive) in D:\a\_work\1\s\src\TestWindow\Internal\RemoteAgent\TestAgentBridge\PathTranslator.cs:line 44
   at Microsoft.VisualStudio.TestWindow.Internal.RemoteAgent.TestAgentBridge.PathTranslator.TranslatePath(String inPath) in D:\a\_work\1\s\src\TestWindow\Internal\RemoteAgent\TestAgentBridge\PathTranslator.cs:line 82
   at Microsoft.VisualStudio.TestWindow.Internal.RemoteAgent.RemoteTestAgentBridge.TranslateRemoteToLocal(String source) in D:\a\_work\1\s\src\TestWindow\Internal\RemoteAgent\TestAgentBridge\RemoteTestAgentBridge.cs:line 48
   at Microsoft.VisualStudio.TestWindow.Internal.RemoteAgent.TestingPlatform.Parsers.BaseMessageParser.CreateTestResultRecord(Node node, TestCaseRecord testCase, String source, ITestRunDataSink dataSink) in D:\a\_work\1\s\src\TestWindow\Internal\RemoteAgent\TestingPlatform\Parsers\BaseMessageParser.cs:line 181
   at Microsoft.VisualStudio.TestWindow.Internal.RemoteAgent.TestingPlatform.TestingPlatformCallbackHandler.GenerateTestResults(IEnumerable`1 resultNodes) in D:\a\_work\1\s\src\TestWindow\Internal\RemoteAgent\TestingPlatform\TestingPlatformCallbackHandler.cs:line 227
   at Microsoft.VisualStudio.TestWindow.Internal.RemoteAgent.TestingPlatform.TestingPlatformCallbackHandler.GetTestRunEventArguments(IEnumerable`1 nodes) in D:\a\_work\1\s\src\TestWindow\Internal\RemoteAgent\TestingPlatform\TestingPlatformCallbackHandler.cs:line 210
   at Microsoft.VisualStudio.TestWindow.Internal.RemoteAgent.TestingPlatform.TestingPlatformCallbackHandler.TestsUpdateAsync(TestingPlatformOperationContext context, IEnumerable`1 nodes) in D:\a\_work\1\s\src\TestWindow\Internal\RemoteAgent\TestingPlatform\TestingPlatformCallbackHandler.cs:line 190
   at Microsoft.VisualStudio.TestWindow.Internal.RemoteAgent.TestingPlatform.TestingPlatformCallbackHandler.TestsUpdateAsync(TestsUpdates testsUpdate) in D:\a\_work\1\s\src\TestWindow\Internal\RemoteAgent\TestingPlatform\TestingPlatformCallbackHandler.cs:line 170
   --- End of inner exception stack trace ---

If I try to debug the WSL2 tests in Visual Studio, the tests do not complete and the "Cancel" button becomes unresponsive.

[Evangelink]

Hi @markm77,

Were you already using MSTest runner (or MSTest SDK) the last time it was working?

@drognanar could you please investigate?

[markm77]

Hi @Evangelink ,

Thanks for the reply. Yes I was using a project with MSTest SDK when it previously worked (v3.6.3 although that also fails now). I also believe I was using Visual Studio v17.12.2 (which fails now too) but I can't be absolutely certain on that. I spent quite a bit of time trying to revert back to a successful state but this is hard as I don't know exactly how the remote testing works (does it rely on dependencies on the Linux side or Windows system components which may have changed?) and ultimately I was not successful.

BR,
Mark

[Evangelink]

Thanks @markm77,

I assume this is a Test Explorer issue so I'll try to have someone from that team looking at the issue after the holidays.

[markm77]

Hi @Evangelink ,

Thanks for this!

In the meantime, I've started running MSTest tests manually in WSL2 with dotnet run --project test/MyTestProject -- --settings .runsettings --filter "TestKeyword". However a major limitation of this is that I can't seem to set a filter to run an individual dynamic test. Also it's not trivial to see exactly what dynamic tests failed even when using dotnet test which generates a .log file instead of console output.

So I do look forward to getting to the bottom of this.

BR,
Mark

[markm77]

Hi @Evangelink ,

Based on your comment, I thought I'd try to disable Microsoft Testing Platform and force Visual Studio to use VSTest to run the tests by adding the following to the .csproj file:

  <ItemGroup>
    <ProjectCapability Remove="TestingPlatformServer"/>
  </ItemGroup>

I assumed this would work as I can run the MSTest tests in Rider which I assume uses VSTest (note Rider does not have a "same-source" WSL2 remote testing feature like Visual Studio).

But this setting does not appear to have any effect (perhaps the MSTest SDK overrides it?).

BR,
Mark