We all have seen this very weird or not so weird phenomenon of the MSTest.exe command line commands, where MSTest.exe commands fails the build even with no actual test failures. I came across such a weird scenario where the build was failing because of few Inconclusive Tests.
The trouble with MSTest.exe is that the Test Tools Task thinks it failed whenever at least one test didn’t pass, and when the Test Tools Task fails, Team Build marks the overall build as failed as well.
I searched across the MSDN and go to know we cannot stop this behavior as MSTest will fail the build, if it has anything except Success.
For a workaround I updated my MSBuild Definition template to not mark the build as fail on error and to do so I changed the Flag value of property “Fail Build on Test Failuer” to “False”. In my case we are using custom build definition templates, so this flag might or might not be available in other scenarios.
I created a target named “CheckDBTestFailureCount” to determine and get the count of Actual Passed Tests, Inconclusive Tests and Failed Tests. This task uses the MSBuild ExtensionPack to read the trx file generated from MSTest.exe and checks for the ResultSummary Counters for any Errors or Inconclusive tests.
In the below example I am checking for only errors inside the trx file and throwing a custom error message if found any errors.
<!-- TST unit tests return a large number of Inconclusive test results because there's no simple mechanism to Ignore tests within the SQL. We therefore check for the count of failed tests when deciding whether the build has failed -->
<Target Name=”CheckDBTestFailureCount” DependsOnTargets=”InitialisePropertiesForDeploySQL”>
<Message Text=”Checking test results file ‘@(trxFile)’ for failed tests… [$(MSBuildThisFileFullPath)]” />
<Prefix>trx</Prefix> <Uri>http://microsoft.com/schemas/VisualStudio/TeamTest/2010 </Uri>
<Output TaskParameter=”Value” PropertyName=”FailedDBUnitTestCount”/>
<Message Text=”FailedDBUnitTestCount: ‘$(FailedDBUnitTestCount)’ ” />
<Error Condition=” $(FailedDBUnitTestCount) > 0 ” Text=”One or more Database Unit Tests failed. Check the Test results file ‘@(trxFile)’ for details ” />
The Target mentioned above is set as AfterTargets=”Targets” in the main target.
In Conclusion, I just tried a hack, which enables me to bypass MSBuild failures because of Inconclusive Tests and marks them green if there is no genuine test failure.