Getting coverage reports with .NET Core

Code coverage calculates the percentage of code that is covered by automated (unit) tests. And unit tests are important to ensure ongoing code quality and predictability in our software. Code coverage reports help investigate how well our software development is doing by showing us that percentage. But we know all this right? Tests are important, so test coverage is also important 🙂

covermec

You can get to this percentage in a few ways, but the easiest is of course with the Visual Studio IDE itself. Although this is a good first step, it is usually not adequate enough for larger/ multiple teams. In that case, generating coverage reports during central builds is your best option.

This post discusses the steps needed to enable code coverage and as a bonus includes how to integrate them with SonarQube.

A fair warning:
The code coverage data collectors (ugh) were only added to the .NET Core SDK recently, so you will need at least .NET Core Sdk 2.1.400. And it is currently only available for Windows based build servers, but my guess is this requirement is likely to drop “soon”.

Oh and integrating them with SonarQube is also not so straightforward. It requires a couple of extra black magic voodoo build steps to manipulate the coverage data and convert it to a format that SonarQube/ SonarCloud understands. But these things are never simple, right?

Local reports with Visual Studio 2017

For local code coverage reports, you need a recent Visual Studio version (15.4 or newer). Setting this up is not that difficult and only requires you to enable the Full Debug Information.

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netstandard2.0</TargetFramework>
    <!-- Code Coverage Analysis requires Full Debug-->
    <DebugType>Full</DebugType>
    <!-- SonarQube requires a valid Project GUID but Core projects dont actually use this -->
    <ProjectGuid>{950585D2-E827-4D75-ADD6-C1C629CBC37C}</ProjectGuid>
  </PropertyGroup>
</Project>

If you then fire up the code coverage analysis from the “Test/Analyze Code Coverage” menu, it shows you the metrics in the results window:

LocalCoverage

Build reports with VSTS or TFS

Local coverage reports are useful, but you get the real benefits from this feature if you enable it as part of the builds.

So what do we currently need to get this to work?

  • .NET Core Sdk >= 2.1.400 on our build machine
  • Full Debug option (same as with the local coverage with Visual Studio)
  • A Windows based build server (for now, see the warning at the top of the post)

There are many alternatives to the built in code coverage reporter (AltCover, Coverlet to name two), but my goal here is to work with the default tooling.

Enabling the reports is simple, and can be configured as part of the build task by suppying the argument “–collect:”Code Coverage”. If you want to show the test/ coverage results on the VSTS/ TFS dashboards, you should also enable the publish option:

test-coverage1b

After a succesful run, the results should show you the test coverage:

test-coverage2

Integration with SonarQube

Big fan of SonarQube, so for me integrating code coverage results there is a big win. It provides me with a single dashboard that tells me how well our projects are doing.

With the default coverage option, we face a couple of challenges:

  • The output is saved with a timestamp and in the Build Agent Temp directory
  • The output format is not compatible with SonarQube as it expects CoverXML
  • The SonarQube scanner expects the test results and the coverage report to reside in a “Test Results” directory within the Build Agent’s source folder

To fix these issues, we will provide additional build tasks for copying the files to the correct location and for converting to CoverXML.

test-coverage3

Visual Studio Test Platform Installer

This out of the box task will download the executable we need to analyze and convert the coverage results.

Prepare analysis on SonarCloud
This is either the SonarCloud task or the SonarQube task. You get them from the Marketplace. Apart from the usual Sonar properties (project, key, version) we need to provide 1 extra property under “Advanced”:

sonar.cs.vscoveragexml.reportsPaths=$(Agent.BuildDirectory)\TestResults\TestCoverage.xml

Copy test results and Analyze coverage file

This first PowerShell task will locate the .trx and .coverage file and copy them to the TestResults directory under the Build directory.

Get-ChildItem -Path $(Agent.TempDirectory) -Filter *.trx -Recurse -ErrorAction SilentlyContinue -Force`
| %{Join-Path -Path $_.Directory -ChildPath $_.Name }`
| Copy-Item -Destination $(Agent.BuildDirectory)\TestResults\TestResults.trx -Force

Get-ChildItem -Path $(Agent.TempDirectory) -Filter *.coverage -Recurse -ErrorAction SilentlyContinue -Force `
|%{Join-Path -Path $_.Directory -ChildPath $_.Name }`
| Copy-Item -Destination $(Agent.BuildDirectory)\TestResults\TestCoverage.coverage -Force

This second PowerShell task will convert the coverage results to CoverageXML. It depends on CodeCoverage.exe, but that part is downloaded with the "Visual Studio Test Platform Installer" task.

$tool= Get-ChildItem -Path '$(Agent.TempDirectory)\VsTest\*\tools\net451\Team Tools\Dynamic Code Coverage Tools\amd64\'`
 -Filter CodeCoverage.exe -Recurse -ErrorAction SilentlyContinue -Force

$parameter1 =  'analyze'
$parameter2 = '/output:$(Agent.BuildDirectory)\TestResults\TestCoverage.xml'
$parameter3 = '$(Agent.BuildDirectory)\TestResults\TestCoverage.coverage'
$parameters= "$parameter1 $parameter2 $parameter3"

Write-Host "Executing $tool $parameters"
$parms = $parameters.Split(" ")
& "$tool" $parms 

Run Code Analysis
Again, the original SonarQube or SonarCloud task. This will actually take the coverage xml file and upload it to Sonar for the results!

test-coverage4

You can grab the scripts mentioned here (and the test project) from the GitHub repo: https://github.com/yuriburger/net-cover-demo

/Y.

One comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s