Category: Tutorial

Code coverage with VS Test and Coverage Validator

By , October 26, 2021 9:33 am

In this blog post I’m going to give you an example for running .Net Core unit tests with VS Test (formerly MS Test) and Coverage Validator. It’s the same process for regular .Net and C++.

First, let’s discuss the program we’re going to launch and the program we’re going to monitor.

vstest.console.exe

VS Tests are run by vstest.console.exe. So that’s the program we’re going to launch. On my machine the path is:

C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe

In this example I’m showing VS 2019 community edition, but it doesn’t matter which VS or version you use.

vstest.console.exe is not a .Net Core application (it’s a regular .Net application). You can check this with our free tool PE File Browser.

testhost.exe

vstest.console.exe executes the tests by running testhost.exe. We need to identify which testhost.exe to run (there will be several installed on your machine) and then configure Coverage Validator to monitor that testhost.exe when vstest.console.exe is run.

We haven’t worked out a way of identifying which testhost.exe VS Test is going to use, but once you’ve found it that will be the one forever.

On my machine testhost.exe is in c:\users\stephen\.nuget\packages\microsoft.testplatform.testhost\16.5.0\build\netcoreapp2.1\x64\testhost.exe

Note that despite the path testhost.exe itself is not a .Net Core application (it’s a regular .Net application).

Video or step-by-step

I’ve created a video showing you how to configure Coverage Validator, but if you prefer step-by-step instructions, these are listed below the video in this blog post.

Coverage Validator

To get started we need to launch vstest.console.exe to run the tests.


Click the Rocket icon on the toolbar. This will display the Launch Application or Service dialog.


Choose Launch Native and .Net applications.

Although we’re going to monitor code coverage in a .Net Core DLL, the application we’re going to launch to do this is not a .Net Core application, so we’ll use the regular .Net and native launcher.

You can also launch using Launch->Applications->Launch Application…, or F4, these will take you straight to the launch dialog/wizard, skipping the previous dialog.

The Start a Native/.Net application dialog is displayed.


Now we have to configure the start application dialog. We’ve got to:


  • choose the application to launch

  • edit the applications to monitor

  • set the application to monitor

  • set the arguments for the unit test

  • set the startup directory

  1. Set the application to launch.

    Next to the Application to Launch field click Browse… and select your vstest.console.exe

  2. Example: C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe


  3. Edit the applications to monitor. This is a multi-stage process. If the application to monitor has never been configured for the application being launched, you will need to configure the applications that can be set as possible applications that can be monitored. If the application to monitor has been configured, you can edit it to add or remove applications, or you can just use the selection of configured applications.

    Note the that after editing the Application to Launch field the Application to Monitor field will auto-populate itself, choosing a default value for the application to monitor. If this has never been configured before it will choose the same application as the application being launched (in this case vstest.console.exe). If this has been configured it will choose the default application for the configuration.

    For this blog post I’m going to assume that the application has never been configured, and show you how to configure it. Editing is just a subset of these actions and does not need it’s own blog post. If you already have applications to monitor configured you can skip this step.

    Next to Application to Monitor, click Edit…




    • The Applications to Monitor dialog is displayed.


    • We need to add an application to the list of application to monitor.

      On the Applications to Monitor dialog click Add…



      • The Application to Monitor dialog is displayed.


        Note that the EXE and DLL are set to represent the current application you are launching. For Native and .Net applications the DLL field will be empty (only set for .Net Core applications).

        If you wish to edit these value to configure for a different application than the one you are launching you can edit these via the Edit… button.


      • On the Application to Monitor dialog click Add…




        • The Application and DLL dialog is displayed.

        • On the Application and DLL dialog click Browse… and select your testhost.exe

          Example: c:\users\stephen\.nuget\packages\microsoft.testplatform.testhost\16.5.0\build\netcoreapp2.1\x64\testhost.exe


        • click OK to accept the EXE and DLL combination.



      • You should now have two entries, one for testhost.exe and one for the testhost.exe with a full path.


        You can repeat the Add… process for any additional applications you wish to configure.

        Optional: If you want to set this as the default application to monitor choose the appropriate entry in the Default application to monitor combo box.

      • click OK to accept the list of applications to monitor.

      The Applications to Monitor dialog should now show one entry for vstest.console.exe and testhost.exe.



    • click OK to accept these definitions of applications to monitor.


  4. Set the application to monitor.

    In the Application to Monitor combo select the entry for testhost.exe.

    We intend to monitor the first testhost.exe that is launched, so set the Launch count to 1.


  5. Arguments: Enter the full path to your DLL to test.

    Example: E:\om\c\testApps\unitTests\HelloWorldCore\HelloWorldCoreNUnitDotNet5\bin\Debug\net5.0\HelloWorldCoreNUnitDotNet5.dll



  6. Startup Directory. Enter a path for the startup directory. A default will have been set based on the Application to Launch, but for unit test work you’ll need a writeable directory, so you’ll need to edit this value to something appropriate.



  7. If you’re using the Launch Dialog click Launch.


    If you’re using the Launch Wizard click Next until you get to the last page of the wizard then click Start Application.





Results

For the test I’ve configured in this blog post, the results show code coverage for the unit tests, for the code tested by the unit tests, for some autogenerated code, and for some code in the test framework.

Some of the results are from code that isn’t your unit tests. You’ll need to filter these results by configuring Coverage Validator to ignore these code locations in future tests.



Filtering Autogenerated Code

To filter out of the autogenerated code, right click on the entry and choose Instrumentation Filter->Exclude Filename.

Filtered out entries are shown in grey. Next time you run the instrumentation they won’t be included in the coverage results.


Filtering Test Framework Code

To filter out the test framework code, right click on the entry and choose Instrumentation Filter->Exclude DLL.

Filtered out entries are shown in grey. Next time you run the instrumentation they won’t be included in the coverage results.


Any questions?

Hopefully this blog post has answered your questions about how to get code coverage with VS Test and Coverage Validator.

But you may have other questions. Please let us know at support@softwareverify.com and we’ll be pleased to help you.

A new take on tutorials

By , October 5, 2011 3:47 pm

We’ve been making changes to how we present our tutorials for our software tools.

First Steps

Our first tutorials were downloaded with evaluation versions of our tools. When the tool started for the first the tutorial would be shown.

Full versions of the tools did not come with tutorials as we felt this was wasted bandwidth and wasted time downloading material that rarely changed. The tutorial could be downloaded from the website when needed and installed alongside the software.

Second attempt

We recently decided it would be better to host the tutorials on our website. This would allow the tutorial content to be updated at will and thus always provide a better tutorial experience than downloading. This would also make the downloads smaller and thus faster to download.

In addition to this change we have started to introduce video versions of the tutorials. We hope you like it. At present these video tutorials are for C++ Memory Validator. We intend to cover all tutorials for all our software tools.

Creating the videos is interesting – some new tutorial videos have been created as a direct result of creating videos to accompany existing tutorials.

The present

Our latest change is to change how the tutorials are shown to the user of the software tool.

Tutorial tab

The change is that each software tool now has a dedicated tutorial tab. The tab lists the tutorials that are available, with a title, text indicating if the tutorial is text and/or video and the expertise level required for the tutorial. Double clicking on any tutorial topic takes you directly to the correct tutorial on our website.

Each time an evaluation version of a tool is used (and the first time a purchased software tool is used) the tutorial tab is displayed. The tutorial tab can be closed if desired. If the tutorial is later needed it is available from the help menu.

Tutorial close buttonTutorial help menu

We hope these changes will make the tutorials more appealing and more accessible than our previous attempts to inform and educate about effective ways to use our software tools.

Panorama Theme by Themocracy