Posts tagged: Coverage

UX Improvements for Coverage Validator

By , August 3, 2012 11:36 am

We recently released new versions of the Coverage Validator tools for all languages.

The main reason for this release was to make the tools more usable and make using them more satisfying. This work was inspired by some user experience research we commissioned with Think UI.

We’re so happy with these improvements we thought we’d share them so that you can learn from our improvements. We’re not finished with the Coverage Validator tools. This is just the start of changes to come.

I’m specifically going to talk about C++ Coverage Validator, but these improvements cut across all our Coverage Validator tools. Some of the improvements cut across all our development tools.

Summary Dashboard

The first thing a user of Coverage Validator sees is the summary dashboard.

The previous version of this dashboard was a grid with sparse use of graphics and lots of text. You had to read the text to understand what was happening with the code coverage for the test application. Additional comments and filter status information was displayed in right hand columns.

Coverage Validator old dashboard

The new version of this dashboard is split into two areas. The top area contains a dial for each metric reported. Each dial displays three items of information: Number of Items, Number of Items visited. How many items are 100% visited. This is done by means of an angular display for one value and a radial display for another value. A couple of the dials are pie charts.

The bottom area of the dashboard displays information that is relevant to the recorded session. Any value that can be viewed or edited is easily reachable via a hyperlink.

Coverage Validator new dashboard

The result of these changes are that the top area makes it easy to glance at a coverage report and instantly know which session has better coverage than another session. You don’t need to read the text to work it out. The bottom area draws attention to instrumentation failures (missing debug information, etc) and which filters are enabled etc. By exposing this information in this way more functionality of Coverage Validator is exposed to the user of the software.

Coverage Dials

We developed a custom control to display each coverage dial.

A coverage dial displays both the amount of data that has been visited, the amount of data that is unvisited and the amount that has been completely visited. For metrics that do not have a partial/complete status the dial just displays as a two part pie chart. An additional version displays data as a three part pie chart. This last version is used for displaying Unit Test results (success, failure, error).

Coverage dial directories Coverage dial unit tests

The difference between unvisited coverage and visited coverage is displayed using an angular value. Items that have been completely visited (100% coverage) are displayed using a radial value emanating from the centre of the dial. Additional information is displayed by a graded colour change between the 100% coverage area and the circumference of the circle to indicate the level of coverage in partially covered areas.

The coverage dial provides tooltips and hyperlinks for each section of the coverage dial.

Dashboard Status

The dashboard status area shows informational messages about the status of code instrumentation, a filter summary, unit test status and session merging status. Most items are either viewable or editable by clicking a hyperlink.

Dashboard status

To implement the hyperlink we created a custom control that supported hyperlinks with support for email hyper links, web hyper links and C++ callbacks. This provides maximal functionality. The hyperlinks are now used in many places in our tools – About box, evaluation feedback box, error report boxes, data export confirmation boxes, etc.

Coverage Scrollbar

We’ve also made high level overview data available on all the main displays (Coverage, Functions, Branches, Unit Tests, Files and Lines) so that you can get an overview of the coverage of each file/function/branch/etc without the need to scroll the view.

We thought of drawing the coverage data onto the scrollbar. Unfortunately this means that you would need an ownerdrawn scrollbar, but Windows does not provide such a thing. An option was to use a custom scroll bar implementation. But doing that would mean having to cater to every different type of Windows scrollbar implementation. We didn’t think that was a good idea. As such we’ve chosen to draw the coverage overview next to the scroll bar.

Coverage scroll bar

Editor Scrollbar

Similarly to the overview for each type of data we also provide a high level overview for the source code editor.

Editor code coverage

Directory Filter

Coverage Validator provides the ability to filter data on a variety of attributes. One of these is the directory in which a file is found. For example if the file was e:\om\c\svlWebAPI\webapi\ProductVersion\action.cpp the filter directory would be e:\om\c\svlWebAPI\webapi\ProductVersion\.

This is useful functionality but Coverage Validator allows you to filter on any directory. In complex software applications it’s quite possible that you would want to filter on a parent directory or a root directory. That would give the following directories for the example above.


The solution to this problem is to create the context menu dynamically rather than use a preformed context menu stored in application resources. Additionally it is more likely that the current directory will be filtered rather than the parent, so it makes sense to reverse the order of the directories, going from leaf to root.

Coverage filter directory context menu

Instrumentation Preferences

The instrumentation preferences dialog is displayed to the user the first time that Coverage Validator starts. The purpose of this dialog is to configure the initial way coverage data is collected. This provides a range of performance levels from fast to slow and incomplete visit counts to complete visit counts. Both options affect the speed of execution of the software. Given that time to complete is an important cost for this is an option that should be chosen carefully.

Previous versions of the software displayed a wordy dialog containing two questions. Each question had two choices.

Coverage instrumentation preferences (old)

The new version of the instrumentation preference dialog has replaced the questions with a sliding scale. Two questions with two choices is effectively four combinations. The instrumentation level sliding scale has four values. As the slider value changes the text below the slider changes to provide a brief explanation of the instrumentation level chosen.

An additional benefit is that the previous version only implied the recommended values (we preset them). The new version also implies the recommended value (also preset) but also explicitly indicates the recommended instrumentation level.

This new design has less words, less visual clutter, is easier to use and presents less cognitive load on the user.

Coverage instrumentation preferences (new)

Export Confirmation

Coverage Validator provides options to export data to HTML and XML. A common desire after exporting is to view the exported data. Previous versions of Coverage Validator overlooked this desire, no doubt causing frustrating for some users. We’ve rectified this with a confirmation dialog displayed after exporting data. The options are to view the exported file or to view the contents of the folder holding the exported data. An option to never display the dialog again is also provided.

Coverage export confirmation

Debug Information

The previous version of the debug information dialog was displayed to the user at the end of instrumentation. After the user dismissed the dialog there was no way to view the data again. The dialog was simply a warning of which DLLs had no debug information. The purpose of this was to alert the user as to why a given DLL had no code coverage (debug information is required to provide code coverage).

The new version of debug information dialog is available from the dashboard. The new dialog displays all DLLs and their status. Status information indicates if debug information was found or not found and if Coverage Validator is interested in that DLL, and if not interested, why it is not interested. This allows you to easily determine if a DLL filter is causing the DLL to be ignored for code coverage.

Coverage Debug Information

When the dialog is displayed a Learn More… link is available. This presents a simple dialog providing some information about debug information for debug and release builds. We’ve used a modified static control on these dialogs to provide useful bold text in dialogs (something that you can’t do with plain MFC applications). It’s a small thing but improves structure of the dialog. This text was displayed as part of the previous debug information dialog. Removing this text to a separate dialog chunks the information making it more accessible.

Coverage Debug Information Learn More

There is more to be done with this part of the software but this is an improvement compared to previous versions.

Tips Dialog

Coverage Validator has always had a “Tip of the day” dialog. This is something of a holdover from earlier forms of application development. We’d never really paid much attention to it, to how it functioned, how it behaved and what it communicated.

Tip of the day dialog

We’re planning to completely overhaul this dialog but that is a longer term activity. As such in this revision we’ve just made some smaller scale changes that still have quite an impact.

Tips dialog

The first change is that the previous “Tip of the day” dialog was displayed at application startup but with the new version the “Tips” dialog is no longer displayed at application startup. The tips dialog is now displayed when you launch an application and are waiting for instrumentation to complete. This means you get displayed tips during “dead time” that you can’t really use effectively – you’re waiting for the tool. The tips dialog is still available from the Help menu as was the previous Tip of the day dialog.

The second change is that the new Tips dialog is modeless. The previous Tip of the day dialog was modal. This means you can leave the dialog displayed and move it out of the way. You don’t have to dismiss it.

We’ve done away with the icon and replaced it with a tip number so you know which tip you are viewing. Tips are no longer viewed sequentially (Next Tip) but in a random order. At first this seems like a crazy thing to do. But when you try it, it actually increases your engagement. You’re wondering how many tips there are and which one you’ll get next. Hipmunk was an inspiration for this – they do something similar when calculating your plane flights (I hadn’t seen this when I used Hipmunk but Roger from ThinkUI had seen it).

There is more to be done with this part of the software but this is a useful improvement until our completely reworked Tips dialog is ready for release.


All of the changes have been made to improve and simplify the way information is communicated to the user of Coverage Validator. Improved graphics displays, interactive dashboards, better data dialogs, hyperlinks and occasional use of bold text all improve the user experience.

We’re not finished improving Coverage Validator. These are just our initial round of improvements.

Doing good work can make you feel a bit stupid

By , July 19, 2010 6:08 pm

Doing good work can make you feel a bit stupid, well thats my mixed bag of feelings for this weekend. Here is why…

Last week was a rollercoaster of a week for software development at Software Verification.

Off by one, again?

First off we found a nasty off-by-one bug in our nifty memory mapped performance tools, specifically the Performance Validator. The off-by-one didn’t cause any crashes or errors or bad data or anything like that. But it did cause us to eat memory like nobodies business. But for various reasons it hadn’t been found as it didn’t trigger any of our tests.

Then along comes a customer with his huge monolithic executable which won’t profile properly. He had already thrown us a curve balled by supplying it as a mixed mode app – half native C++, half C#. That in itself causes problems with profiling – the native profiler has to identify and ignore any functions that are managed (.Net). He was pleased with that turnaround but then surprised we couldn’t handle his app, as we had handled previous (smaller) versions of his app. The main reason he was using our profiler is that he had tried others and they couldn’t handle his app – and now neither could we! Unacceptable – well that was my first thought – I was half resigned to the fact that maybe there wasn’t a bug and this was just a goliath of an app that couldn’t be profiled.

I spent a day adding logging to every place, no matter how insignificant, in our function tree mapping code. This code uses shared memory mapped space exclusively, so you can’t refer to other nodes by addresses as the address in one process won’t be valid in the other processes reading the data. We had previously reorganised this code to give us a significant improvement in handling large data volumes and thus were surprised at the failure presented to us. Then came a long series of tests, each which was very slow (the logging writes to files and its a large executable to process). The logging data was huge. Some of the log files were GBs in size. Its amazing what notepad can open if you give it a chance!

Finally about 10 hours in I found the first failure. Shortly after that I found the root cause. We were using one of our memory mapped APIs for double duty. And as such the second use was incorrect – it was multiplying our correctly specified size by a prefixed size offset by one. This behaviour is correct for a different usage. Main cause of the problem – in my opinion, incorrectly named methods. A quick edit later and we have two more sensibly named methods and a much improved memory performance. A few tests later and a lot of logging disabled and we are back to sensible performance with this huge customer application (and a happy customer).

So chalk up one “how the hell did that happen?” followed by feelings of elation and pleasure as we fixed it so quickly.
I’m always amazed by off-by-one bugs. It doesn’t seem to matter how experienced you are – it does seem that they do reappear from time to time. Maybe that is one of the persils of logic for you, or tiredness.

I guess there is a Ph.D. for someone in studying CVS commits, file modification timestamps and off-by-one bugs and trying to map them to time-of-day/tiredness attributes.

That did eat my Wednesday and Thursday evenings, but it was worth it.

Not to be outdone…

I had always thought .Net Coverage Validator was a bit slow. It was good in GUI interaction tests (which is part of what .Net Coverage Validator is about – realtime code coverage feedback to aid testing) but not good on long running loops (a qsort() for example). I wanted to fix that. So following on from the success with the C++ profiling I went exploring an idea that had been rattling around in my head for some time. The Expert .Net 2.0 IL Assembler book (Serge Lidin, Microsoft Press) was an invaluable aid in this.

What were we doing that was so slow?

The previous (pre V3.00) .Net Coverage Validator implementation calls a method for each line that is visited in a .Net assembly. That method is in a unique DLL and has a unique ID. We were tracing application execution and when we found our specific method we’d walk up the callstack one item and that would be the location of a coverage line visit. This technique works, but it has a high overhead:

  1. ICorProfiler / ICorProfiler2 callback overhead.
  2. Callstack walking overhead.

The result is that for GUI operations, code coverage is fast enough that you don’t notice any problems. But for long running functions, or loops code coverage is very slow.

This needed replacing.

What are we doing now that is so fast?

The new implementation doesn’t trace methods or call a method of our choosing. For each line we modify a counter. The location of the counter and modification of it are placed directly into the ilAsm code for each C#./VB.Net method. Our first implementation of .Net Coverage Validator could not do this because our shared memory mapped coverage data architecture did not allow it – the shared memory may have moved during the execution run and thus the embedded counter location would be invalidated. The new architecture allows the pointer to the counter to be fixed.

The implementation and testing for this only took a few hours. Amazing. I thought it was going to fraught with trouble, not having done much serious ilAsm for a year or so.


The new architecture is so lightweight that you barely notice the performance overhead. Less than 1%. Your code runs just about at full speed even with code coverage in place.

As you can imagine, getting that implemented, working and tested in less than a day is an incredible feeling. Especially compared to the previous performance level we had.

So why feel stupid?

Having acheived such good performance (and naturally feeling quite good about yourself for a while afterwards) its hard not to look back on the previous implementation and think “Why did we accept that?, We could have done so much better”. And that is where the feeling stupid comes in. You’ve got to be self critical to improve. Pat yourself on the back for the good times and reflect on the past to try to recognise where you could have done better so that you don’t make the same mistake in the future.

And now for our next trick…

The inspiration for our first .Net Coverage Validator implementation came from our Java Coverage Validator tool. Java opcodes don’t allow you to modify memory directly like .Net ilAsm does, so we had to use the method calling technique for Java. However given our success with .Net we’ve gone back to the JVMTI header files (which didn’t exist when we first wrote the Java tools) and have found there may be a way to improve things. We’ll be looking at that soon.

Panorama Theme by Themocracy