Posts tagged: coverage validator

The Command Line Builder

By , April 4, 2022 1:32 pm

Last week we introduced the command line builder to all our Validator tools – Bug Validator, Coverage Validator, Memory Validator, Performance Validator and Thread Validator.

All our tools provide command line support allowing you to use them from your favourite automation tool, be that a set of batch files, or something more comprehensive like Jenkins.

Command lines are great, but writing out your command line can be daunting, even if like me, you’ve done it many times. It’s not that it’s hard, it’s just that you don’t do it very often so the options are never top of mind, you’re going to have to go and look at the dreaded documentation.

Yup, documentation, even when it’s as good as ours is, you’re still reluctant to look at it. I know the feeling. You’ve got so used to using a nice GUI that it seems odd to look at the help. But you’ve got to do that if you’re going to write a command line for a tool.

That’s where the Command Line Builder comes in. To reduce the amount of time spent looking at the documentation, and to reduce the likelihood of you making mistakes when creating your command line.

How do I start the command line builder?

The command line builder can be started from the Launch menu and from any of the Launch dialogs and Launch wizards.

Launch Menu

Launch menu

When the command line builder is started from the launch menu you are presented with a set of choices that allow you to preconfigure the command line builder.

Command line builder wizard

The choices provided aim to cover every possible way of using the Validator.

  • Build my own command line.
  • Use a predefined template that provides the command line arguments for common command line tasks. These templates are configured specific to the validator being used. If the Validator supports .Net and .Net Core as well as native there will be .Net specific templates and .Net Core specific templates. If the Validator can also do specific things like merging sessions or comparing sessions, templates for these actions are also provided.
  • Use an existing command line I use to launch my program. You know what you’d type on a command prompt to launch your program. Type that here and let the command line builder turn that into a command line for the Validator.
  • Use an existing Validator command line. Reuse a command line you already have and tweak it a bit to do the new task.
  • Use an existing Validator command file. Reuse a command file you already have and tweak it a bit to do the new task.

With the exception of the first option that allows you to build your command line from scratch, all of these options are present to enable you to save time when building your command line.

Once you’ve chosen your option (and template if you’re using a predefined template) you need to move to then next stage, with the thoughtfully named Next button.

Launch dialog / wizard

When you start the command line builder from a launch dialog or wizard it’s via the Cmd Line… button.

Launch dialog showing Cmd Line... button

When you start the command line builder this way, a command line is built based on the options you have selected on the launch dialog/wizard, and then passed to the command line builder. This allows you to select a known good command line you’ve previously used then click the Cmd Line… button and instantly be able to configure that command line in the command line builder.

Command Line Builder dialog

Once you’ve started the command line builder you’ll be presented with a dialog that will look very similar to this. This example is using a preconfigured template for Coverage Validator that merges coverage data to a central session, exports HTML and XML coverage data, and saves the coverage session.

The first thing you’ll notice is there are many argument values highlighted in red. These are values that are incorrect. This is because they are from a template and you’ll need to edit them to match your own working practices – in this example you’ll need to edit the program being tested and change c:\myResults to be where you wish to save your test results and edit the filenames to something meaningful for your tests.

Each command line argument is displayed on it’s own line, with a value (for the arguments that have a value) and a description of what this argument/value pair represent. If you need help, the ? button in the dialog border will take you there.

Editing

The buttons on the right hand side provide simple editing of the list of argument values to allow you to Add an argument, Edit an argument or value, Remove an argument and Remove All arguments. You can also start editing any existing argument or value by double clicking it.

Editing an argument presents you with a combo box of valid Validator argument values, preventing you from entering incorrect command line arguments.

Editing a value that is an On/Off, Yes/No, True/False type of argument presents you with those choices.

Editing a value that is an enumeration will only show you valid enumeration values.

Editing a value that needs a directory will open a directory chooser dialog, editing a value that needs a file will open a file chooser dialog set to choose that type of file.

All other editing types use a standard text dialog, with limited text input (for example integer values only allow integer input).

As you can see we are trying to reduce the scope for input error as much as we can.

Additional Customization

Next to the editing controls are some customization controls which allow you to add command line arguments (and associated values) to your command line if they are not already present.

Add Hide adds the -hideUI argument which you will need if you are writing command lines for automated use that close the Validator after the target program finishes.

Add Debug adds options to help debugging failures with the command line. These options ensure the command prompt is displayed, that the arguments you think you are supplying are the arguments you’ve supplied, displays error information on the command prompt and displays error information in a message box.

Add Export adds options for exporting in HTML and XML. For Coverage Validator an additional XML Cobertura export option is available.

A fourth button is displayed for some Validators. This provides options specific for that Validator, for example merging coverage data, or comparing memory leak reports or comparing performance reports. In the image show above you can see Add Merge, which is for Coverage Validator.

The last button allows you to Import an existing command file. Using this destroys all existing arguments and replaces them with the values in the command file.

Creating the command line

Once you’ve finished creating your command line the next thing is to get the command line in a form that you can use. We provide two forms of command line creation. The normal command line that most people expect and a much shorter command line combined with a command file that contains the actual command line instructions for the Validator.

The choice of command line is made using the combo box towards the lower half of the dialog.

Command line with arguments

This option is the typical command line that people expect. It starts the Validator and passes all command line arguments and values to the Validator.

The command line is updated each time the arguments and values are edited.

The Copy button copies the command line to the clipboard.

Command line with command file

This option provides a short command line that loads the command file, with the command file containing all the command line instructions for the Validator. There are two reasons for using the command file:

  • A shorter command line. Current versions of Windows support very long command lines. Earlier versions of Windows supported much shorter command lines – it was quite easy to construct a Validator command line that was too long for Windows to process, resulting in incorrect behaviour when you tried to use the command line. Using the command file allows you to specify long command lines on older operating systems.
  • Simplifying command line management. Storing all your command lines in command files allows you to easily manage the command files, storing them all in one place. It also allows some indirection, allowing you to edit the command line without editing the script/tool that invokes the Validator.

When you choose to use a command line with a command file, if you have not specified a command file to use you will be prompted for a command file name. When a command file name is specified the command file will be updated each time the arguments and values are edited.

The Browse… button allows you to use a file browser to choose a command file name. The View… button allows you to view the command file in your favourite editor.

Testing

Once you have created the command line arguments and values, and chosen the type of command line you want to use, you may want to test the command line. This is done by using the thoughtfully named Test Command Line button.

If all goes well a new Validator instance will be launched with your command line, and some time later the target program will complete.

Unfortunately not all command lines are equal, and some will fail to run correctly. So how do you debug that?

Incorrect program

If you have misconfigured the command line (for example: by specifying a 64 bit program to be tested by a 32 bit Validator) you will not get the behaviour you expect.

If you have specified the startup program correctly but failed to ensure that all dependent DLLs can be found the program won’t start, or it will start and then fail shortly afterwards. Check you can start the program in this directory by double clicking it – if that doesn’t work you’ve probably got a missing dependency. You can check your dependencies using PE File Browser.

If you have specified a startup directory that does not exist, the program won’t start.

Unable to see error dialogs and warning dialogs

If you have specified -hideUI you won’t be able to see what happens, so if you are having problems, the first thing to try is to change -hideUI to -showUI and test again to see if any warning dialogs are displayed.

Show the command prompt

Use -showCommandPrompt to ensure the command prompt can be seen. You may find a helpful error message waiting for you.

Check the arguments supplied

Use -echoArgsToUser to check the arguments supplied are what you think they should be.

Ensure you can see errors

Use -showErrorsWithMessageBox to display any error messages with a message box. Use this when your program has a graphical user interface.

Use -showErrorsOnCommandPrompt to display any error messages on the command prompt. Use this when your program is a console program and doesn’t have a GUI.

Conclusion

The Command Line Builder simplifies the creation of valid command lines for use with the Validator tools. We’ve tried to eliminate errors by making it easier to create valid command lines while also highlighting errors with proposed command line values.

Speeding up merging with Coverage Validator

By , December 16, 2015 11:43 am

Coverage Validator has an option to automatically merge the coverage results of the current session with a central session. This allows you to get an automatic overview of all code coverage without having to merge the results yourself.

Some people use this but some people prefer to record individual sessions then merge the sessions later. This is effective but the merging stage can be slow as to merge two files you need to start Coverage Validator, load two sessions then merge them then save the result. This is known as pairwise merging. Even with the command line support for this, this is time consuming.

-mergeMultiple

To speed this up we’ve just added the -mergeMultiple command line option.

-mergeMultiple takes one argument, a filename. The file contains the list of session files to merge, one per line.

Example command line:
-mergeMultiple e:\cv_merge_multiple.txt -mergeSessions -saveMergeResult e:\cv_merge_result.cvm -hideUI

Example merge multiple file:
e:\cv_help.cvm
e:\cv_red.cvm
e:\cv_green.cvm
e:\cv_blue.cvm
e:\cv_magenta.cvm
e:\cv_cyan.cvm
e:\this_file_doesnt_exist.cvm

Files that don’t exist are not merged. They do not cause any error conditions. This is deliberate – to provide fault tolerance if an intended merge target doesn’t exist for some reason. The last thing you want is a failed merge.

Performance improvement

We’ve tested this with one of our customers that could benefit from merging multiple files in one go. The performance improvement for merging 84 files (resulting in a 3.66GB merged session file – 64 bit Coverage Validator) is a speed up 8 times (pairwise merge time was 32 minutes, with -mergeMultiple the merge time is now 4 minutes).

UX Improvements for Coverage Validator

By , August 3, 2012 11:36 am

We recently released new versions of the Coverage Validator tools for all languages.

The main reason for this release was to make the tools more usable and make using them more satisfying. This work was inspired by some user experience research we commissioned with Think UI.

We’re so happy with these improvements we thought we’d share them so that you can learn from our improvements. We’re not finished with the Coverage Validator tools. This is just the start of changes to come.

I’m specifically going to talk about C++ Coverage Validator, but these improvements cut across all our Coverage Validator tools. Some of the improvements cut across all our development tools.

Summary Dashboard

The first thing a user of Coverage Validator sees is the summary dashboard.

The previous version of this dashboard was a grid with sparse use of graphics and lots of text. You had to read the text to understand what was happening with the code coverage for the test application. Additional comments and filter status information was displayed in right hand columns.

Coverage Validator old dashboard

The new version of this dashboard is split into two areas. The top area contains a dial for each metric reported. Each dial displays three items of information: Number of Items, Number of Items visited. How many items are 100% visited. This is done by means of an angular display for one value and a radial display for another value. A couple of the dials are pie charts.

The bottom area of the dashboard displays information that is relevant to the recorded session. Any value that can be viewed or edited is easily reachable via a hyperlink.

Coverage Validator new dashboard

The result of these changes are that the top area makes it easy to glance at a coverage report and instantly know which session has better coverage than another session. You don’t need to read the text to work it out. The bottom area draws attention to instrumentation failures (missing debug information, etc) and which filters are enabled etc. By exposing this information in this way more functionality of Coverage Validator is exposed to the user of the software.

Coverage Dials

We developed a custom control to display each coverage dial.

A coverage dial displays both the amount of data that has been visited, the amount of data that is unvisited and the amount that has been completely visited. For metrics that do not have a partial/complete status the dial just displays as a two part pie chart. An additional version displays data as a three part pie chart. This last version is used for displaying Unit Test results (success, failure, error).

Coverage dial directories Coverage dial unit tests

The difference between unvisited coverage and visited coverage is displayed using an angular value. Items that have been completely visited (100% coverage) are displayed using a radial value emanating from the centre of the dial. Additional information is displayed by a graded colour change between the 100% coverage area and the circumference of the circle to indicate the level of coverage in partially covered areas.

The coverage dial provides tooltips and hyperlinks for each section of the coverage dial.

Dashboard Status

The dashboard status area shows informational messages about the status of code instrumentation, a filter summary, unit test status and session merging status. Most items are either viewable or editable by clicking a hyperlink.

Dashboard status

To implement the hyperlink we created a custom control that supported hyperlinks with support for email hyper links, web hyper links and C++ callbacks. This provides maximal functionality. The hyperlinks are now used in many places in our tools – About box, evaluation feedback box, error report boxes, data export confirmation boxes, etc.

Coverage Scrollbar

We’ve also made high level overview data available on all the main displays (Coverage, Functions, Branches, Unit Tests, Files and Lines) so that you can get an overview of the coverage of each file/function/branch/etc without the need to scroll the view.

We thought of drawing the coverage data onto the scrollbar. Unfortunately this means that you would need an ownerdrawn scrollbar, but Windows does not provide such a thing. An option was to use a custom scroll bar implementation. But doing that would mean having to cater to every different type of Windows scrollbar implementation. We didn’t think that was a good idea. As such we’ve chosen to draw the coverage overview next to the scroll bar.

Coverage scroll bar

Editor Scrollbar

Similarly to the overview for each type of data we also provide a high level overview for the source code editor.

Editor code coverage

Directory Filter

Coverage Validator provides the ability to filter data on a variety of attributes. One of these is the directory in which a file is found. For example if the file was e:\om\c\svlWebAPI\webapi\ProductVersion\action.cpp the filter directory would be e:\om\c\svlWebAPI\webapi\ProductVersion\.

This is useful functionality but Coverage Validator allows you to filter on any directory. In complex software applications it’s quite possible that you would want to filter on a parent directory or a root directory. That would give the following directories for the example above.

e:
e:\om
e:\om\c
e:\om\c\svlWebAPI
e:\om\c\svlWebAPI\webapi
e:\om\c\svlWebAPI\webapi\ProductVersion

The solution to this problem is to create the context menu dynamically rather than use a preformed context menu stored in application resources. Additionally it is more likely that the current directory will be filtered rather than the parent, so it makes sense to reverse the order of the directories, going from leaf to root.

Coverage filter directory context menu

Instrumentation Preferences

The instrumentation preferences dialog is displayed to the user the first time that Coverage Validator starts. The purpose of this dialog is to configure the initial way coverage data is collected. This provides a range of performance levels from fast to slow and incomplete visit counts to complete visit counts. Both options affect the speed of execution of the software. Given that time to complete is an important cost for this is an option that should be chosen carefully.

Previous versions of the software displayed a wordy dialog containing two questions. Each question had two choices.

Coverage instrumentation preferences (old)

The new version of the instrumentation preference dialog has replaced the questions with a sliding scale. Two questions with two choices is effectively four combinations. The instrumentation level sliding scale has four values. As the slider value changes the text below the slider changes to provide a brief explanation of the instrumentation level chosen.

An additional benefit is that the previous version only implied the recommended values (we preset them). The new version also implies the recommended value (also preset) but also explicitly indicates the recommended instrumentation level.

This new design has less words, less visual clutter, is easier to use and presents less cognitive load on the user.

Coverage instrumentation preferences (new)

Export Confirmation

Coverage Validator provides options to export data to HTML and XML. A common desire after exporting is to view the exported data. Previous versions of Coverage Validator overlooked this desire, no doubt causing frustrating for some users. We’ve rectified this with a confirmation dialog displayed after exporting data. The options are to view the exported file or to view the contents of the folder holding the exported data. An option to never display the dialog again is also provided.

Coverage export confirmation

Debug Information

The previous version of the debug information dialog was displayed to the user at the end of instrumentation. After the user dismissed the dialog there was no way to view the data again. The dialog was simply a warning of which DLLs had no debug information. The purpose of this was to alert the user as to why a given DLL had no code coverage (debug information is required to provide code coverage).

The new version of debug information dialog is available from the dashboard. The new dialog displays all DLLs and their status. Status information indicates if debug information was found or not found and if Coverage Validator is interested in that DLL, and if not interested, why it is not interested. This allows you to easily determine if a DLL filter is causing the DLL to be ignored for code coverage.

Coverage Debug Information

When the dialog is displayed a Learn More… link is available. This presents a simple dialog providing some information about debug information for debug and release builds. We’ve used a modified static control on these dialogs to provide useful bold text in dialogs (something that you can’t do with plain MFC applications). It’s a small thing but improves structure of the dialog. This text was displayed as part of the previous debug information dialog. Removing this text to a separate dialog chunks the information making it more accessible.

Coverage Debug Information Learn More

There is more to be done with this part of the software but this is an improvement compared to previous versions.

Tips Dialog

Coverage Validator has always had a “Tip of the day” dialog. This is something of a holdover from earlier forms of application development. We’d never really paid much attention to it, to how it functioned, how it behaved and what it communicated.

Tip of the day dialog

We’re planning to completely overhaul this dialog but that is a longer term activity. As such in this revision we’ve just made some smaller scale changes that still have quite an impact.

Tips dialog

The first change is that the previous “Tip of the day” dialog was displayed at application startup but with the new version the “Tips” dialog is no longer displayed at application startup. The tips dialog is now displayed when you launch an application and are waiting for instrumentation to complete. This means you get displayed tips during “dead time” that you can’t really use effectively – you’re waiting for the tool. The tips dialog is still available from the Help menu as was the previous Tip of the day dialog.

The second change is that the new Tips dialog is modeless. The previous Tip of the day dialog was modal. This means you can leave the dialog displayed and move it out of the way. You don’t have to dismiss it.

We’ve done away with the icon and replaced it with a tip number so you know which tip you are viewing. Tips are no longer viewed sequentially (Next Tip) but in a random order. At first this seems like a crazy thing to do. But when you try it, it actually increases your engagement. You’re wondering how many tips there are and which one you’ll get next. Hipmunk was an inspiration for this – they do something similar when calculating your plane flights (I hadn’t seen this when I used Hipmunk but Roger from ThinkUI had seen it).

There is more to be done with this part of the software but this is a useful improvement until our completely reworked Tips dialog is ready for release.

Conclusion

All of the changes have been made to improve and simplify the way information is communicated to the user of Coverage Validator. Improved graphics displays, interactive dashboards, better data dialogs, hyperlinks and occasional use of bold text all improve the user experience.

We’re not finished improving Coverage Validator. These are just our initial round of improvements.

Panorama Theme by Themocracy