Rss Feed
Tweeter button
Facebook button
Technorati button
Reddit button
Myspace button
Linkedin button
Webonews button
Delicious button
Digg button
Flickr button
Stumbleupon button
Newsvine button

Category: Profiler

More Effective Line Timing with C++ Performance Validator

By , December 20, 2016 5:06 pm

C++ Performance Validator has always had the ability to provide timing information for each line executed. However it has always been a bit clumsy to use – you could only enable line timing for all the methods that were being timed. This could result in slow execution due to methods you didn’t want line timed being line timed. You could filter them by file using the “Hooked Source Files” filter or the “Hooked File Types” filter, but that was about as granular as it got.

We’ve just introduced the capability to specify which classes, class methods and functions get line timed. So now, if you want to time just one function, or one method in a class, or time just the methods in a class, you can tell C++ Performance Validator to do that.

To introduce this capability we’ve had to add an additional filters setting in the Settings dialog, we’ve updated the line timing warning dialog to allow you to edit line timing filters right from the warning dialog and we’ve had to update the context menus on all the displays to include new options for line timing filters. We’ve also had to update the display of filtering information on the line timing display.

Updated Settings Dialog

We’ve added a new “Line Timing Filters” section to the settings dialog.

pvlinetimingfilters

The image above shows three filters have been selected and that only the classes, methods and functions identified by the filters will be line timed. All other class, methods and functions will be ignored for line timing. All methods in the class seoPage will be filtered for line timing, as well as the method CSpellCheckTabDlg::addWord and the function strncasecmp.

Updated Line Timing Warnings

The Line Timing Warning dialog has been updated to allow you to edit line timing filters directly from the warning dialog. The Edit Line Timing Filters… button displays the Line Timing Filters dialog previously described.

C++ Performance Validator Line Timing Warning Dialog

Updated Context Menus

We’ve added a new submenu to the context menus on all displays and renamed the existing “Instrumentation” submenu to “Function Filter (instrumentation)”. The new submenu added is “Line Timing Filter (instrumentation)”. These new context menu entries allow you to easily specify that a class, a class method or a function should be selected for line timing.

pvlinetimingmenu

How we expect these filters to be used

There are two main methods for using the line timing filters that come to mind.

  • Specify the filters ahead of time.
  • Specify the filters in response to function timing data you are looking at.

Specify the filters ahead of time

This method relies on you knowing your software code and having a very good idea which methods you think need line timing. Open the settings dialog, go to “Line Timing Filters” and enter your filters as appropriate. Be sure to set the filters to include or exclude methods for the use case you think is useful. For line timing, most of the time we expect you to be using the include setting and specifying just a few methods to line time.

pvlinetimingfilters

Specify filters based on data.

The alternative way of using line timing is in response to some profiling data that you already have. Ideally you’ll have recorded this profiling data with line timing turned off (you can turn this on/off from the launch application dialog). Once you have the profiling data, search for functions that you think will benefit from additional tuning. When you have a candidate function, right click to get the context menu, then select if you want to profile the whole class, just that class method or just that function for line timing.

pvlinetimingfilterexample

At the point you add line timing filters, if you have no line timing filters or the line timing filters are set to profile all classes, methods and functions, then the line timing filters will be set to profile just the classes, methods and functions identified by the filters. However, if there are pre-existing filters, the include/exclude setting of the line timing filter will not be changed – you will need to visit the Line Timing Filters settings dialog to change that setting if you want to switch from include to exclude or exclude to include. This design is deliberate as the main use case is to profile one or two methods (or all the methods in one class) – that use case is “include”, which is why this is the default behaviour.

If you have already recorded a session with line timing data and you specify some line timing filters, C++ Performance Validator will update the line timing display to grey out the classes, class methods and functions that will not be instrumented on the next run.

pvlinetimingresultsfiltered

Line Timing Results

Once you have the filter(s) you need setup you can run a new profiling session to get the line timing data you need.

pvlinetimingresults

Share

New graphics in Performance Validator

By , December 18, 2015 12:32 pm

We’ve just introduced some new visualisations of performance data into C++ Performance Validator and .Net Performance Validator.

The new visualisations make it much easier to understand what is happening – the relationship between child and parent functions and where you should be looking to make performance improvements to your code.

The past

First, lets look at what the existing visualisations were:

Functions:
pv_old_functions

Lines:
pv_old_lines

These are not awful, but they’re not brilliant – they don’t really help you understand what’s happening.

The new visualisations

We’ve replaced the above visualisations with some new visualisations that are easier to read, easier to understand and allow you make better conclusions as to where to focus in your code.

Functions:
pv_new_functions

The top bar shows the function time and the average function time as fractions of the total runtime.

The middle bar shows the shortest function time and longest function time as fractions of the total runtime.

The bottom bar shows the function time, the average function and the child time as fractions of the total time for the function.

You can improve your code by looking at each performance bar and evaluating them as shown below.

image03

Lines:

pv_new_lines

The top bar shows the line time and the average line time as fractions of the function time.

The bottom bar shows the shortest line time and longest lines time as fractions of the function time.

These allow you to identify the lines that are most dominant and also which have the greatest variability in execution time.

Conclusion

The new visualisations provide more insight into the performance data than the previous visualisations.

Credits

I’d like to mention the user experience consultancy Think UI for providing the design for these performance bars and for the very useful explanatory graphic that goes with them.

Share

64 bit C++ software tool Beta Tests are complete.

By , January 9, 2014 1:33 pm

We recently closed the beta tests for the 64 bit versions of C++ Coverage Validator, C++ Memory Validator, C++ Performance Validator and C++ Thread Validator.

We launched the software on 2nd January 2014. A soft launch, no fanfare, no publicity. We just wanted to make the software available and then contact all the beta testers so that we could honour our commitments made at the start of the beta test.

Those commitments were to provide a free single user licence to any beta tester that provided feedback, usage reports, bugs reports, etc about the software. This doesn’t include anyone that couldn’t install the software because they used the wrong licence key!

We’ve written a special app here that we can use to identify all email from beta test participants and allow us to evaluate that email for beta test feedback criteria. It’s saved us a ton of time and drudge work even though writing this extension to the licence manager software took a few days. It was interesting using the tool and seeing who provided feedback and how much.

We’ve just sent out the licence keys and download instructions to all those beta testers that were kind enough to take the time to provide feedback, bug reports etc. to us. A few people went the extra mile. These people bombarded us with email containing huge bugs, trivial items and everything in between. Two of them, we were on the verge of flying out to their offices when we found some useful software that allowed to us to remotely debug their software. Special mentions go to:

Bengt Gunne (Mimer.com)
Ciro Ettorre (Mechworks.com)
Kevin Ernst (Bentley.com)

We’re very grateful for everyone taking part in the beta test. Thank you very much.

Why didn’t I get a free licence?

If you didn’t receive a free licence and you think you did provide feedback, please contact us. It’s always possible that a few people slipped through our process of identifying people.

Dang! I knew I should’ve provided feedback

If you didn’t provide us with any feedback, check your inbox. You’ll find a 50% off coupon for the tool that you tested.

Share

Command line support for .Net services and ASP.Net

By , September 29, 2011 3:59 pm

Today we have released updated versions of our software tools for .Net, .Net Coverage Validator, .Net Memory Validator and .Net Performance Validator.

The updates to each tool add support for monitoring .Net services and ASP.Net processes when working with the tool from the command line. This allows you to, for example, control the tool from batch files and easy create large suites of tests that you can run and control from batch files or using other scripting technologies. This command line support builds on the already existing command line support in each tool for working with .Net desktop appplications. For information on existing command line options for each tool please see the help file that ships with each tool.

I’m going to outline the new command line options for each tool and provide some basic examples of how you might use these options with each tool. Each tool has been given the same basic options, with each tool getting some additional options specific to that tool.

.Net Coverage Validator

  • -serviceName fileName

    The -serviceName option is used to specify which service .Net Coverage Validator should monitor. The filename argument should be quoted if the filename contains spaces.

    -serviceName c:\path\myservice.exe
    -serviceName "c:\path with spaces\myservice.exe"
    
  • -urlToVisit url

    The -urlToVisit option specifies the web page that should be opened by the web browser when working with ASP.Net web servers.

    -urlToVisit http://localhost/myTestPage.aspx
    -urlToVisit "http://localhost/myTestPage.aspx"
    
  • -aspNetName filename

    The -aspNetName option is used to specify the ASP.Net process that is used by IIS. The filename argument should be quoted if the filename contains spaces.

    -aspNetName c:\windows\Microsoft.Net\Framework\v4.0.30319\aspnet_wp.exe
    -aspNetName "c:\windows\Microsoft.Net\Framework\v4.0.30319\aspnet_wp.exe"
    

    The value specified should be the value that you would specify if you used the .Net Coverage Validator interface to work with ASP.Net applications.

  • -webRoot directoryname

    The -webRoot option is used to specify the web root for this ASP.Net process. The directoryname argument should be quoted if the filename contains spaces.

    -webRoot c:\inetpub\wwwroot
    -webRoot "c:\inetpub\wwwroot"
    
  • -webBrowser filename

    The -webBrowser option is used to specify which web browser to use to open the web page if the user has chosen to specify a particular web browser. This option is used when the -aspNetWeb option specifies to use a specific web browser. The filename argument should be quoted if the filename contains spaces.

    -webBrowser c:\mozilla\firefox.exe
    -webBrowser "c:\program files\internet explorer\iexplore.exe"
    
  • -coverageDirectory directoryname

    The -coverageDirectory option specifies the directory .Net Coverage Validator will use to communicate with the GUI if security privileges do not allow named pipes and shared memory usage. The directoryname argument should be quoted if the filename contains spaces.

    -coverageDirectory c:\temp
    
  • -aspNetWeb default|user|specific

    The -aspNetWeb option is used to specify which web browser will be used to open the web page you have specified.

    The options are:

    default Use the default web browser.
    user The user will open a web browser themselves.
    specific Use a web browser identified by a filepath. Use in conjunction with -webBrowser option.
    -aspNetWeb default
    -aspNetWeb user
    -aspNetWeb specfic
    
  • -aspNetDelay integer

    The -aspNetDelay option is used to specify how long .Net Coverage Validator will wait for IIS to reset and restart itself. The delay is specified in milliseconds.

    -aspNetDelay 5000
    

Working with .Net services

This example shows how to use .Net Coverage Validator with a .Net service.

dnCoverageValidator.exe -serviceName E:\WindowsService\bin\Debug\WindowsService.exe 
-coverageDirectory c:\test\coverage -saveSession "c:\test results\testbed.dncvm" 
-hideUI
  • -serviceName E:\WindowsService\bin\Debug\WindowsService.exe

    This specifies the service to monitor. The service must be started after .Net Coverage Validator has been instructed to monitor the service.

  • -coverageDirectory c:\test\coverage

    This specifies the directory .Net Coverage Validator will use to communicate with the GUI if security privileges do not allow named pipes and shared memory usage.

  • -saveSession “c:\test results\testbed.dncvm”

    This specifies that after the application finishes the session should be saved in the file c:\test results\testbed.dncvm.

  • -hideUI

    This specifies that the user interface should not be displayed during the test. When the target service closes .Net Coverage Validator will close.

Working with ASP.Net

This example shows how to use .Net Coverage Validator with ASP.Net.

dnCoverageValidator.exe -urlToVisit http://localhost/testWebApp.aspx 
-aspNetName c:\windows\Microsoft.Net\Framework\v4.0.30319\aspnet_wp.exe 
-aspNetWeb default -aspNetDelay 5000 -webRoot c:\inetput\wwwroot 
-coverageDirectory c:\test\coverage -saveSession "c:\test results\testbed.dncvm" 
-hideUI
  • -urlToVisit http://localhost/testWebApp.aspx

    This specifies the web page that will be opened when working with ASP.Net

  • -aspNetName c:\windows\Microsoft.Net\Framework\v4.0.30319\aspnet_wp.exe

    This specifies the ASP.Net worker process that IIS will start.

  • -aspNetWeb default

    This specifies that the system defined web browser will be used to open the web page.

  • -aspNetDelay 5000

    This specifies a delay of 5 seconds to allow the IIS webserver to restart.

  • -webRoot c:\inetput\wwwroot

    This specifies the web root of the IIS web server.

  • -coverageDirectory c:\test\coverage

    This specifies the directory .Net Coverage Validator will use to communicate with the GUI if security privileges do not allow named pipes and shared memory usage.

  • -saveSession “c:\test results\testbed.dncvm”

    This specifies that after the application finishes the session should be saved in the file c:\test results\testbed.dncvm.

  • -hideUI

    This specifies that the user interface should not be displayed during the test. When the target service closes .Net Coverage Validator will close.

.Net Memory Validator

  • -collectData

    The -collectData option causes .Net Memory Validator to collect memory allocation events until the user chooses to disable data collection from the user interface.

    -collectData
    
  • -doNotCollectData

    The -doNotCollectData option causes .Net Memory Validator to ignore memory allocation events until the user chooses to enable data collection from the user interface.

    -doNotCollectData
    
  • -serviceName fileName

    The -serviceName option is used to specify which service .Net Memory Validator should monitor. The filename argument should be quoted if the filename contains spaces.

    -serviceName c:\path\myservice.exe
    -serviceName "c:\path with spaces\myservice.exe"
    
  • -urlToVisit url

    The -urlToVisit option specifies the web page that should be opened by the web browser when working with ASP.Net web servers.

    -urlToVisit http://localhost/myTestPage.aspx
    -urlToVisit "http://localhost/myTestPage.aspx"
    
  • -aspNetName filename

    The -aspNetName option is used to specify the ASP.Net process that is used by IIS. The filename argument should be quoted if the filename contains spaces.

    -aspNetName c:\windows\Microsoft.Net\Framework\v4.0.30319\aspnet_wp.exe
    -aspNetName "c:\windows\Microsoft.Net\Framework\v4.0.30319\aspnet_wp.exe"
    

    The value specified should be the value that you would specify if you used the .Net Memory Validator interface to work with ASP.Net applications.

  • -webRoot directoryname

    The -webRoot option is used to specify the web root for this ASP.Net process. The directoryname argument should be quoted if the filename contains spaces.

    -webRoot c:\inetpub\wwwroot
    -webRoot "c:\inetpub\wwwroot"
    
  • -webBrowser filename

    The -webBrowser option is used to specify which web browser to use to open the web page if the user has chosen to specify a particular web browser. This option is used when the -aspNetWeb option specifies to use a specific web browser. The filename argument should be quoted if the filename contains spaces.

    -webBrowser c:\mozilla\firefox.exe
    -webBrowser "c:\program files\internet explorer\iexplore.exe"
    
  • -aspNetWeb default|user|specific

    The -aspNetWeb option is used to specify which web browser will be used to open the web page you have specified.

    The options are:

    default Use the default web browser.
    user The user will open a web browser themselves.
    specific Use a web browser identified by a filepath. Use in conjunction with -webBrowser option.
    -aspNetWeb default
    -aspNetWeb user
    -aspNetWeb specfic
    
  • -aspNetDelay integer

    The -aspNetDelay option is used to specify how long .Net Memory Validator will wait for IIS to reset and restart itself. The delay is specified in milliseconds.

    -aspNetDelay 5000
    

Working with .Net services

This example shows how to use .Net Memory Validator with a .Net service.

dnMemoryValidator.exe -serviceName E:\WindowsService\bin\Debug\WindowsService.exe 
-saveSession "c:\test results\testbed.dnmvm" -hideUI
  • -serviceName E:\WindowsService\bin\Debug\WindowsService.exe

    This specifies the service to monitor. The service must be started after .Net Memory Validator has been instructed to monitor the service.

  • -saveSession “c:\test results\testbed.dnmvm”

    This specifies that after the application finishes the session should be saved in the file c:\test results\testbed.dnmvm.

  • -hideUI

    This specifies that the user interface should not be displayed during the test. When the target service closes .Net Memory Validator will close.

Working with ASP.Net

This example shows how to use .Net Memory Validator with ASP.Net.

dnMemoryValidator.exe -urlToVisit http://localhost/testWebApp.aspx 
-aspNetName c:\windows\Microsoft.Net\Framework\v4.0.30319\aspnet_wp.exe 
-aspNetWeb default -aspNetDelay 5000 -webRoot c:\inetput\wwwroot 
-saveSession "c:\test results\testbed.dnmvm" -hideUI
  • -urlToVisit http://localhost/testWebApp.aspx

    This specifies the web page that will be opened when working with ASP.Net

  • -aspNetName c:\windows\Microsoft.Net\Framework\v4.0.30319\aspnet_wp.exe

    This specifies the ASP.Net worker process that IIS will start.

  • -aspNetWeb default

    This specifies that the system defined web browser will be used to open the web page.

  • -aspNetDelay 5000

    This specifies a delay of 5 seconds to allow the IIS webserver to restart.

  • -webRoot c:\inetput\wwwroot

    This specifies the web root of the IIS web server.

  • -saveSession “c:\test results\testbed.dnmvm”

    This specifies that after the application finishes the session should be saved in the file c:\test results\testbed.dnmvm.

  • -hideUI

    This specifies that the user interface should not be displayed during the test. When the target service closes .Net Memory Validator will close.

.Net Performance Validator

  • -collectData

    The -collectData option causes .Net Performance Validator to collect memory allocation events until the user chooses to disable data collection from the user interface.

    -collectData
    
  • -doNotCollectData

    The -doNotCollectData option causes .Net Performance Validator to ignore memory allocation events until the user chooses to enable data collection from the user interface.

    -doNotCollectData
    
  • -collectFunctionTimes

    The -collectFunctionTimes option causes .Net Performance Validator to collect timing information for functions in the application/service/ASP.Net webserver.

    -collectFunctionTimes
    
  • -collectLineTimes
    The -collectLinesTimes option causes .Net Performance Validator to collect timing information for lines in the application/service/ASP.Net webserver.

    -collectLineTimes
    
  • -doNotCollectFunctionTimes
    The -doNotCollectFunctionTimes option causes .Net Performance Validator not to collect timing information for functions in the application/service/ASP.Net webserver.

    -doNotCollectFunctionTimes
    
  • -doNotCollectLineTimes
    The -doNotCollectLinesTimes option causes .Net Performance Validator not to collect timing information for lines in the application/service/ASP.Net webserver.

    -doNotCollectLineTimes
    
  • -serviceName fileName

    The -serviceName option is used to specify which service .Net Performance Validator should monitor. The filename argument should be quoted if the filename contains spaces.

    -serviceName c:\path\myservice.exe
    -serviceName "c:\path with spaces\myservice.exe"
    
  • -urlToVisit url

    The -urlToVisit option specifies the web page that should be opened by the web browser when working with ASP.Net web servers.

    -urlToVisit http://localhost/myTestPage.aspx
    -urlToVisit "http://localhost/myTestPage.aspx"
    
  • -aspNetName filename

    The -aspNetName option is used to specify the ASP.Net process that is used by IIS. The filename argument should be quoted if the filename contains spaces.

    -aspNetName c:\windows\Microsoft.Net\Framework\v4.0.30319\aspnet_wp.exe
    -aspNetName "c:\windows\Microsoft.Net\Framework\v4.0.30319\aspnet_wp.exe"
    

    The value specified should be the value that you would specify if you used the .Net Performance Validator interface to work with ASP.Net applications.

  • -webRoot directoryname

    The -webRoot option is used to specify the web root for this ASP.Net process. The directoryname argument should be quoted if the filename contains spaces.

    -webRoot c:\inetpub\wwwroot
    -webRoot "c:\inetpub\wwwroot"
    
  • -webBrowser filename

    The -webBrowser option is used to specify which web browser to use to open the web page if the user has chosen to specify a particular web browser. This option is used when the -aspNetWeb option specifies to use a specific web browser. The filename argument should be quoted if the filename contains spaces.

    -webBrowser c:\mozilla\firefox.exe
    -webBrowser "c:\program files\internet explorer\iexplore.exe"
    
  • -profilerDirectory directoryname

    The -profilerDirectory option specifies the directory .Net Performance Validator will use to communicate with the GUI if security privileges do not allow named pipes and shared memory usage. The directoryname argument should be quoted if the filename contains spaces.

    -profilerDirectory c:\temp
    
  • -aspNetWeb default|user|specific

    The -aspNetWeb option is used to specify which web browser will be used to open the web page you have specified.

    The options are:

    default Use the default web browser.
    user The user will open a web browser themselves.
    specific Use a web browser identified by a filepath. Use in conjunction with -webBrowser option.
    -aspNetWeb default
    -aspNetWeb user
    -aspNetWeb specfic
    
  • -aspNetDelay integer

    The -aspNetDelay option is used to specify how long .Net Performance Validator will wait for IIS to reset and restart itself. The delay is specified in milliseconds.

    -aspNetDelay 5000
    

Working with .Net services

This example shows how to use .Net Performance Validator with a .Net service.

dnPerformanceValidator.exe -serviceName E:\WindowsService\bin\Debug\WindowsService.exe 
-profilerDirectory c:\test\profiler -saveSession "c:\test results\testbed.dnpvm" 
-hideUI
  • -serviceName E:\WindowsService\bin\Debug\WindowsService.exe

    This specifies the service to monitor. The service must be started after .Net Performance Validator has been instructed to monitor the service.

  • -profilerDirectory c:\test\profiler

    This specifies the directory .Net Performance Validator will use to communicate with the GUI if security privileges do not allow named pipes and shared memory usage.

  • -saveSession “c:\test results\testbed.dnpvm”

    This specifies that after the application finishes the session should be saved in the file c:\test results\testbed.dnpvm.

  • -hideUI

    This specifies that the user interface should not be displayed during the test. When the target service closes .Net Performance Validator will close.

Working with ASP.Net

This example shows how to use .Net Performance Validator with ASP.Net.

dnPerformanceValidator.exe -urlToVisit http://localhost/testWebApp.aspx 
-aspNetName c:\windows\Microsoft.Net\Framework\v4.0.30319\aspnet_wp.exe 
-aspNetWeb default -aspNetDelay 5000 -webRoot c:\inetput\wwwroot 
-profilerDirectory c:\test\profiler -saveSession "c:\test results\testbed.dnpvm" 
-hideUI
  • -urlToVisit http://localhost/testWebApp.aspx

    This specifies the web page that will be opened when working with ASP.Net

  • -aspNetName c:\windows\Microsoft.Net\Framework\v4.0.30319\aspnet_wp.exe

    This specifies the ASP.Net worker process that IIS will start.

  • -aspNetWeb default

    This specifies that the system defined web browser will be used to open the web page.

  • -aspNetDelay 5000

    This specifies a delay of 5 seconds to allow the IIS webserver to restart.

  • -webRoot c:\inetput\wwwroot

    This specifies the web root of the IIS web server.

  • -profilerDirectory c:\test\profiler

    This specifies the directory .Net Performance Validator will use to communicate with the GUI if security privileges do not allow named pipes and shared memory usage.

  • -saveSession “c:\test results\testbed.dnpvm”

    This specifies that after the application finishes the session should be saved in the file c:\test results\testbed.dnpvm.

  • -hideUI

    This specifies that the user interface should not be displayed during the test. When the target service closes .Net Performance Validator will close.

Share

Perl Profiling API

By , August 6, 2011 3:48 pm

For some time we have had tools for use with Perl in the beta stage. The reason the tools never left beta is because the instrumentation part of the tools was tool brittle. It relied up scanning the Perl binary for suitable insertion points then modifying the binary to insert a suitable profiling callback. The callback would then ask the interpreter for filename, line number and function name information.

This worked fine so long as the version of Perl being used had data stored in the same format as the version we had inspected when we wrote the tools. As soon as the internal design of the Perl runtime changed our tools stopped working.

The Perl interpreter has some rudimentary profiling capabilities built in. The are only suitable for gathering timing statistics and cannot be used to provide code coverage flow tracing or call graph timing results. The only was to get statistics for these uses is to build your own profiling API. That is what we have done with the Perl Profiling API.

The Perl Profiling API is inspired by similar functionality in the Python interpreter, the Ruby interpreter and the Mozilla JavaScript interpreter. The API consists of four functions:

  • void Perl_set_do_profiling(int enable);
  • int Perl_get_do_profiling();
  • void Perl_set_coverage_callback(PERL_PROFILER_CALLBACK callback, void *userData);
  • void Perl_set_profiling_callback(PERL_PROFILER_CALLBACK callback, void *userData);

PERL_PROFILER_CALLBACK

PERL_PROFILER_CALLBACK is defined as

void (*PERL_PROFILER_CALLBACK)(PERL_PROFILER_EVENT event,
                               const char          *fileName,
                               const int            lineNumber,
                               const char          *functionName,
                               const char          *packageName,
                               void                *userData);

The PERL_PROFILER_EVENT event can receive values of PPE_LINE, PPE_CALL, PPE_RETURN. PPE_LINE is used for line visits, PPE_CALL is used when a function is called, PPE_RETURN is used when a function returns. The fileName and lineNumber specify which part of which fileName the code is executing, the functionName specifies the current function and the packageName specifies which package/namespace the function is part of (if any). The userData value is the value passed when setting the callback using Perl_set_coverage_callback() and/or Perl_set_profiling_callback.

void Perl_set_do_profiling(int enable);

Perl_set_do_profiling() is used to enable or disable profiling with either of the callbacks. Pass TRUE to enable and FALSE to disable.

int Perl_get_do_profiling();

Perl_get_do_profiling() is used to determine if profiling is enabled or not. The function returns TRUE for enabled and FALSE for disabled.

void Perl_set_coverage_callback(PERL_PROFILER_CALLBACK callback, void *userData);

Perl_set_coverage_callback() is used to set the callback that will be called for line visit events. The callback function will be called for all lines the Perl program visits. The userData value specified will be passed to the callback.

void Perl_set_profiling_callback(PERL_PROFILER_CALLBACK callback, void *userData);

Perl_set_profiling_callback() is used to set the callback that will be called for function call and function return events. The callback function will be called for all functions the Perl program visits. The userData value specified will be passed to the callback.

Using the Perl Profiling API

By default the Perl Profiling API is not enabled and no callback values are set.

To use the Perl Profiling API we need to define a callback to do the work.

void callback(PERL_PROFILER_EVENT event,
              const char          *fileName,
              const int            lineNumber,
              const char          *functionName,
              const char          *packageName,
              void                *userData)
{
    switch(event)
    {
    case PPE_LINE:
        storeLineVisitInfo(fileName, lineNumber, userData);
        break;

    case PPE_CALL:
        startFunctionCallTiming(fileName, lineNumber, functionName, packageName, userData);
        break;

    case PPE_RETURN:
        endFunctionCallTiming(fileName, lineNumber, functionName, packageName, userData);
        break;
    }
}

The implementation of storeLineVisitInfo(), startFunctionCallTiming() and endFunctionCallTiming() are down to the profiler writer. You can use these to implement a timing profiler, code coverage, flow tracing or some other tool.

We also need to tell the Perl to use the Perl Profiling API.

	Perl_set_coverage_callback(callback, userData);
	Perl_set_profiling_callback(callback, userData);
	Perl_set_dp_profiling(TRUE);

Conclusion

As you can see using the Perl Profiling API is easy.

Adding the Perl Profiling API to the version of Perl you are using is also straightforward – there are two files that implement the API and two simple modifications to the runtime loop in dump.c and run.c. The overhead of the API is trivial when not in use and when in use the overhead is defined by what the callback writer chooses to do.

Learn more about the sources and binaries for the Perl Profiling API.

Share

Doing good work can make you feel a bit stupid

By , July 19, 2010 6:08 pm

Doing good work can make you feel a bit stupid, well thats my mixed bag of feelings for this weekend. Here is why…

Last week was a rollercoaster of a week for software development at Software Verification.

Off by one, again?

First off we found a nasty off-by-one bug in our nifty memory mapped performance tools, specifically the Performance Validator. The off-by-one didn’t cause any crashes or errors or bad data or anything like that. But it did cause us to eat memory like nobodies business. But for various reasons it hadn’t been found as it didn’t trigger any of our tests.

Then along comes a customer with his huge monolithic executable which won’t profile properly. He had already thrown us a curve balled by supplying it as a mixed mode app – half native C++, half C#. That in itself causes problems with profiling – the native profiler has to identify and ignore any functions that are managed (.Net). He was pleased with that turnaround but then surprised we couldn’t handle his app, as we had handled previous (smaller) versions of his app. The main reason he was using our profiler is that he had tried others and they couldn’t handle his app – and now neither could we! Unacceptable – well that was my first thought – I was half resigned to the fact that maybe there wasn’t a bug and this was just a goliath of an app that couldn’t be profiled.

I spent a day adding logging to every place, no matter how insignificant, in our function tree mapping code. This code uses shared memory mapped space exclusively, so you can’t refer to other nodes by addresses as the address in one process won’t be valid in the other processes reading the data. We had previously reorganised this code to give us a significant improvement in handling large data volumes and thus were surprised at the failure presented to us. Then came a long series of tests, each which was very slow (the logging writes to files and its a large executable to process). The logging data was huge. Some of the log files were GBs in size. Its amazing what notepad can open if you give it a chance!

Finally about 10 hours in I found the first failure. Shortly after that I found the root cause. We were using one of our memory mapped APIs for double duty. And as such the second use was incorrect – it was multiplying our correctly specified size by a prefixed size offset by one. This behaviour is correct for a different usage. Main cause of the problem – in my opinion, incorrectly named methods. A quick edit later and we have two more sensibly named methods and a much improved memory performance. A few tests later and a lot of logging disabled and we are back to sensible performance with this huge customer application (and a happy customer).

So chalk up one “how the hell did that happen?” followed by feelings of elation and pleasure as we fixed it so quickly.
I’m always amazed by off-by-one bugs. It doesn’t seem to matter how experienced you are – it does seem that they do reappear from time to time. Maybe that is one of the persils of logic for you, or tiredness.

I guess there is a Ph.D. for someone in studying CVS commits, file modification timestamps and off-by-one bugs and trying to map them to time-of-day/tiredness attributes.

That did eat my Wednesday and Thursday evenings, but it was worth it.

Not to be outdone…

I had always thought .Net Coverage Validator was a bit slow. It was good in GUI interaction tests (which is part of what .Net Coverage Validator is about – realtime code coverage feedback to aid testing) but not good on long running loops (a qsort() for example). I wanted to fix that. So following on from the success with the C++ profiling I went exploring an idea that had been rattling around in my head for some time. The Expert .Net 2.0 IL Assembler book (Serge Lidin, Microsoft Press) was an invaluable aid in this.

What were we doing that was so slow?

The previous (pre V3.00) .Net Coverage Validator implementation calls a method for each line that is visited in a .Net assembly. That method is in a unique DLL and has a unique ID. We were tracing application execution and when we found our specific method we’d walk up the callstack one item and that would be the location of a coverage line visit. This technique works, but it has a high overhead:

  1. ICorProfiler / ICorProfiler2 callback overhead.
  2. Callstack walking overhead.

The result is that for GUI operations, code coverage is fast enough that you don’t notice any problems. But for long running functions, or loops code coverage is very slow.

This needed replacing.

What are we doing now that is so fast?

The new implementation doesn’t trace methods or call a method of our choosing. For each line we modify a counter. The location of the counter and modification of it are placed directly into the ilAsm code for each C#./VB.Net method. Our first implementation of .Net Coverage Validator could not do this because our shared memory mapped coverage data architecture did not allow it – the shared memory may have moved during the execution run and thus the embedded counter location would be invalidated. The new architecture allows the pointer to the counter to be fixed.

The implementation and testing for this only took a few hours. Amazing. I thought it was going to fraught with trouble, not having done much serious ilAsm for a year or so.

Result?

The new architecture is so lightweight that you barely notice the performance overhead. Less than 1%. Your code runs just about at full speed even with code coverage in place.

As you can imagine, getting that implemented, working and tested in less than a day is an incredible feeling. Especially compared to the previous performance level we had.

So why feel stupid?

Having acheived such good performance (and naturally feeling quite good about yourself for a while afterwards) its hard not to look back on the previous implementation and think “Why did we accept that?, We could have done so much better”. And that is where the feeling stupid comes in. You’ve got to be self critical to improve. Pat yourself on the back for the good times and reflect on the past to try to recognise where you could have done better so that you don’t make the same mistake in the future.

And now for our next trick…

The inspiration for our first .Net Coverage Validator implementation came from our Java Coverage Validator tool. Java opcodes don’t allow you to modify memory directly like .Net ilAsm does, so we had to use the method calling technique for Java. However given our success with .Net we’ve gone back to the JVMTI header files (which didn’t exist when we first wrote the Java tools) and have found there may be a way to improve things. We’ll be looking at that soon.

Share

The cost of using OutputDebugString

By , May 21, 2010 10:02 am

Should you use OutputDebugString? Is that an unusual question to ask? How often have you thought about the potential cost of using OutputDebugString (or TRACE, which uses OutputDebugString under the hood)?

Benefits of using OutputDebugString

The benefit of using OutputDebugString is the ability to leave a relatively benign function in your code that can output useful textual information which can be monitored by a debugger or a suitable utility (such as DebugView).

The TRACE() macro allows you to output information, but only in Debug builds. OutputDebugString() allows you to output the information in Debug and Release builds.

Problems caused by using OutputDebugString

The problem of using OutputDebugString is that it has a performance overhead, which although minimal outside a debugger is much higher in a debugger. If in a busy loop this overhead can be an unwanted burden.

It may be that the information being output by OutputDebugString is not information that you want your customers (or competitors!) to see.

Finally, depending on your software application, it may be that your customers will be using your software component (for example our injected monitoring DLLs) with their software in their debugger, debugging their software. In that situation, your customer may not appreciate the extra OutputDebugString() information filling up their Output tab with information and obscuring whatever information their own OutputDebugString() usage is providing.

I’m sorry to say it, but we have been guilty of this in the past! You may want to check your code to ensure you are not doing this by accident. Its all to easy to let things like this happen – after all there is no obvious adverse effect (like a crash or undefined behaviour) to fail your testing patterns.

Performance cost of using OutputDebugString

I’ve noticed questions asking about the cost of OutputDebugString() on a few forums but never seen any hard numbers to go with the received opinions offered. Being a curious animal I decided that I should investigate. The benefits being that I get to scratch this curious itch and if the news is bad we get to make a few modifications to the software at Software Verification.

OutputDebugString() comes in two flavours, OutputDebugStringA() for ANSI builds and OutputDebugStringW() for Unicode builds. The tests I ran tested both of these WIN32 APIs on Windows XP 64 bit, on a dual quad core Intel Xeon running at 2.83GHz. All tests done under the same load.

Testing OutputDebugString

To test the APIs we need to test a few scenarios:

  • Calling the API when running the test application outside of a debugger. This will be the equivalent of if you leave OutputDebugString() calls in your release mode application and ship it to your customer.
  • Call the API when running the test application in a debugger. This tests the overhead of OutputDebugString() communicating with the debugger so that the debugger can display the message on its output pane (assuming the debugger does that).
  • Call the API when running the test application in a debugger, adding a \r\n at the end of each line. This tests the overhead of OutputDebugString() communicating with the debugger so that the debugger can display the message on its output pane (assuming the debugger does that).

We have chosen to test with Visual Studio 6.0, Visual Studio 2005 and Visual Studio 2010. We have chosen these two IDEs/debuggers since VS6 is the old pre-.Net IDE which is still well loved by a lot of developers. We have also chosen VS2005 because based on what we can tell from our customer base this is a very popular IDE/debugger.

The test for each scenario consists of outputting three different size strings 4000 times. The three strings are a zero length strings, a short string and a longer string. The test is also repeated with the same strings with \r\n appended to the non-zero length strings. We added this test when we realized that the Visual Studio output panel behaves differently between VS6 and VS2005 for lines that do not contain \r\n at the end.

You can download full source code and project files for both VS6 and VS2005 so that you can build and run these tests for yourself. For VS2010 you can load the VS2005 solution file and convert it automatically during the load.

timeOutputDebugString screenshot

Results

We have 7 groups of test results spanning no debugger, Visual Studio 6, Visual Studio 2005, and Visual Studio 2010.

Test 1 – No Debugger

OutputDebugString() called 4000 times, no debugger monitoring the process. No \r\n per output.
What we can see is that OutputDebugStringW() is 9% slower than OutputDebugStringA() and that both calls are very fast.

Function String Time
OutputDebugStringA "" 0.0112s
OutputDebugStringA short string 0.0198s
OutputDebugStringA long string 0.0255s
Average 0.00000470s
 
OutputDebugStringW "" 0.0121s
OutputDebugStringW short string 0.0214s
OutputDebugStringW long string 0.0281s
Average 0.00000513s

Test 2 – Visual Studio 6

OutputDebugString() called 4000 times, Visual Studio 6 monitoring the process. No \r\n per output.
What we can see is that OutputDebugStringW() is 1% slower than OutputDebugStringA() and that both calls are over 19 times slower than without the debugger.

Function String Time
OutputDebugStringA "" 0.03631s
OutputDebugStringA short string 0.03837s
OutputDebugStringA long string 0.3885s
Average 0.00009641s
 
OutputDebugStringW "" 0.3693s
OutputDebugStringW short string 0.3977s
OutputDebugStringW long string 0.4068s
Average 0.00009782s

Test 3 – Visual Studio 6

OutputDebugString() called 4000 times, Visual Studio 6 monitoring the process. One \r\n per output.
What we can see is that OutputDebugStringW() is 2% slower than OutputDebugStringA() and that both calls are over 22 times slower than without the debugger.

Function String Time
OutputDebugStringA "" 0.4048s
OutputDebugStringA short string 0.4247s
OutputDebugStringA long string 0.4267s
Average 0.00010468s
 
OutputDebugStringW "" 0.4127s
OutputDebugStringW short string 0.4346s
OutputDebugStringW long string 0.4419s
Average 0.00010743s

Test 4 – Visual Studio 2005

OutputDebugString() called 4000 times, Visual Studio 2005 monitoring the process. No \r\n per output.
What we can see is that OutputDebugStringW() is 54% slower than OutputDebugStringA() and that both calls are over 65 times (95 times slower for OutputDebugStringW) slower than without the debugger.

Function String Time
OutputDebugStringA "" 1.0270s
OutputDebugStringA short string 1.2200s
OutputDebugStringA long string 1.3982s
Average 0.00030377s
 
OutputDebugStringW "" 1.5850s
OutputDebugStringW short string 1.8874s
OutputDebugStringW long string 2.1672s
Average 0.00046997s

Test 5 – Visual Studio 2005

OutputDebugString() called 4000 times, Visual Studio 2005 monitoring the process. One \r\n per output.
What we can see is that OutputDebugStringW() is 48% slower than OutputDebugStringA() and that both calls are over 68 times (92 times slower for OutputDebugStringW) slower than without the debugger.

Function String Time
OutputDebugStringA "" 1.1133s
OutputDebugStringA short string 1.2766s
OutputDebugStringA long string 1.4455s
Average 0.00031962s
 
OutputDebugStringW "" 1.6444s
OutputDebugStringW short string 1.9108s
OutputDebugStringW long string 2.1501s
Average 0.00047543s

Test 6 – Visual Studio 2010

OutputDebugString() called 4000 times, Visual Studio 2010 monitoring the process. No \r\n per output.
What we can see is that OutputDebugStringW() is 2% slower than OutputDebugStringA() and that both calls are over 133 times (142 times slower for OutputDebugStringA) slower than without the debugger.

Function String Time
OutputDebugStringA "" 2.8112s
OutputDebugStringA short string 2.6041s
OutputDebugStringA long string 2.6408s
Average 0.00067134s
 
OutputDebugStringW "" 2.5735s
OutputDebugStringW short string 2.5891s
OutputDebugStringW long string 3.0845s
Average 0.00068727s

Test 7 – Visual Studio 2010

OutputDebugString() called 4000 times, Visual Studio 2010 monitoring the process. One \r\n per output.
What we can see is that OutputDebugStringW() is 2% slower than OutputDebugStringA() and that both calls are over 132 times (141 times slower for OutputDebugStringA) slower than without the debugger.

Function String Time
OutputDebugStringA "" 2.6517s
OutputDebugStringA short string 2.6604s
OutputDebugStringA long string 2.6423s
Average 0.00066287s
 
OutputDebugStringW "" 2.6675s
OutputDebugStringW short string 2.7529s
OutputDebugStringW long string 2.7410s
Average 0.00068011s

Conclusion

Calling OutputDebugString() when the application is not being monitored by a debugger does not incur significant overhead, although in tight loops this could be problematic.

Calling OutputDebugString() when the application is monitored by Visual Studio 6 can result in OutputDebugString() running between 19 and 22 times slower than without Visual Studio 6 monitoring the application.

Calling OutputDebugString() when the application is monitored by Visual Studio 2005 can result in OutputDebugString() running between 65 and 95 times slower than without Visual Studio 2005 monitoring the application.

Calling OutputDebugString() when the application is monitored by Visual Studio 2010 can result in OutputDebugString() running between 132 and 142 times slower than without Visual Studio 2005 monitoring the application.

The most surprising aspect is that the newer, more modern Visual Studio 2005 and Visual Studio 2010 are so much slower at handling OutputDebugString() than the old Visual Studio 6 IDE, which is now 12 years old. The performance difference is a factor of 3x or 4x (for VS2005) and a factor of 6x or 7x (for VS2010) depending on the test. Our initial tests ran OutputDebugString 100,000 times, but the tests using Visual Studio 2005 were so slow we had to reduce the test run to 4,000 times so that we could get a result in a realistic time frame to do the test.

The other interesting aspect is that with Visual Studio 2005 monitoring the application, the disparity in performance between OutputDebugStringA() and OutputDebugStringW() is even greater. With Visual Studio 2010, the disparity is not so great, but the actual performance level is worse than for Visual Studio 2005.

It may be tempting to write off all of the above information with the remark that this test does not reflect real world usage of OutputDebugString because we are calling it in a tight loop. It is the case that our loop is exceptionally tight as we are trying to time just the function call, but we see in our own development work occasions where OutputDebugString is called frequently enough that while in the debugger there is a serious drop off in performance. For example our interprocess comms for sending data from our monitoring stub to the GUI, if we are debugging so items we either enable some logging or OutputDebugString. This data is provided from the application being monitored at a rate and volume determined by what the application is doing. For Memory Validator, say, monitoring certain apps, that could be 500,000 events just to start the target application. That is 500,000 OutputDebugString calls in our GUI, reporting comms activity. In such a case, using logging may be more efficient. I mention this just to show that although our test has a tight loop, the actual test number (4,000) is well within real world usage.

Share

Speed up your MFC program without a profiler!

By , February 13, 2010 3:35 pm

I’m going to explain how to potentially improve the performance of MFC programs using CMap<> and related containers (CMapStringToPtr, etc) by a substantial amount.

We use MFC for quite a lot of our software. We also use STL and also some custom containers. It is easy to fall into certain programming habits of using a particular container for a particular task. Sometimes the result is not what you expected!

During the Christmas break I decided to write some software to analyse the Software Verification website log files. Initially I wanted to do keyword mining, but it soon blossomed into a stats package calculating different statistics for keywords, referrers, domains, URLs, favourite domains, bounce rates, evaluation download rates, etc.

I was also reading the wonderful "Web Analytics an Hour a Day" book by Avinash Kauschik. And to top it all, I was ill. I’ve since been told by various people that I had Swine Flu.

The initial version didn’t calculate much and was quite fast, but when I wanted to scale up to calculating monthly data as well as yearly, that was 13 times as much calculation and the inefficiencies in container choice started to show. I’d been lazy, this was a hobby project, nothing serious, so I’d chosen the MFC class CMap.

I wanted to map keyword strings to countData objects which would hold all the data on that keyword. Later we could discard the map and just use an array of keyword objects for random access and sorting.  I wanted data for unique keywords, so during processing of the data it seemed natural to want to map the keyword to the keyword data. I was doing similar things for referrers, referring domains, requested URLs, etc.

A CMap<> is an MFC hash table. A declaration would look like this:

CMap<CString, LPCTSTR, countData *, countData *&> mfcData;

The data processing would be something like this (simplified)

	BOOL	b;

	while(TRUE)
	{
		CString	s;
			
		b = file.ReadString(s);
		if (!b)
			break;

		// lookup string

		countData	*cd = NULL;

		if (mfcData.Lookup(s, cd))
		{
			// if we know about the string update the count

			cd->addCount(1);
		}
		else
		{
			// otherwise create the string and update the count

			cd = new countData(s, 1);
			mfcData.SetAt(s, cd);
		}
	}

The problem with the CMap<> class is that its hash key calculation isn’t very sophisticated, so you get collisions reasonably quickly, which forces the CMap to start making linked lists for the non-identical collisions, and walking the linked lists for subsequent searches is linear time. That gets slower as you add more data.

After processing for one particlularly large set of data for 2009 took 36 hours I thought a better solution was required. Why did I let it run so long? Partly because once it got past a few hours I was curious to see how long it would take. By this time I was ill, so it didn’t really matter, I was spending much of this time in bed alternately too hot or too cold 🙁

The STL has a non-standard hash table, so we didn’t want to use that, so we opted to use the STL <map> class. The declaration would look like this

map<CString, countData *> stlData;

The data processing would be something like this (simplified)

	BOOL	b;

	while(TRUE)
	{
		if (stopScan)
			break;

		CString	s;
			
		b = file.ReadString(s);
		if (!b)
			break;

		// lookup string

		MAP_STL::iterator	iter;

		iter = stlData.find(s);
		if (iter != stlData.end())
		{
			// if we know about the string update the count

			iter->second->addCount(1);
		}
		else
		{
			// otherwise create the string and update the count

			countData	*cd;

			cd = new countData(s, 1);
			stlData[s] = cd;
		}
	}

This is not a hash table, its a B-tree, so it will use more memory than the hash map, and unique entries should be slower than the hash table, but colliding entries should be faster. With this new style of map substituted all over the program, for all the many unique datatypes being calculated, the processing time dropped from 36 hours to 1 hour.

In other words, the STL version processed the data in 2.77% of the time the MFC version processed the data. That is a 97% improvement in processing time.

I am not saying that you will get this improvement. This improvement was acheived partly because of the type of statistics being calculated and partly due to the input data having so many collisions. The dataset for the above test was 17.1GB of Apache log files.

I have created an example MFC program that shows a trivial example of reading in lines from a file, testing them for uniqueness and storing them and updating the counts for them. The program also does the same using STL. Some sample times shown below:

Size (MB) MFC STL Improvement
34 00:00:12 00:00:06 50%
2048 00:22:46 00:05:21 76%

Time comparison of reading a 34MB file with MFC and STL
For a sample 34MB input file, the MFC program processes the data in 12+ seconds and the STL program processes the same data in 6+ seconds. An approximately 50% improvement.

Time comparison of reading a 2GB file with MFC and STL
For a sample 2GB input file, the MFC program processes the data in 22 minutes 46 seconds seconds and the STL program processes the same data in 5 minutes 21 seconds. An approximately 76% improvement.

The larger the dataset, or the more different CMap<> you are using calculating the data, the chances are that changing them to STL <map> would be highly effective in speeding up that part of your application. Note that if you are only loading a few tens of items into the map, it will not be worth making the change.

You can download the executable and the source code with project files so that you can test this yourself.

I don’t recommend changing all occurrences of CMap<> for STL <map>, but changing a the right ones should make some easy and worthwhile gains for you.

Share

Support for MinGW and QtCreator

By , December 4, 2009 5:01 pm

Everyone uses Visual Studio to write C and C++ software don’t they? Yes! you all chorus. Apart from some guys at the back who like to use gcc and g++. They use MinGW when working on Windows. And they may even use Emacs, or perish the thought, vi!

Up until now we haven’t been able to cater to the needs of gcc and g++ users. We’d get email every month asking when we were going to support MinGW or if we supported QtCreator. It was frustrating admitting we couldn’t support that environment. Even more so as the founders of Software Verification wrote large GIS applications using gcc and g++ back in the early 1990s.

During October we integrated support for MinGW and QtCreator into Coverage Validator, Memory Validator, Performance Validator and Thread Validator. Both COFF and STABS debug formats are supported, which provides some flexibility in how you choose to handle your symbols.

We’ll continue to add support for additional compilers to our tools as long as there is interest from you, the kind people that use our software tools.

Share

Robot footballers in China

By , August 8, 2008 5:04 pm

A while ago a German University Robot Football team asked us if we could help them with their software. We provided Performance Validator to them to see what they could do. Today we found out.

The Darmstadt Dribblers have just taken their robot Isra to RoboCup 2008, held in Suzhou, China. They came 4th in the championship (24 teams competing), losing to the championship winners (also from Germany). The other two quarter finalists were both from Japan. Thats a good result, against some good competition.

Isra running to the ball.

Isra running and kicking

Isra tackles an opponent.

Isra tackles

Isra scores a goal.

Isra scores a goal

This YouTube video shows Isra in action at Suzhou:
http://www.youtube.com/watch?v=QTbGXqQNkqQ&feature=PlayList&p=28F7B3DCA3198715&index=7

The Dramstadt Dribblers have uploaded various videos here:
http://www.youtube.com/profile_videos?user=DarmstadtDribblers

Still think software engineering can’t be fun?

We’ll be reporting on the Dribblers’ performance as they enter more competitions.

Share

Panorama Theme by Themocracy