Rss Feed
Tweeter button
Facebook button
Technorati button
Reddit button
Myspace button
Linkedin button
Webonews button
Delicious button
Digg button
Flickr button
Stumbleupon button
Newsvine button

Posts tagged: api

Detecting memory leaks in Visual Test unit tests

By , June 6, 2018 5:40 pm

Introduction

We recently had a request asking if C++ Memory Validator could detect memory leaks in unit tests managed by Microsoft’s Visual Test and Visual Test Explorer. They told us what they’d tried to do and that it had failed. Our response was to find out what was failing, fix it and then describe what you need to do to use our tools with Visual Test. That’s what this article is about.

There were a few bugs specific to working with Visual Test, plus a very novel environment variable data corruption that can only happen in very unusual circumstances (you almost certainly would not hit these in normal usage of our tools). We fixed these tools, then experimented with several ways of working with Visual Test. We’re only going to talk about C++ Memory Validator here, but this article also applies to our Coverage, Performance and Thread tools.

You can run Visual Test from the command line, and also via Visual Test Explorer, which is a component of Visual Studio.

Monitoring Visual Test from the command line

The full details of how to work with Visual Test from the command line are documented in this article from Microsoft.

Using this information we know that we need to launch vstest.console.exe and pass the unit test DLL to that as an argument. For example:

vstest.console.exe unitTest.dll 

Using this information we can launch vstest.console.exe from C++ Memory Validator and pass it the appropriate DLL. You can set the startup directory to whatever you like. We’ve chose to set it to the same directory as the unit test DLL.

Unit Test Code

namespace UnitTest1
{		
	TEST_CLASS(UnitTest1)
	{
	public:
		
		TEST_METHOD(TestMethod1)
		{
			// pretend this is a real unit test, 
			// exercising a real target class/function
			// and that it leaks some memory

			char *ptr;

			ptr = new char[123];
			strcpy_s(ptr, 123, "Excellent Adventure");

			ptr = (char *)malloc(456);
			strcpy_s(ptr, 456, "Bogus Journey");

			ptr = (char *)malloc(789);
			strcpy_s(ptr, 789, "Face the Music");

			Assert::AreNotEqual("Bill", "Ted");
		}

	};
}

You’ll notice that you don’t need to specify the TEST_MODULE_INITIALIZE(moduleInit) function, or the TEST_MODULE_CLEANUP(moduleCleanup) function. You only need to specify the unit tests you want tested. C++ Memory Validator does the rest.

Monitoring Visual Test Explorer

When working with Visual Test Explorer, Visual Test is launched from Dev Studio, and then loads the unit tests for testing from a DLL. Visual Test then stays running in the background and does not shutdown. This means the “program has ended” signal that C++ Memory Validator needs doesn’t get sent. It also means that subsequent tests run with Visual Test Explorer won’t cause Visual Test to startup, meaning that the “program has started” signal that C++ Memory Validator needs doesn’t get sent.

To get around these problems we need to use the NT Service API (see help file, or online documentation for details) to contact C++ Memory Validator at the start of the unit tests, and also at the end of the unit tests. We do that using the TEST_MODULE_INITIALIZE(moduleInit) function, or the TEST_MODULE_CLEANUP(moduleCleanup) functions.

Functions

We use svlMVStub_LoadMemoryValidator() to load C++ Memory Validator into the unit test, then svlMVStub_StartMemoryValidator() to start it monitoring the unit test and communicating with the user interface. At the end of the tests we use svlMVStub_UnloadMemoryValidatorAndTerminateProcess(1000) to shutdown C++ Memory Validator and set a thread running that will after a delay of 1000ms terminate the vstest.executionengine.x86.exe (Visual Test) process. You may need to experiment with this delay on your machine.

Header Files

We need to include two header files from C++ Memory Validator. svlMVStubService.h and svlServiceError.h. You’ll find these in the svlMVStubService folder in the C++ Memory Validator install directory.

Libraries

We also need to link to svlMVStubService.lib (32 bit builds) or svlMVStubService_x64.lib (64 bit builds). You’ll find versions of these libraries in the svlMVStubService folder (one library per version of Visual Studio).

Unit Test Code

#include "svlMVStubService.h"
#include "svlServiceError.h"

namespace UnitTest1
{		
	TEST_MODULE_INITIALIZE(moduleInit)
	{
		SVL_SERVICE_ERROR	sse;

		sse = svlMVStub_LoadMemoryValidator();

		sse = svlMVStub_StartMemoryValidator();
	}

	TEST_MODULE_CLEANUP(moduleCleanup)
	{
		SVL_SERVICE_ERROR	sse;
		
		sse = svlMVStub_UnloadMemoryValidatorAndTerminateProcess(1000);
	}

	TEST_CLASS(UnitTest1)
	{
	public:
		
		TEST_METHOD(TestMethod1)
		{
			// pretend this is a real unit test, 
			// exercising a real target class/function
			// and that it leaks some memory

			char *ptr;

			ptr = new char[123];
			strcpy_s(ptr, 123, "Excellent Adventure");

			ptr = (char *)malloc(456);
			strcpy_s(ptr, 456, "Bogus Journey");

			ptr = (char *)malloc(789);
			strcpy_s(ptr, 789, "Face the Music");

			Assert::AreNotEqual("Bill", "Ted");
		}

	};
}

Having reworked the unit tests to support the NT Service API, we now need to launch devenv.exe from C++ Memory Validator, but with instructions to ignore devenv.exe and monitor Visual Test (vstest.executionengine.x86.exe). We do that from the launch dialog/wizard.

First choose devenv.exe to monitor using the Browse… button next to the Application to launch field. In this example we chose C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\devenv.exe


Next we need to setup the applications to monitor. Click the Edit… button next to the Application to monitor field. The applications to monitor dialog is displayed.


Choose devenv.exe using the Browse… button next to the Application to launch field.

Click the Add… button and add the vstest.executionenginex86.exe that corresponds with the devenv.exe you selected. In this example we chose C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstestexecutionengine.x86.exe. Be sure to choose the correct one – there are several with similar names to choose from.


Click OK to accept the application to monitor.


Click OK to accept the new definitions.

Now on the launch dialog, change the Application to monitor combo to select the vstest.execution.x86.exe entry.

Enter the name of your unit test dll(s) in the Arguments field, separated by spaces. If you specify the directory containing the DLLs as the startup directory, you can specify the DLL names without paths.

Running Your Tests

Now that we’ve setup C++ Memory Validator to monitor Visual Test when started from devenv, let’s get to work.

To start devenv, click the Go! button. Devenv will start. Load the solution that contains your unit tests. When you choose Run Selected Tests or Debug Selected Tests from Visual Test Explorer, the tests will run and will be monitored by C++ Memory Validator at the same time.

Perl Profiling API

By , August 6, 2011 3:48 pm

For some time we have had tools for use with Perl in the beta stage. The reason the tools never left beta is because the instrumentation part of the tools was tool brittle. It relied up scanning the Perl binary for suitable insertion points then modifying the binary to insert a suitable profiling callback. The callback would then ask the interpreter for filename, line number and function name information.

This worked fine so long as the version of Perl being used had data stored in the same format as the version we had inspected when we wrote the tools. As soon as the internal design of the Perl runtime changed our tools stopped working.

The Perl interpreter has some rudimentary profiling capabilities built in. The are only suitable for gathering timing statistics and cannot be used to provide code coverage flow tracing or call graph timing results. The only was to get statistics for these uses is to build your own profiling API. That is what we have done with the Perl Profiling API.

The Perl Profiling API is inspired by similar functionality in the Python interpreter, the Ruby interpreter and the Mozilla JavaScript interpreter. The API consists of four functions:

  • void Perl_set_do_profiling(int enable);
  • int Perl_get_do_profiling();
  • void Perl_set_coverage_callback(PERL_PROFILER_CALLBACK callback, void *userData);
  • void Perl_set_profiling_callback(PERL_PROFILER_CALLBACK callback, void *userData);

PERL_PROFILER_CALLBACK

PERL_PROFILER_CALLBACK is defined as

void (*PERL_PROFILER_CALLBACK)(PERL_PROFILER_EVENT event,
                               const char          *fileName,
                               const int            lineNumber,
                               const char          *functionName,
                               const char          *packageName,
                               void                *userData);

The PERL_PROFILER_EVENT event can receive values of PPE_LINE, PPE_CALL, PPE_RETURN. PPE_LINE is used for line visits, PPE_CALL is used when a function is called, PPE_RETURN is used when a function returns. The fileName and lineNumber specify which part of which fileName the code is executing, the functionName specifies the current function and the packageName specifies which package/namespace the function is part of (if any). The userData value is the value passed when setting the callback using Perl_set_coverage_callback() and/or Perl_set_profiling_callback.

void Perl_set_do_profiling(int enable);

Perl_set_do_profiling() is used to enable or disable profiling with either of the callbacks. Pass TRUE to enable and FALSE to disable.

int Perl_get_do_profiling();

Perl_get_do_profiling() is used to determine if profiling is enabled or not. The function returns TRUE for enabled and FALSE for disabled.

void Perl_set_coverage_callback(PERL_PROFILER_CALLBACK callback, void *userData);

Perl_set_coverage_callback() is used to set the callback that will be called for line visit events. The callback function will be called for all lines the Perl program visits. The userData value specified will be passed to the callback.

void Perl_set_profiling_callback(PERL_PROFILER_CALLBACK callback, void *userData);

Perl_set_profiling_callback() is used to set the callback that will be called for function call and function return events. The callback function will be called for all functions the Perl program visits. The userData value specified will be passed to the callback.

Using the Perl Profiling API

By default the Perl Profiling API is not enabled and no callback values are set.

To use the Perl Profiling API we need to define a callback to do the work.

void callback(PERL_PROFILER_EVENT event,
              const char          *fileName,
              const int            lineNumber,
              const char          *functionName,
              const char          *packageName,
              void                *userData)
{
    switch(event)
    {
    case PPE_LINE:
        storeLineVisitInfo(fileName, lineNumber, userData);
        break;

    case PPE_CALL:
        startFunctionCallTiming(fileName, lineNumber, functionName, packageName, userData);
        break;

    case PPE_RETURN:
        endFunctionCallTiming(fileName, lineNumber, functionName, packageName, userData);
        break;
    }
}

The implementation of storeLineVisitInfo(), startFunctionCallTiming() and endFunctionCallTiming() are down to the profiler writer. You can use these to implement a timing profiler, code coverage, flow tracing or some other tool.

We also need to tell the Perl to use the Perl Profiling API.

	Perl_set_coverage_callback(callback, userData);
	Perl_set_profiling_callback(callback, userData);
	Perl_set_dp_profiling(TRUE);

Conclusion

As you can see using the Perl Profiling API is easy.

Adding the Perl Profiling API to the version of Perl you are using is also straightforward – there are two files that implement the API and two simple modifications to the runtime loop in dump.c and run.c. The overhead of the API is trivial when not in use and when in use the overhead is defined by what the callback writer chooses to do.

Learn more about the sources and binaries for the Perl Profiling API.

Ruby Memory Tracking API

By , December 1, 2006 5:15 pm

To provide the memory tracking, heap dump and object referencing facilities in Ruby Memory Validator we had to reverse engineer some hooks into the Ruby executable. This is not the easiest task in the world and beyond the abilities of anyone with less than a cursory knowledge of x86 assembly and Win32 hooking techniques.

To make it easier for this work to be done, to make it more reliable and to make these hooks available for those that just want to mainly work with Ruby and write a little C (as opposed to getting immersed in PE formats, x86 assembly and so on) we have created an extension to Ruby that provides for memory allocation and deallocation tracking, heap dumping and object reference determination. The extension is in the form of a simple C API.

You can read about this work, download the source code and download a working Ruby DLL with the extra code here.

http://www.softwareverify.com/ruby/customBuild/index.html

If you have any comments, please contact support at the usual address.

Panorama Theme by Themocracy