Category: Memory

Command line support for .Net services and ASP.Net

By , September 29, 2011 3:59 pm

Today we have released updated versions of our software tools for .Net, .Net Coverage Validator, .Net Memory Validator and .Net Performance Validator.

The updates to each tool add support for monitoring .Net services and ASP.Net processes when working with the tool from the command line. This allows you to, for example, control the tool from batch files and easy create large suites of tests that you can run and control from batch files or using other scripting technologies. This command line support builds on the already existing command line support in each tool for working with .Net desktop appplications. For information on existing command line options for each tool please see the help file that ships with each tool.

I’m going to outline the new command line options for each tool and provide some basic examples of how you might use these options with each tool. Each tool has been given the same basic options, with each tool getting some additional options specific to that tool.

.Net Coverage Validator

  • -serviceName fileName

    The -serviceName option is used to specify which service .Net Coverage Validator should monitor. The filename argument should be quoted if the filename contains spaces.

    -serviceName c:\path\myservice.exe
    -serviceName "c:\path with spaces\myservice.exe"
    
  • -urlToVisit url

    The -urlToVisit option specifies the web page that should be opened by the web browser when working with ASP.Net web servers.

    -urlToVisit http://localhost/myTestPage.aspx
    -urlToVisit "http://localhost/myTestPage.aspx"
    
  • -aspNetName filename

    The -aspNetName option is used to specify the ASP.Net process that is used by IIS. The filename argument should be quoted if the filename contains spaces.

    -aspNetName c:\windows\Microsoft.Net\Framework\v4.0.30319\aspnet_wp.exe
    -aspNetName "c:\windows\Microsoft.Net\Framework\v4.0.30319\aspnet_wp.exe"
    

    The value specified should be the value that you would specify if you used the .Net Coverage Validator interface to work with ASP.Net applications.

  • -webRoot directoryname

    The -webRoot option is used to specify the web root for this ASP.Net process. The directoryname argument should be quoted if the filename contains spaces.

    -webRoot c:\inetpub\wwwroot
    -webRoot "c:\inetpub\wwwroot"
    
  • -webBrowser filename

    The -webBrowser option is used to specify which web browser to use to open the web page if the user has chosen to specify a particular web browser. This option is used when the -aspNetWeb option specifies to use a specific web browser. The filename argument should be quoted if the filename contains spaces.

    -webBrowser c:\mozilla\firefox.exe
    -webBrowser "c:\program files\internet explorer\iexplore.exe"
    
  • -coverageDirectory directoryname

    The -coverageDirectory option specifies the directory .Net Coverage Validator will use to communicate with the GUI if security privileges do not allow named pipes and shared memory usage. The directoryname argument should be quoted if the filename contains spaces.

    -coverageDirectory c:\temp
    
  • -aspNetWeb default|user|specific

    The -aspNetWeb option is used to specify which web browser will be used to open the web page you have specified.

    The options are:

    default Use the default web browser.
    user The user will open a web browser themselves.
    specific Use a web browser identified by a filepath. Use in conjunction with -webBrowser option.
    -aspNetWeb default
    -aspNetWeb user
    -aspNetWeb specfic
    
  • -aspNetDelay integer

    The -aspNetDelay option is used to specify how long .Net Coverage Validator will wait for IIS to reset and restart itself. The delay is specified in milliseconds.

    -aspNetDelay 5000
    

Working with .Net services

This example shows how to use .Net Coverage Validator with a .Net service.

dnCoverageValidator.exe -serviceName E:\WindowsService\bin\Debug\WindowsService.exe 
-coverageDirectory c:\test\coverage -saveSession "c:\test results\testbed.dncvm" 
-hideUI
  • -serviceName E:\WindowsService\bin\Debug\WindowsService.exe

    This specifies the service to monitor. The service must be started after .Net Coverage Validator has been instructed to monitor the service.

  • -coverageDirectory c:\test\coverage

    This specifies the directory .Net Coverage Validator will use to communicate with the GUI if security privileges do not allow named pipes and shared memory usage.

  • -saveSession “c:\test results\testbed.dncvm”

    This specifies that after the application finishes the session should be saved in the file c:\test results\testbed.dncvm.

  • -hideUI

    This specifies that the user interface should not be displayed during the test. When the target service closes .Net Coverage Validator will close.

Working with ASP.Net

This example shows how to use .Net Coverage Validator with ASP.Net.

dnCoverageValidator.exe -urlToVisit http://localhost/testWebApp.aspx 
-aspNetName c:\windows\Microsoft.Net\Framework\v4.0.30319\aspnet_wp.exe 
-aspNetWeb default -aspNetDelay 5000 -webRoot c:\inetput\wwwroot 
-coverageDirectory c:\test\coverage -saveSession "c:\test results\testbed.dncvm" 
-hideUI
  • -urlToVisit http://localhost/testWebApp.aspx

    This specifies the web page that will be opened when working with ASP.Net

  • -aspNetName c:\windows\Microsoft.Net\Framework\v4.0.30319\aspnet_wp.exe

    This specifies the ASP.Net worker process that IIS will start.

  • -aspNetWeb default

    This specifies that the system defined web browser will be used to open the web page.

  • -aspNetDelay 5000

    This specifies a delay of 5 seconds to allow the IIS webserver to restart.

  • -webRoot c:\inetput\wwwroot

    This specifies the web root of the IIS web server.

  • -coverageDirectory c:\test\coverage

    This specifies the directory .Net Coverage Validator will use to communicate with the GUI if security privileges do not allow named pipes and shared memory usage.

  • -saveSession “c:\test results\testbed.dncvm”

    This specifies that after the application finishes the session should be saved in the file c:\test results\testbed.dncvm.

  • -hideUI

    This specifies that the user interface should not be displayed during the test. When the target service closes .Net Coverage Validator will close.

.Net Memory Validator

  • -collectData

    The -collectData option causes .Net Memory Validator to collect memory allocation events until the user chooses to disable data collection from the user interface.

    -collectData
    
  • -doNotCollectData

    The -doNotCollectData option causes .Net Memory Validator to ignore memory allocation events until the user chooses to enable data collection from the user interface.

    -doNotCollectData
    
  • -serviceName fileName

    The -serviceName option is used to specify which service .Net Memory Validator should monitor. The filename argument should be quoted if the filename contains spaces.

    -serviceName c:\path\myservice.exe
    -serviceName "c:\path with spaces\myservice.exe"
    
  • -urlToVisit url

    The -urlToVisit option specifies the web page that should be opened by the web browser when working with ASP.Net web servers.

    -urlToVisit http://localhost/myTestPage.aspx
    -urlToVisit "http://localhost/myTestPage.aspx"
    
  • -aspNetName filename

    The -aspNetName option is used to specify the ASP.Net process that is used by IIS. The filename argument should be quoted if the filename contains spaces.

    -aspNetName c:\windows\Microsoft.Net\Framework\v4.0.30319\aspnet_wp.exe
    -aspNetName "c:\windows\Microsoft.Net\Framework\v4.0.30319\aspnet_wp.exe"
    

    The value specified should be the value that you would specify if you used the .Net Memory Validator interface to work with ASP.Net applications.

  • -webRoot directoryname

    The -webRoot option is used to specify the web root for this ASP.Net process. The directoryname argument should be quoted if the filename contains spaces.

    -webRoot c:\inetpub\wwwroot
    -webRoot "c:\inetpub\wwwroot"
    
  • -webBrowser filename

    The -webBrowser option is used to specify which web browser to use to open the web page if the user has chosen to specify a particular web browser. This option is used when the -aspNetWeb option specifies to use a specific web browser. The filename argument should be quoted if the filename contains spaces.

    -webBrowser c:\mozilla\firefox.exe
    -webBrowser "c:\program files\internet explorer\iexplore.exe"
    
  • -aspNetWeb default|user|specific

    The -aspNetWeb option is used to specify which web browser will be used to open the web page you have specified.

    The options are:

    default Use the default web browser.
    user The user will open a web browser themselves.
    specific Use a web browser identified by a filepath. Use in conjunction with -webBrowser option.
    -aspNetWeb default
    -aspNetWeb user
    -aspNetWeb specfic
    
  • -aspNetDelay integer

    The -aspNetDelay option is used to specify how long .Net Memory Validator will wait for IIS to reset and restart itself. The delay is specified in milliseconds.

    -aspNetDelay 5000
    

Working with .Net services

This example shows how to use .Net Memory Validator with a .Net service.

dnMemoryValidator.exe -serviceName E:\WindowsService\bin\Debug\WindowsService.exe 
-saveSession "c:\test results\testbed.dnmvm" -hideUI
  • -serviceName E:\WindowsService\bin\Debug\WindowsService.exe

    This specifies the service to monitor. The service must be started after .Net Memory Validator has been instructed to monitor the service.

  • -saveSession “c:\test results\testbed.dnmvm”

    This specifies that after the application finishes the session should be saved in the file c:\test results\testbed.dnmvm.

  • -hideUI

    This specifies that the user interface should not be displayed during the test. When the target service closes .Net Memory Validator will close.

Working with ASP.Net

This example shows how to use .Net Memory Validator with ASP.Net.

dnMemoryValidator.exe -urlToVisit http://localhost/testWebApp.aspx 
-aspNetName c:\windows\Microsoft.Net\Framework\v4.0.30319\aspnet_wp.exe 
-aspNetWeb default -aspNetDelay 5000 -webRoot c:\inetput\wwwroot 
-saveSession "c:\test results\testbed.dnmvm" -hideUI
  • -urlToVisit http://localhost/testWebApp.aspx

    This specifies the web page that will be opened when working with ASP.Net

  • -aspNetName c:\windows\Microsoft.Net\Framework\v4.0.30319\aspnet_wp.exe

    This specifies the ASP.Net worker process that IIS will start.

  • -aspNetWeb default

    This specifies that the system defined web browser will be used to open the web page.

  • -aspNetDelay 5000

    This specifies a delay of 5 seconds to allow the IIS webserver to restart.

  • -webRoot c:\inetput\wwwroot

    This specifies the web root of the IIS web server.

  • -saveSession “c:\test results\testbed.dnmvm”

    This specifies that after the application finishes the session should be saved in the file c:\test results\testbed.dnmvm.

  • -hideUI

    This specifies that the user interface should not be displayed during the test. When the target service closes .Net Memory Validator will close.

.Net Performance Validator

  • -collectData

    The -collectData option causes .Net Performance Validator to collect memory allocation events until the user chooses to disable data collection from the user interface.

    -collectData
    
  • -doNotCollectData

    The -doNotCollectData option causes .Net Performance Validator to ignore memory allocation events until the user chooses to enable data collection from the user interface.

    -doNotCollectData
    
  • -collectFunctionTimes

    The -collectFunctionTimes option causes .Net Performance Validator to collect timing information for functions in the application/service/ASP.Net webserver.

    -collectFunctionTimes
    
  • -collectLineTimes
    The -collectLinesTimes option causes .Net Performance Validator to collect timing information for lines in the application/service/ASP.Net webserver.

    -collectLineTimes
    
  • -doNotCollectFunctionTimes
    The -doNotCollectFunctionTimes option causes .Net Performance Validator not to collect timing information for functions in the application/service/ASP.Net webserver.

    -doNotCollectFunctionTimes
    
  • -doNotCollectLineTimes
    The -doNotCollectLinesTimes option causes .Net Performance Validator not to collect timing information for lines in the application/service/ASP.Net webserver.

    -doNotCollectLineTimes
    
  • -serviceName fileName

    The -serviceName option is used to specify which service .Net Performance Validator should monitor. The filename argument should be quoted if the filename contains spaces.

    -serviceName c:\path\myservice.exe
    -serviceName "c:\path with spaces\myservice.exe"
    
  • -urlToVisit url

    The -urlToVisit option specifies the web page that should be opened by the web browser when working with ASP.Net web servers.

    -urlToVisit http://localhost/myTestPage.aspx
    -urlToVisit "http://localhost/myTestPage.aspx"
    
  • -aspNetName filename

    The -aspNetName option is used to specify the ASP.Net process that is used by IIS. The filename argument should be quoted if the filename contains spaces.

    -aspNetName c:\windows\Microsoft.Net\Framework\v4.0.30319\aspnet_wp.exe
    -aspNetName "c:\windows\Microsoft.Net\Framework\v4.0.30319\aspnet_wp.exe"
    

    The value specified should be the value that you would specify if you used the .Net Performance Validator interface to work with ASP.Net applications.

  • -webRoot directoryname

    The -webRoot option is used to specify the web root for this ASP.Net process. The directoryname argument should be quoted if the filename contains spaces.

    -webRoot c:\inetpub\wwwroot
    -webRoot "c:\inetpub\wwwroot"
    
  • -webBrowser filename

    The -webBrowser option is used to specify which web browser to use to open the web page if the user has chosen to specify a particular web browser. This option is used when the -aspNetWeb option specifies to use a specific web browser. The filename argument should be quoted if the filename contains spaces.

    -webBrowser c:\mozilla\firefox.exe
    -webBrowser "c:\program files\internet explorer\iexplore.exe"
    
  • -profilerDirectory directoryname

    The -profilerDirectory option specifies the directory .Net Performance Validator will use to communicate with the GUI if security privileges do not allow named pipes and shared memory usage. The directoryname argument should be quoted if the filename contains spaces.

    -profilerDirectory c:\temp
    
  • -aspNetWeb default|user|specific

    The -aspNetWeb option is used to specify which web browser will be used to open the web page you have specified.

    The options are:

    default Use the default web browser.
    user The user will open a web browser themselves.
    specific Use a web browser identified by a filepath. Use in conjunction with -webBrowser option.
    -aspNetWeb default
    -aspNetWeb user
    -aspNetWeb specfic
    
  • -aspNetDelay integer

    The -aspNetDelay option is used to specify how long .Net Performance Validator will wait for IIS to reset and restart itself. The delay is specified in milliseconds.

    -aspNetDelay 5000
    

Working with .Net services

This example shows how to use .Net Performance Validator with a .Net service.

dnPerformanceValidator.exe -serviceName E:\WindowsService\bin\Debug\WindowsService.exe 
-profilerDirectory c:\test\profiler -saveSession "c:\test results\testbed.dnpvm" 
-hideUI
  • -serviceName E:\WindowsService\bin\Debug\WindowsService.exe

    This specifies the service to monitor. The service must be started after .Net Performance Validator has been instructed to monitor the service.

  • -profilerDirectory c:\test\profiler

    This specifies the directory .Net Performance Validator will use to communicate with the GUI if security privileges do not allow named pipes and shared memory usage.

  • -saveSession “c:\test results\testbed.dnpvm”

    This specifies that after the application finishes the session should be saved in the file c:\test results\testbed.dnpvm.

  • -hideUI

    This specifies that the user interface should not be displayed during the test. When the target service closes .Net Performance Validator will close.

Working with ASP.Net

This example shows how to use .Net Performance Validator with ASP.Net.

dnPerformanceValidator.exe -urlToVisit http://localhost/testWebApp.aspx 
-aspNetName c:\windows\Microsoft.Net\Framework\v4.0.30319\aspnet_wp.exe 
-aspNetWeb default -aspNetDelay 5000 -webRoot c:\inetput\wwwroot 
-profilerDirectory c:\test\profiler -saveSession "c:\test results\testbed.dnpvm" 
-hideUI
  • -urlToVisit http://localhost/testWebApp.aspx

    This specifies the web page that will be opened when working with ASP.Net

  • -aspNetName c:\windows\Microsoft.Net\Framework\v4.0.30319\aspnet_wp.exe

    This specifies the ASP.Net worker process that IIS will start.

  • -aspNetWeb default

    This specifies that the system defined web browser will be used to open the web page.

  • -aspNetDelay 5000

    This specifies a delay of 5 seconds to allow the IIS webserver to restart.

  • -webRoot c:\inetput\wwwroot

    This specifies the web root of the IIS web server.

  • -profilerDirectory c:\test\profiler

    This specifies the directory .Net Performance Validator will use to communicate with the GUI if security privileges do not allow named pipes and shared memory usage.

  • -saveSession “c:\test results\testbed.dnpvm”

    This specifies that after the application finishes the session should be saved in the file c:\test results\testbed.dnpvm.

  • -hideUI

    This specifies that the user interface should not be displayed during the test. When the target service closes .Net Performance Validator will close.

How to prevent a memory tool from monitoring your C/C++ allocations

By , July 10, 2010 10:53 am

A little known fact is that the Microsoft C Runtime (CRT) has a feature which allows some allocations (in the debug runtime) to be tagged with flags that causes these allocations to be ignored by the built in memory tracing routines. A good memory allocation tool will also use these flags to determine when to ignore memory allocations – thus not reporting any allocations that Microsoft think should remain hidden.

A customer problem

The inspiration for this article was a customer reporting that Memory Validator was not reporting any allocations in a particular DLL of his mixed mode .Net/native application. The application was interesting in that it was a combination of C#, C++ written with one version of Visual Studio and some other DLLs also written in C++ with another version of Visual Studio. Only the memory for one of the DLLs was not being reported by Memory Validator and the customer wanted to know why and could we please fix the problem?

After some investigation we found the problem was a not with Memory Validator but with the DLL in question making a call to _CrtSetDbgFlag(0); which turned off all memory tracking for that DLL. Memory Validator honours the memory tracking flags built into Visual Studio and thus did not report these memory allocations. Armed with this information the customer did some digging into their code base and found that someone had deliberately added this call into their code. Removing the call fixed the problem.

The rest of this article explains how Microsoft tags data to be ignored and what flags are used to control this process.

Why does Microsoft mark these allocation as ignore?

The reason for this is that these allocations are for internal housekeeping and sometimes also for one-off allocations that will exist until the end of the application lifetime. Such allocations could show up as memory leaks at the end of the application – that would be misleading as they were intended to persist. Better to mark them as “ignore” and not report them during a memory leak report.

Microsoft debug CRT header block

Microsoft’s debug CRT prefixes each allocation with a header block. That header block looks like this:

#define nNoMansLandSize 4

typedef struct _CrtMemBlockHeader
{
    struct _CrtMemBlockHeader * pBlockHeaderNext;
    struct _CrtMemBlockHeader * pBlockHeaderPrev;
    char *                      szFileName;
    int                         nLine;
#ifdef _WIN64
    /* These items are reversed on Win64 to eliminate gaps in the struct
     * and ensure that sizeof(struct)%16 == 0, so 16-byte alignment is
     * maintained in the debug heap.
     */
    int                         nBlockUse;
    size_t                      nDataSize;
#else  /* _WIN64 */
    size_t                      nDataSize;
    int                         nBlockUse;
#endif  /* _WIN64 */
    long                        lRequest;
    unsigned char               gap[nNoMansLandSize];
    /* followed by:
     *  unsigned char           data[nDataSize];
     *  unsigned char           anotherGap[nNoMansLandSize];
     */
} _CrtMemBlockHeader;

How does Microsoft tag an allocation as ignore?

When the CRT wishes an allocation to be ignored for memory tracking purposes, six values in the debug memory allocation header for each allocation are set to specific values.

Member Value #define
nLine 0xFEDCBABC IGNORE_LINE
nBlockUse 0x3 IGNORE_BLOCK
lRequest 0x0 IGNORE_REQ
szFileName NULL
pBlockHeaderNext NULL
pBlockHeaderPrev NULL

The Microsoft code goes out of its way to ensure no useful information can be gained from the header block for these ignored items.

When we first created MV we noticed that items marked as ignored should be ignored, otherwise you can end up with FALSE positive noise reported at the end of a memory debugging session due to the internal housekeeping of MFC/CRT.

How can you use this information in your application?

Microsoft also provides some flags which you can control which allows you to influence if any memory is reported as leaked. This is in addition to the CRT marking its own allocations as “ignore”. You can set these flags using the _CrtSetDbgFlag(int); function.

The following flags can be passed to _CrtSetDbgFlag() in any combination.

Flag Default Meaning
_CRTDBG_ALLOC_MEM_DF On On: Enable debug heap allocations and use of memory block type identifiers.
_CRTDBG_CHECK_ALWAYS_DF Off On: Call _CrtCheckMemory at every allocation and deallocation request. (Very slow!)
_CRTDBG_CHECK_CRT_DF Off On: Include _CRT_BLOCK types in leak detection and memory state difference operations.
_CRTDBG_DELAY_FREE_MEM_DF Off Keep freed memory blocks in the heap’s linked list, assign them the _FREE_BLOCK type, and fill them with the byte value 0xDD. CAUTION! Using this option will use lots of memory.
_CRTDBG_LEAK_CHECK_DF Off ON: Perform automatic leak checking at program exit via a call to _CrtDumpMemoryLeaks and generate an error report if the application failed to free all the memory it allocated.

How do I disable memory tracking for the CRT?

If you call _CrtSetDbgFlag(0); any memory allocated after that point will not be tracked.

With the above settings, all blocks are marked as ignore. You can see the code for this in the Microsoft C runtime.

The code that marks the block as “ignore” is at line 404 in dbgheap.c in the Microsoft C runtime (also used by MFC). When your code arrives here, nBLockUse == 1 and _crtDbgFlag == 0.

dbgheap.c line 404 (line number will vary with Visual Studio version)
                if (_BLOCK_TYPE(nBlockUse) != _CRT_BLOCK &&
                     !(_crtDbgFlag & _CRTDBG_ALLOC_MEM_DF))
                     fIgnore = TRUE;

This sets fIgnore to TRUE. From this point onwards the memory tracking code ignores the memory and sets the values mentioned above in the memory block header.

Default values

The default value for _crtDbgFlag is set elsewhere in the Microsoft code with this line:

extern "C"
int _crtDbgFlag = _CRTDBG_ALLOC_MEM_DF | _CRTDBG_CHECK_DEFAULT_DF;

How to replace IsBadReadPtr?

By , May 26, 2010 12:45 pm

The Microsoft Win32 API contains various functions that look useful at first sight but which have now become regarded as a pariahs. A good example of this is the IsBadReadPtr() function.

Under the hood this function uses structured exception handling to catch any exceptions that are thrown when the memory location read. If an exception is thrown the function returns TRUE, otherwise FALSE. So far so good.

So what could be wrong with that? A simplistic or naive interpretation would be “nothing”. But that ignores the fact that the callstacks of your application threads grow “on-demand”. This is done to avoid committing the full (default) 1MB upfront for each thread. This places a lower demand on application virtual memory and provides slightly faster startup for each thread. To allow each thread to grow on demand the stack has guard pages, which if you try to access them cause a guard page exception to be thrown which the OS handles gracefully, extends your stack space an appropriate amount and then returns execution to your application.

The problem with IsBadReadPtr() is that the exception handler inside IsBadReadPtr() eats the exception and thus the OS will not see it. So for the case where you end up using IsBadReadPtr() on a guard page you break the on-demand stack extension mechanism.

Raymond Chen of Microsoft has written a passionate post on this topic.

Raymond (and a few other folks) say that you should never use IsBadReadPtr(). I think thats a bit strong.

There are a few occasions where you may know what the datastructure is but you also know that it may have various memory protections on it. Such a case is when inspecting a DLL. Various parts are readonly. We have found during the last 10 years of writing tools like Memory Validator that it is not uncommon for a DLL loaded by LoadLibrary() to have memory protections on parts of the DLL that you don’t expect. We can’t control what DLLs our customer’s applications choose to load, so we have to handle all eventualities. We can’t just allow a crash to happen because we read a data location (in a customer DLL) that should be valid but isn’t.

Its also worth noting that the members of the team that wrote Boundschecker also came to the same conclusion and also tested certain DLL headers this way. You can find such code examples in the BugSlayer column in issues of Microsoft Systems Journal (MSJ) before it morphed into MSDN magazine.

One argument would be “Put an exception handler around it. Its an exceptional condition, handle it that way”.

The problem with that is sometimes that breaks the flow of the code and causes all manner of problems with the inability to mix C++ objects and SEH in the same function. Sometimes its much easier and simpler just to test for readability and abandon the function if you encounter one of these unusually constructed DLLs.

We are not advocating that you routinely use IsBadReadPtr() to hide the fact that you don’t know which objects to free, so you just call free on anything that passes IsBadReadPtr(). If you do that you will end up with exactly the problems that Raymond Chen describes.

But for the case where you do want IsBadReadPtr() functionality but you don’t want to use IsBadReadPtr(), what do you do? Here are drop in replacements for IsBadReadPtr() and IsBadWritePtr() that will not affect guard pages etc.

int isNotOKToReadMemory(void    *ptr,
                        DWORD   size)
{
	SIZE_T                          dw;
	MEMORY_BASIC_INFORMATION	mbi;
	int                             ok;

	dw = VirtualQuery(ptr, &mbi, sizeof(mbi));
	ok = ((mbi.Protect & PAGE_READONLY) ||
		  (mbi.Protect & PAGE_READWRITE) ||
		  (mbi.Protect & PAGE_WRITECOPY) ||
		  (mbi.Protect & PAGE_EXECUTE_READ) ||
		  (mbi.Protect & PAGE_EXECUTE_READWRITE) ||
		  (mbi.Protect & PAGE_EXECUTE_WRITECOPY));

	// check the page is not a guard page

	if (mbi.Protect & PAGE_GUARD)
		ok = FALSE;
	if (mbi.Protect & PAGE_NOACCESS)
		ok = FALSE;

	return !ok;
}

int isNotOKToWriteMemory(void   *ptr,
                         DWORD  size)
{
	SIZE_T                          dw;
	MEMORY_BASIC_INFORMATION	mbi;
	int                             ok;

	dw = VirtualQuery(ptr, &mbi, sizeof(mbi));
	ok = ((mbi.Protect & PAGE_READWRITE) ||
		  (mbi.Protect & PAGE_WRITECOPY) ||
		  (mbi.Protect & PAGE_EXECUTE_READWRITE) ||
		  (mbi.Protect & PAGE_EXECUTE_WRITECOPY));

	// check the page is not a guard page

	if (mbi.Protect & PAGE_GUARD)
		ok = FALSE;
	if (mbi.Protect & PAGE_NOACCESS)
		ok = FALSE;

	return !ok;
}

Remember: Use with caution, use sparingly and only if you need to. If you are routinely using IsBadReadPtr() or an equivalent to avoid keeping track of which data you should or should not use, you should think again about your software design.

Monitoring memory use in a JNI DLL called from Java

By , May 8, 2010 1:35 pm

Java is a garbage collected language that allows you to extend the language with code written in C and C++.

Given that the extensions are written in C or C++ the normal tools for monitoring Java memory usage will not report the C/C++ memory usage. So how do you monitor the memory usage of these JNI extensions when called from Java? This article is going to explain how to do this task.

Building an example JNI test

The first thing to do is to create an example to work with. The post Creating a JNI example for testing describes how to create a JNI example (both Java code and JNI C/C++ code, with downloadable source and project files). Please download the example code and build the example.

Monitoring JNI Memory

Monitoring memory in JNI extensions is straightforward. Java programs are executed by running a Java Virtual Machine (JVM). These are typically named java.exe, javaw.exe, jre.exe, jrew.exe. We can just launch java.exe with a C/C++ memory leak detection software tool and monitor the results. For this example we are going to monitor java with C++ Memory Validator. The image below shows the launch dialog (note if you are using Memory Validator for the first time you will be using the launch wizard, which is slightly different)

Memory Validator launch dialog showing launch of Java application

Items to note:

  • Application is set to the path to java.exe. C:\Program Files (x86)\Java\jdk1.5.0_07\bin\java.exe
  • Arguments is set to the name of the class to execute. Main
  • Startup directory is set to the directory containing the class to execute (and the native DLL to monitor). E:\om\c\memory32\testJavaJNI
  • If you wish to set the CLASSPATH you can set it in the Environment Variables part of the launch dialog. If you do not set the CLASSPATH here (as in this example) the CLASSPATH will be taken from the inherited environment variables of Memory Validator’s environment. For this example CLASSPATH is set in the global environment variables and thus does not need to be set on the launch dialog.

Click Go! to start the Java program. Java is started, Memory Validator monitors the application and records all memory allocations and deallocations. Any memory not deallocated by the end of the program is a leak.

JNI Leaks with no filtering

As you can see from the screenshot above, there is a quite a bit of memory left over after a simple run of this example Java program which does very little except load a native extension that prints Hello World! twice and deliberately leaks one 20 byte chunk of memory. The example image indicates there are 857 items, some of which are handles, the remainder C/C++ memory allocations. There are 3493 events. The memory leak we are interested in occurs at event 2985.

Clearly this is inefficient. To find memory leaks in your code you are going to have to wade through all the noise of the memory allocations made by Java and the JVM. There must be a better way!

There is. We’ll focus only on the native DLL that prints the Hello World! messages. Open the settings dialog and go to the Hooked DLLs section.

Setting up DLL filters to focus on the JNI DLL

  • Select the second radio box to indicate that only the DLLs we list will be monitored.
  • Now click Add Module… and select the HelloWorldImp.dll.
  • Click OK.

Memory Validator is now configured to monitor only HelloWorldImp.dll for memory and handle allocations.
Relauch the java application with these new settings.

JNI Leaks with DLL filtering

As you can see from the picture above much less data is collected. A total of 54 events compared to the previous session’s 3493 events. This is much more managable.

The list of items Memory Validator reports for this run contains only 11 events. This reduced amount makes it very easy to identify errors in the DLL.

  • 8 events are DLL loads (MV always reports DLL loads regardless of settings)
  • A one time, per-thread, buffer allocation inside printf
  • A memory leak that is deliberately present in the example JNI code
  • The final event is the status information for the application at the end of its run

If you don’t wish to see the DLL loads you can filter them out with a global, session or local filter.

Detail view of source code of memory leak in JNI code

The image above shows the source code of the location of the leaking memory in the JNI extension.

Conclusion

Monitoring memory allocations in JNI Dlls called from Java is a straightforward task.

Things to remember:

  • Ensure your JNI DLL is compiled with debugging information and linked with debugging information.
  • Ensure your JNI DLL debug information is present (put the PDB file in the same directory as the DLL).
  • Ensure your CLASSPATH is set correctly so that when Memory Validator starts your Java application the correct CLASSPATH is used.

What is the difference between a page and a paragraph?

By , March 8, 2010 11:40 am

In the real world we all know that pages contain paragraphs and that paragraphs are full of sentences created from words.

In the world of Microsoft Windows Operating Systems it is somewhat different – paragraphs contain pages!

In this article I’m to going to explain what a page is and what a paragraph is and how they relate to each other and why this information can be useful helping to identify and resolve certain memory related bugs.

Virtual Memory Pages
A virtual memory page is the smallest unit of memory that can be mapped by the CPU. In the case of 32 bit x86 processors such as the Intel Pentium and AMD Athlon, a page is 4Kb. When you make a call to VirtualProtect() or VirtualQuery() you will be setting or querying the memory protection for sizes that are multiples of a page.

The size of a page may vary from CPU type to CPU type. For example a 64 bit x86 CPU will have a page size of 8Kb.

You can determine the size of a page by calling GetSystemInfo() and reading the SYSTEM_INFO.dwPageSize value.

Virtual Memory Paragraphs
A virtual memory paragraph is the minimum amount of memory that can be commited/reserved using the VirtualAlloc() call. On 32 bit x86 CPUs this value is 64Kb (0x00010000). If you have ever used the debugger and looked at the load addresses of DLLs in the Modules list you may have noticed that DLLs always load on 64Kb boundaries. This is the reason – the area a DLL is loaded into is initialised by a call to VirtualAlloc to reserve the memory prior to the DLL being loaded.

Loaded Modules

You can determine the size of a paragraph by calling GetSystemInfo() and reading the SYSTEM_INFO.dwAllocationGranularity value.

Given these values, you can see that (on 32 bit 86 systems) a virtual memory paragraph is composed of 16 virtual memory pages.

How can I use this information?
If you are using VirtualAlloc() it is important to know the granularity at which the allocations will be returned. This is the size of a paragraph. This information is fundamental in deciding how you would implement a custom heap. You know there are fixed boundaries at which your data can exist. You can enumerate the list of possible paragraph locations very quickly (there are 32,768 possible locations in a 2GB space, as opposed to 2 billion locations if the paragraph could start anywhere).

Custom heaps
If you are writing a custom heap, a key indicator to keep track of is memory fragmentation and memory utilisation. Knowing your paragraph and page sizes you can inspect how each page and each paragraph of memory are used by the application and the custom heap to determine if there is wastage, what wastage there is and what form the wastage takes. This information could lead you to modify your heap algorithm to use pages differently to reduce fragmentation. See Delete memory 5 times faster for one simple technique, using HeapAlloc, the same principles apply here.

Loading large data files
Another use for this information is finding out why a certain large file will not load into memory despite Task Manager saying that you have 2GB of free memory. It is not uncommon to find a forum posting somewhere from someone that has a large image file (a satellite phote, MRI scan, etc) that is about 1GB in size. They wish to load it into memory, do in-memory processing on it, save the results, discard the memory then repeat the process, often for numerous images.
Typically on the third attempt to load a large file, the file will not load and the forum poster is left very confused.

The typical implementation is to allocate space for the large file using a call such as malloc() or operator new(). Both of which use the C runtime heap to allocate the memory.

The principle seems fine, but the problem is caused by memory fragmentation which results in a less accessible, totally usable, free space because the remaining free space blocks are separated into a many smaller regions, most of which are smaller than any forthcoming large allocation required by the application. Without the information about where pages and paragraphs are situated, how big they are and what their status is, identifying the cause of this failure could be very time consuming. Once you know the cause, you can think about allocating and managing your memory differently and prevent the bug from happening in the first place.

For situations like these, using HeapAlloc() with a dedicate heap (created using HeapCreate()) or even just directly using VirtualAlloc() will most likely lead to superior results than using the C runtime heap.

Tools
A first step in understanding such bugs is to be able to visualize the memory and to also inspect the various page and paragraph information.

To aid in these tasks we have just added a VM Pages view and VM Paragraphs view to VM Validator to make identifying such issues easier. VM Validator is a free download.

Memory Validator will also be updated with a VM Paragraphs view in the next release (Memory Validator already has a more detailed VM Pages view).

Thank you to Blake Miller of Invensys for suggesting an alternative wording for one paragraph of this article.

Delete memory 5 times faster

By , March 2, 2010 12:56 pm

Memory management in C and C++ is typically done using either the malloc/realloc/free C runtime functions or the C++ operators new and delete. Typically the C++ operators call down to the underlying malloc/free implementation to do the actual memory allocations.

This is great, its useful, it works, BUT it puts all the allocations in the same heap. So when you come around to deallocating the heap manager will need to take into account all the other objects and allocations that are also in the heap but unrelated to the data you are deallocating. That adds overhead to the heap manager, causing memory fragmentation and slower heap management.

There is a different way you can handle this situation – you can use your own heap for a given group of allocations.

	HANDLE	hHeap;

	hHeap = HeapCreate(0, 0x00010000, 0); // 64K growable heap

The downside to this is that you have to remember to use the correct allocators and deallocators for these objects and not to use malloc/free etc. You can mitigate this in C++ by overriding new/delete to use your own heap.

void *myClass::operator new(size_t size)
{
	return HeapAlloc(hHeap, 0, size);
}

void myClass::operator delete(void *ptr)
{
	HeapFree(hHeap, 0, ptr);
}

The upside to this technique is that you can delete all your deallocations in one call by deleting the heap and not bothering with deleting individual allocations. This is also about 5 times faster.

Old style:

	for(i = 0; i < count; i++)
	{
		HeapFree(hHeap, 0, ptrs[i]);
	}

New style:

	HeapDestroy(hHeap);
	hHeap = NULL;

There is another benefit to this technique: By deleting the heap and then re-creating a heap for new allocations you remove all fragmentation from the heap and start the new heap with 0% fragmentation.

HeapDestroy timing demonstration

Download the source of the demonstration application. Project and solution files for Visual Studio 6.0 and Visual Studio 2008 are provided.

We use this technique for some of our tools where we want a high performance heap and zero fragmentation.

You will also want to ensure that whatever software tool you are using to monitor memory allocations will mark all entries in a heap that is destroyed as deallocated.

Support for MinGW and QtCreator

By , December 4, 2009 5:01 pm

Everyone uses Visual Studio to write C and C++ software don’t they? Yes! you all chorus. Apart from some guys at the back who like to use gcc and g++. They use MinGW when working on Windows. And they may even use Emacs, or perish the thought, vi!

Up until now we haven’t been able to cater to the needs of gcc and g++ users. We’d get email every month asking when we were going to support MinGW or if we supported QtCreator. It was frustrating admitting we couldn’t support that environment. Even more so as the founders of Software Verification wrote large GIS applications using gcc and g++ back in the early 1990s.

During October we integrated support for MinGW and QtCreator into Coverage Validator, Memory Validator, Performance Validator and Thread Validator. Both COFF and STABS debug formats are supported, which provides some flexibility in how you choose to handle your symbols.

We’ll continue to add support for additional compilers to our tools as long as there is interest from you, the kind people that use our software tools.

Got a crash with a strange pointer value?

By , December 4, 2009 5:00 pm

Ever had a crash with a pointer value of 0xcdcdcdcd or 0xdddddddd or 0xfeeefeee?

You have? Sooner or later when you work with C or C++ you’ll bump into one of these crashes. You will no doubt try to reproduce the crash and thats when you’ll probably notice this unusual value has repeated itself. Perhaps the value is significant, perhaps it is telling you something?

I’ve has just written an article about the unusual bit patterns you can find in variables on the stack or in the various Win32 heaps or in the CRT heap. The article closes with some suggestions on how to prevent these errors from occurring.

A few minutes spent reading about the bit patterns may help you see these bit patterns less frequently and dig you out of any 0xfeeefeee hole you may have fallen into.

New .Net software tools

By , January 22, 2007 5:14 pm

We’ve spent the last few years creating our software tools for C++, Java and all the funky scripting languages that are now getting the recognition they deserve (Python, Ruby, JavaScript, etc).

During all this time we’ve have been asked if we have .Net versions of our tools, nearly always a question related to C#. We had to answer “No, but we will have .Net versions at some time in the future.”. That time has come. We now have .Net versions of Memory Validator and Performance Validator available as beta products.

Users of our popular Memory Validator software tool (for C++, Delphi…) will notice that the UI for .Net is quite different. This is because detecting memory leaks in garbage collected environments requires different approaches to collecting and analysing data. We have some innovative ideas in .Net Memory Validator, including the Allocations view, which provides a breakdown of objects allocated per function name; the Objects view, which provides a breakdown of objects allocated per object type; the Generations view, which provides an easy to read display of how many objects allocated per generation per object type. You can easily spot the trend graph of object usage and determine which objects are climbing or falling. A reference view allows you to view the object heap as a graph. The hotspots, memory, analysis, virtual and diagnostic tabs will be familiar to users of the original Memory Validator for C++.

.Net Memory Validator has the same user interface features you will find on our other Memory Validator products for other garbage collected languages (Java, JavaScript, Python, Ruby, etc) making ease of use when switching languages a doddle. As with all our products the .Net version, although different under the hood has the same team behind it. You are probably familiar with the case of a company creating a software tool for language X and then porting it support language Y, but the language Y version is lacking because it was a ground up rewrite, often by people unfamiliar with the language X version. This leads to incompatiblities, differnt UI behaviour, new bugs. That isn’t how we create our new language versions. Each version has its own code base, which allows for the radical under the hood changes to accomodate each language. But it has the same team and keeps the same user interface elements where applicable. So even if the code under the hood changes you get the same experience regardless of language.

This also allows our bug fixing to be improved. Many bugs from one version of a product apply to our other language versions. Thus if we find and fix a bug in say Java Memory Validator that bug fix can often be applied to JavaScript Memory Validator, Python Memory Validator, Ruby Memory Validator and .Net Memory Validator, etc. And our customers usually get the bug fix the very next day. This development method has been carried into the .Net line of software tools.

.Net Performance Validator continues this trend of the same user interface. If you know how to use any of our Performance Validator products (including the C++ version) you will know how to use .Net Performance Validator. Its that easy.

The callstack view provides a real time insight onto where a particular thread is running. Raw Statistics lets you inspect the raw data collected about performance and Statistics lets you inspect this data in a more orderly fashion. Relations provides the same information but allows you to view which function was called from which function. Call Tree provides the a call tree which you can expand and contract to view the performance data. Call Graph provides this information as a graph with each function listed as infrequently as possible. Call Graph is a very useful way to find an expensive function, then right click, choose goto Call Tree Node and the first node in the call tree that relates to the same node in the Call Graph expands with the node highlighted and source code displayed if available. Analysis allows complex queries onto the data and the diagnostic tab provides information about the instrumentation process.

Cheers for now, we have more .Net tools to work on.

Linux? MacOS X? Not right now, but some time in the future. Where have you read that before? 🙂

Ruby Memory Tracking API

By , December 1, 2006 5:15 pm

To provide the memory tracking, heap dump and object referencing facilities in Ruby Memory Validator we had to reverse engineer some hooks into the Ruby executable. This is not the easiest task in the world and beyond the abilities of anyone with less than a cursory knowledge of x86 assembly and Win32 hooking techniques.

To make it easier for this work to be done, to make it more reliable and to make these hooks available for those that just want to mainly work with Ruby and write a little C (as opposed to getting immersed in PE formats, x86 assembly and so on) we have created an extension to Ruby that provides for memory allocation and deallocation tracking, heap dumping and object reference determination. The extension is in the form of a simple C API.

You can read about this work, download the source code and download a working Ruby DLL with the extra code here.

http://www.softwareverify.com/ruby/customBuild/index.html

If you have any comments, please contact support at the usual address.

Panorama Theme by Themocracy