Detecting memory leaks in ISAPI extensions

By , April 21, 2020 11:51 am

Three weeks ago I wrote about how to setup IIS for use with ISAPI extensions.

Today I’m going to show you how easy it is to monitor memory and handle allocations in IIS, and use that information to detect any memory and handle leaks that may be present in your ISAPI extension.

IIS is a very secure Windows service. The account security for IIS mean that it can’t access most parts of the Windows filesystem or even Windows objects. CGI and ISAPI extensions will only execute if you’ve configured IIS properly (see the above linked article for details). You can’t launch IIS from a tool like Memory Validator, and you can’t inject into IIS from a tool like Memory Validator because of the security constraints. That leaves you the option of using an API to interact with IIS from your tool of choice.

But I don’t want to use an API!

“But I don’t want to use an API” I hear you say. You’re concerned about having to build multiple versions of your ISAPI. One for use with Memory Validator and one for production. That’s a valid concern, but it’s not the case. You can just build one version of your ISAPI that uses Memory Validator and use that. If you haven’t done “Monitor ISAPI…” from the Launch menu prior to loading your ISAPI the Memory Validator dlls won’t be loaded. There is no dependency on Software Verify DLLs, you can ship your DLL without any need to ship the Memory Validator DLLs.

But should you need to examine memory allocation behaviour you can just fire up Memory Validator and get to work without making a special build.

ISAPI API

ISAPI extensions provide 3 APIs for IIS to use:

BOOL WINAPI GetExtensionVersion(HSE_VERSION_INFO *pVer);

BOOL WINAPI TerminateExtension(DWORD	dwFlags);

DWORD WINAPI HttpExtensionProc(EXTENSION_CONTROL_BLOCK *pECB);

GetExtensionVersion() is called when the ISAPI is first loaded. We’ll use this to load Memory Validator into IIS.

TerminateExtension() is called when the ISAPI is unloaded. We’ll use this to tell Memory Validator that the work with IIS is done.

HttpExtensionProc() is used to process any requests made to the ISAPI extension.

Memory Validator API

To add the Memory Validator API to the ISAPI we do the following to the ISAPI source code:


  1. Add two header files:

    #include "svlMVStubService.h"
    #include "svlServiceError.h"
    

  2. We add the following code to GetExtensionVersion(). This includes logging success and failure so that we can identify if anything has gone wrong. Without the logging determining failure inside of IIS is very hard (a debugger and a special build of the Memory Validator DLL is required).

        // load Validator here
    
        svlMVStub_setLogFileName(L"C:\\testISAPIWebsite\\svl_MV_log.txt");
        svlMVStub_deleteLogFile();
    
        SVL_SERVICE_ERROR   errCode;
    #ifdef IS6432
        // x86 with x64 GUI
        errCode = svlMVStub_LoadMemoryValidator6432();
    #else   //#ifdef IS6432
        // x86 with x86 GUI
        // x64 with x64 GUI
        errCode = svlMVStub_LoadMemoryValidator();
    #endif   //#ifdef IS6432
        if (errCode != SVL_OK)
        {
            DWORD   lastError;
    
            lastError = GetLastError();
            svlMVStub_writeToLogFileW(L"C++ Memory Validator load failed. \r\n");
            svlMVStub_writeToLogFileLastError(lastError);
            svlMVStub_writeToLogFile(errCode);
    
            svlMVStub_dumpPathToLogFile();
        }
        else
        {
            svlMVStub_writeToLogFileW(L"C++ Memory Validator load success. \r\n");
    
            errCode = svlMVStub_StartMemoryValidatorForIIS();
    	if (errCode != SVL_OK)
    	{
                DWORD   lastError;
    
                lastError = GetLastError();
                svlMVStub_writeToLogFileW(L"Starting C++ Memory Validator failed. \r\n");
                svlMVStub_writeToLogFileLastError(lastError);
                svlMVStub_writeToLogFile(errCode);
    	}
    
            svlMVStub_writeToLogFileW(L"Finished starting C++ Memory Validator\r\n");
        }
    

    You’ll need to edit the location of the logfile to match the name of your website directory.

  3. We add the following code to TerminateExtension().

        // unload Validator here
    
        svlMVStub_UnloadMemoryValidator();
    


Monitoring memory allocations

Start Memory Validator.

From the Launch menu choose the IIS sub menu and then Monitor ISAPI….

The monitor ISAPI dialog is displayed. Any settings from a previous launch are displayed. Edit the settings appropriately.

You need to identify the ISAPI dll that is being processed, the website directory, the IIS process (choose Any IIS if you don’t know or don’t care), the web browser (the default is Microsoft Edge as it seems more tolerant of data format errors than Chrome) and finally the URL you wish to use to test the ISAPI.

Click OK.

Memory Validator copies some DLLs to the directory containing the ISAPI, sets up various variables that will be used inside the IIS process, resets IIS, and launches the web browser to load the specified URL that will load the ISAPI.

When you’ve finished interacting with the ISAPI, you’ll need to stop IIS. Go to the Launch menu, choose the IIS sub menu then Stop IIS.

When IIS has stopped executing Memory Validator displays “Ready” on the status bar, the Memory tab will show any memory leaks and handle leaks that are present.

A short video of this process is shown below.

Conclusion

We’ve completely reworked our ISAPI support so that you have very little to do, just use the API as shown above and launch using the Monitor ISAPI command.

In comparison, here are some instructions for using Boundschecker with IIS. These are out of date and will no longer work. But look how much you have to do manually.

Setting up ISAPI on IIS 10

By , April 3, 2020 11:27 am

Introduction

OK so it’s 2020 and how many people are developing ISAPI extensions? More than you might imagine. Yeah Ruby on Rails and Rust are all the rage these days, but some people still need to work with ISAPI for a bunch of business reasons. I recently had to setup IIS 10 for work with ISAPI on Windows 10. I read a lot of articles on how to do it. None of them were complete, resulting in reading several articles to get something working so I put this together, mainly for my own benefit (because I really don’t need to spend that much time doing this again!). I’m sharing it so you don’t have to go through this.

There’s an interesting gotcha if you’re developing a 32 bit ISAPI extension. Don’t worry I cover that at the end.

I was trying to get a simple ISAPI extension to work before trying anything else. My guess is most of you are working on legacy code, but a few of you may have been instructed to write a new ISAPI. Here’s a good starting point for a simple ISAPI extension if you haven’t already written one.

Creating an ISAPI extension: https://www.codeproject.com/Articles/1432/What-is-an-ISAPI-Extension

Installing IIS components

IIS components are installed via the Windows features dialog.

In the Windows 10 search box type “Turn Windows features on and off”, when windows shows you the result that matches press return (or click it).



The feature selection box is displayed. Select the items highlighted red in the image shown below. Click OK.



If you’ve already got partway through configuring IIS Manager and have realised you don’t have all the required components installed that’s OK, just install them and then close IIS Manager and reopen it (I found that if I didn’t do that not all the component parts would show in IIS Manager, making finding say ISAPI and CGI Restrictions impossible.

Configuring IIS Manager

Start Internet Information Services Manager.

Website

First of all we need a website to work with. If you’ve already got one skip the next few lines.

Add a test website. Right click on “Sites” in the left hand menu and choose “Add Website…”

Choose a website name. For example: “test”.

Choose a location for the website. For example: C:\testISAPIWebsite

Change the port number (just for testing) so that it doesn’t conflict with any other sites you have. For example: 81.

Handler Mappings

Select the server node on the left hand side and double click click on Handler Mappings on the right hand size.



The handler mappings are displayed.



Right click in empty space and choose “Edit Feature Permissions…”.

The Edit Feature Permissions dialog is displayed. Enable Read, Script and Execute persmissions. When you select the execute check box you’ll notice the entry for ISAPI dlls is added to the displayed Handler Mappings. Click OK.



ISAPI and CGI Restrictions

Select the server node on the left hand side and double click click on “ISAPI and CGI Restrictions” on the right hand size.

Right click in empty space and choose “Add…”.

Add the path to your ISAPI dll, a description and select the check box so that it is allowed to execute. Click OK.



This will place a web.config in the directory that contains the DLL. It will look something like this:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <handlers accessPolicy="Read, Execute, Script">
            <remove name="ISAPI-dll" />
            <add name="ISAPI-dll" path="*.dll" verb="*" modules="IsapiModule" scriptProcessor="C:\testISAPIWebsite\validate.dll" resourceType="File" requireAccess="Execute" allowPathInfo="true" preCondition="bitness32" />
        </handlers>
    </system.webServer>
</configuration>

32 bit ISAPI extensions

If your ISAPI is 32 bit you’ll need to enable them. Go to application pools (under the server node), select the application pool that your website is in, right click, choose “Advanced Settings…”. Change the “Enable 32-Bit Applications” setting to True.


64 bit ISAPI extensions

If your ISAPI is 64 bit you’ll need to ensure that you haven’t got 32 bit extensions enabled. Go to application pools (under the server node), select the application pool that your website is in, right click, choose “Advanced Settings…”. Change the “Enable 32-Bit Applications” setting to False.

Authentication problems

If when trying to view your web pages you get strange error messages, select the server node on the left then go to “Feature Delegation” and turn any entries that are “Read only” to “Read/Write”. Then restart the server (top of the right hand bar).

Note that I’m assuming you’re working on a Dev machine. If you’re working on a production machine you might want to be a bit less cavalier than just turning all settings to Read/Write – work through them one at a time to find out what you need and change only that.

Trying out the website

If we assume your ISAPI is called validate.dll you should be able to test your ISAPI in a browser using http://localhost:81/validate.dll?12345678

Panorama Theme by Themocracy