Rss Feed
Tweeter button
Facebook button
Technorati button
Reddit button
Myspace button
Linkedin button
Webonews button
Delicious button
Digg button
Flickr button
Stumbleupon button
Newsvine button

Category: Development

Changes to the first run wizard

By , February 15, 2018 2:41 pm

When you run one of our tools for the first time it displays the first run wizard.

The first run wizard takes you through the steps of configuring a few settings that are important to the tool your a using. Most of these settings are the same for each tool, although some of the first run wizards have settings pages that are specific to that tool.

We’ve just add a new page to the first run wizard. We’ve added the ability to configure the behaviour of the launch, inject and wait dialogs. The default behaviour is wizard. The behaviour can be modified from the Settings menu (formerly, Configure menu). Now this setting can also be set on the first run wizard.

The reason for adding this option to the first run wizard is because when we’ve done interactive debugging with customers we often notice that the user still has their Validator configured to use the launch wizard, rather than the more concise launch dialog. We thought there was a chance that some people don’t realise they can change the style of these dialogs/wizards.


Share

New tools and old tools

By , February 6, 2018 11:16 am

Last week we released a versions of our tools that no longer install DLLs in the Windows system32 directory. We documented these changes in an article Changing How We Install Our Software.

In hindsight, I think we rushed that release. Which is a strange thing to write given that the work that went into it was nearly a month in duration and the last two weeks of testing every use case for the update NT Service API and the new DLL locations, that was very detailed and deliberate.

But where we let ourselves down was how we communicated about how the old versions of the tools and the new versions of the tools can live side by side. This blog post is about clearing up any misunderstanding. We’ve also improved the user interface for informing you about the old DLLs and choosing what action to take if old DLLs are found.

How do old tools and new tools co-exist?

Old Tools. The old tools always copy the DLLs they need to System32 every time they start. Because of this, even if a new tool removes the DLLs, the old tools will reinstall the DLLs automatically.

New tools. The new tools don’t expect to find DLLs in System32 and thus may not work correctly if DLLs are found there (because of how Windows search paths work). To deal with this we’ve provided a helper utility system32 cleanup which you can use to automatically delete any DLLs from old tools.

If you set the cleanup utility to automatically delete any dlls from old tools, then the new tools will always cleanup any dlls from the old tools, while the old tools will always replace them. This system will work unless you have an anti-virus tool that starts deleting anything copied into System32 – if that is happening you need to start using the new versions of our tools.

System32 Cleanup

The system32 cleanup user interface is only displayed if there are DLLs from old versions of the tools in the Windows system32 directory.

Please note that on 64 bit machines the 32 bit versions of our tools will be looking in SysWow64 (although the path will be reported as system32 due to the Windows file redirection mechanism).

We’ve replaced the message box with a dedicated dialog that allows you to choose how you want the old DLLs treated.


The dialog provides a list of DLLs that are from old versions of our software tools.

Below the list is a combo box allowing you to choose what action is taken when any subsequent check for DLLs is run (regardless of which Validator tool starts the check for old DLLs). The options are:

  • Always ask to delete files.
  • Always delete files.
  • Never delete files.

Each option results in different behaviour.

Always Ask To Delete Files

If Software Verify DLLs are found in system32 the dialog is displayed. The user of the software gets to choose if the DLLs are deleted or not. The DLLs can be deleted by clicking the Delete Files button. The future behaviour of the dialog can be changed by choosing one of the options in the combo box. If Always delete files is chosen the DLLs will be deleted when the dialog is closed.

Always Delete Files

If Software Verify DLLs are found in system32 the DLLs are deleted without informing the user. The dialog is never displayed.

Never Delete Files

No DLLs are deleted. The dialog is never displayed.

/reset

Because only one of the above behaviours results in the dialog being displayed it is possible to set the dialog up so that you can never interact with it again. But what if you wanted to change the behaviour once later on? To do that, run the svlSystem32Cleanup.exe (or svlSystem32Cleanup_x64.exe as appropriate) with the /reset command line argument. This will reset the behaviour to the default state and cause the user interface to display even if there are no DLLs to delete.

Share

Changing how we install our software

By , February 3, 2018 10:46 am

With the next release of our software (later today) we are changing how we install our software.

The installers stay the same, the software (as you see it) stays the same, but under the hood, things are different. This article describes that change, why we made that change, the consequences of that change, and the work we did to ensure the software keeps functioning.

First, some history.

System32

Once upon a time, software developers routinely placed their DLLs in the Windows system directory, and then when Windows 95 came along, they placed their DLLs in the Windows system32 directory. This eventually resulted in a mess, as far too many DLLs from different software vendors ended up stored in one place.

To solve this Microsoft stated that an application’s DLLs should be stored with the application. Microsoft also made installing their redistributables easier and made various other changes that made “DLL Hell” go away (or manifest in subtly different forms). Over time most software vendors changed how they distributed their software.

Debugging Tools

When we started Software Verify Limited (originally as Software Verification Limited, which no-one could spell!), we noticed that for certain types of problem we needed to investigate, it was quite hard to get the appropriate DLLs to load into the target application unless the DLLs were stored in the system32 directory.

We took the view, rightly or wrongly, that because we were providing tools that do unusual things with applications we could also do non-standard things with how we installed the software. We also added code to the user interface so that each time it started it would copy it’s DLLs to where it thought they needed to be (usually system32, sometimes SysWow64 – the 32 bit system32 on a 64 bit machine).

This behaviour was so ingrained and so core to how the software worked, that even when we spent 3 months rewriting the software many years ago to work nicely in the administrator mode enabled world of Windows Vista/7/8/10, it never occurred to us to change our abuse of the system32 directory.

Anti-Virus

Then one day, a developer researching our tools emailed us saying that he liked our tools but he was constantly fighting his anti-virus tool (BitDefender) that was flagging our tools as viruses because we copied files into the Windows system32 directory. He had disabled this feature but really wanted our software modifying so that we didn’t trigger the Anti-Virus software.

We’d also had reports from other developers that other anti-virus tools had flagged our tools as viruses – which we had assumed was a reference to the fact that our software uses CreateRemoteThread(). The function CreateRemoteThread() can be used for good and for ill. The Toolhelp and PSAPI libraries use this function to query data from running executables. So does Process Explorer from SysInternals (now part of Microsoft). We also use it. So do Bad Guys ™.

These various reports indicated to us that we had a problem, so we started investigating what needed to change to get the many moving parts of our software to work correctly without us touching Windows system32 or Windows sysWow64.

The change to what we install

Things we don’t do any more:

  • We no longer install any Software Verify DLLs or EXEs to system32 or sysWow64.
  • We no longer copy any Software Verify DLLs or EXEs to system32 or sysWow64 when the user interface runs.

So how do the DLLs get found?

One solution to this would be to place the path to the installed tool in the PATH environment variable. We considered this for a short while. But we have a lot of tools and the paths to them will be long if stored in the default location of c:\program files etc. We didn’t like the idea of someone installing C++ Developer Suite x64 / x86 and suddenly being faced with 8 new, very long paths added to their PATH environment variable. We had to come up with a better way.

We decided that we’d update the PATH environment variable to point to the tool in use, at the time the tool being used launched an application, launched a batch file, attached to a running application, or started working with a service. This way, only the application being monitored has it’s PATH updated, and it only has it’s PATH updated with the path to the tool in use, not all tools that have been installed. It’s elegant. It results in path isolation, so incompatible DLLs for one tool can’t affect another tool. And the user’s PATH environment variable doesn’t get polluted with lots of PATHs to our tools.

OK, so that sounds easy to do. Is it? Not exactly. There are quite a few use cases we need to cover. Some are trivial to implement, other quite involved. For all of the cases listed below we need to be able to handle:

  • 32 bit tool working with 32 bit application.
  • 64 bit tool working with 32 bit application.
  • 64 bit tool working with 64 bit application.

We made a grid of everything we needed to test, in every combination of tool (32 bit / 64 bit) and launch/batch/inject/service (32 bit and 64 bit). It came to 140 separate tests cases, each with it’s own test peculiarities, often requiring new and inventive ways to address the problem. When working through the services section we had to reboot a few times as we managed to get the service manager into a rather confused state :-).

Why does the PATH matter so much?

The reason the PATH matters so much is because when you’re injecting a DLL into another process so that you can gather information (flow tracing, code coverage, memory allocations, profiling, thread analysis) the DLL you’re injecting almost certainly has dependencies on other DLLs. If those dependencies can’t be satisfied by being found on the DLL loader search path, the DLL won’t load.

The reason we used to store the DLLs in system32 (or sysWow64) is because it’s an easy (and lazy) way to get the DLL dependencies found.

However, if we can modify the PATH suitably, we can ensure the DLLs can still be found.

Launching native (unmanaged) applications

This is when we start a typical application built using C, C++, Delphi, Visual Basic etc.

We modify the PATH in the tool prior to launching the target application. The target application inherits the environment variables of the parent process (the tool launching it).

Launching .Net (managed) applications

This is when we start a typical application built using C#, Visual Basic .Net etc.

We modify the PATH in the tool prior to launching the target application. The target application inherits the environment variables of the parent process (the tool launching it).

Launching applications from batch files

This is when we start an application from a batch file.

We modify the PATH in the tool prior to launching the target application. The target application inherits the environment variables of the parent process (the tool launching it).

This method will work for most use cases. If only life were that simple…

Launching applications from batch files with no path

This is when we start an application from a batch file but the creator of the batch file has explicitly modified the PATH environment variable, nullifying any changes we made to the PATH environment variable when we launched the command processor that is executing the batch file.

We need to intercept where the command processor starts the process and then modify the PATH in the environment block passed to the target application.

Launching applications from other applications

This is where we monitor the Nth application launched by another application.

We need to intercept where the first application starts the process we wish to monitor and then modify the PATH in the environment block passed to the target application.

Attaching to running applications

This is where we need to inject our DLL (which is in a known location that is not in the PATH) into the address space of a running process that already has the PATH environment variable set, most likely without including a path to where our DLL (and it’s dependencies) is located. The key is the fact that the injected DLL has dependencies. If the dependencies can’t be found, the DLL load will fail.

We had to invent a way to modify the PATH in the already running application before we inject our DLL with all it’s dependencies.

Working with services

This is where we need to get our DLL loaded by the tool’s NT Service API (each Validator has it’s own NT Service API).

This isn’t as straightforward as it seems. The path to the DLL isn’t known (the user could have installed the Validator anywhere). We have to find a way to get the PATH information into the service so that the various NT Service APIs can load the DLL from the correct location. To do this we also updated all the NT Service APIs with additional functions to help debug any NT Service API failures. A separate blog post covers this topic. Services run in quite locked down environments, getting this information into each service was quite an interesting adventure.

Changes to installer and Validators

The way our software tools are written you can normally install one version over the top of a previous version without needing to uninstall the previous version. For this next release, with the new DLL layout, we recommend uninstalling the previous version. However this is not essential as we have added some code to the licence key installer and also to the main Validator user interface, code that will check for the existence of Software Verify DLLs in the Windows system32 directory. If Software Verify DLLs are found in this directory, you will be informed and offered the ability to remove these DLLs.

We recommend removing the DLLs, as we cannot predict the interaction between DLLs left in the system32 directory and the new version of the Validator software. It is possible that for most, if not all, use cases that the software will function correctly. This depends on the way the Windows DLL Loader searches for DLLs and how it loads DLLs. If the system32 directory DLLs are found first then the wrong DLLs will be loaded. This is why we recommend removing them.

“But if I delete the svl*.* files in system32, won’t my older tools stop working?”

Older versions of our software tools will continue to work. Every time you start the GUI for an older tool, it will copy the DLLs it needs back into system32 so that it can function. Then, the next time you start a modern tool, you’ll get a prompt about the svl*.* files and have the option to delete them. With both solutions, the tools keep working. However, if you have an anti-virus tool (for example, BitDefender) installed, that will then delete those files, preventing the tool from working. That’s why the new mechanism is better, the tools work and anti-virus tools are happy.

The above scenario isn’t ideal, but it will allow older and newer tools to work together. If this is a serious issue for some people we may add an “always delete the files” option so that subsequent starts of the newer tools automatically delete the svl*.* files from system32. If you need this functionality, please let us know.

We believe the current versions of our tools are better than the older versions and that upgrading to the current versions will provide a better experience, both in functionality, better user experience, and better documentation. This is the 2nd time in our company history (nearly 16 years) that we’ve chosen to make a non-backwards compatible change with previous behaviour (excluding changes to saved data file formats).

Summary

We’ve changed how we install our software. We no longer do things that might alarm anti-virus tools. As a user of our software, you shouldn’t notice any change in functionality. Everything should continue working as normal.

Share

Changes to the NT Service API

By , February 3, 2018 10:25 am

All of our Validator tools have an NT Service API.

We’ve just made some changes to the NT Service API. This article documents those changes and why we made them.

The changes to the NT Service API, break the existing API behaviour, and improve the existing API behaviour.

If you are using the NT Service API, you should read this article.

For the purposes of this article I’m going to describe the NT Service API for C++ Memory Validator. The changes apply to the NT Service APIs for each C++ Validator tool, the only thing that is different is the function name prefix of svlBVStub_, svlCVStub_, svlMVStub_, svlPVStub_, svlTVStub_. If you can’t directly use the NT Service API due to compiler/linker issues, please consult the documentation for your Validator to see an alternative way of getting the same result.

Changes to the API

  • We changed the name of an enumeration and extended the values in the enumeration.
  • We split some functions into two parts so that they could be reused with other functions to achieve a better result.
  • We added some debugging functions to help you debug any failures when using the API.

SVL_SERVICE_ERROR

The SVL_ERROR enumeration has been renamed to SVL_SERVICE_ERROR.

The definition for SVL_SERVICE_ERROR is now in it’s own include file svlServiceError.h rather than defined in the NT Service API header file (svlMVStubService.h for C++ Memory Validator).

typedef enum _svlServiceError
{
   SVL_OK,                              // Normal behaviour
   SVL_ALREADY_LOADED,                  // Stub DLL already loaded into service
   SVL_LOAD_FAILED,                     // Failed to load stub DLL into service
   SVL_FAILED_TO_ENABLE_STUB_SYMBOLS,   // Loaded DLL, but failed to enable stub symbols because couldn't find function
   SVL_NOT_LOADED,                      // Couldn't unload DLL because DLL not loaded
   SVL_FAIL_UNLOAD,                     // Couldn't unload DLL because couldn't find function
   SVL_FAIL_TO_CLEANUP_INTERNAL_HEAP,   // Couldn't get the internal stub heap and thus couldn't clean it up
   SVL_FAIL_MODULE_HANDLE               // Couldn't get the stub DLL handle so couldn't continue
   SVL_FAIL_SETSERVICECALLBACK,         // Couldn't call the set service callback
   SVL_FAIL_COULD_NOT_FIND_ENTRY_POINT, // Couldn't find the DLL entry point to start the validator
   SVL_FAIL_TO_START                    // Failed to start the Validator
} SVL_SERVICE_ERROR;

Split functionality

The previous implementation of svlMVStub_LoadMemoryValidator() and svlMVStub_LoadMemoryValidator6432() would load the Validator DLL, then start the Validator profiling the service.

The new implementation of svlMVStub_LoadMemoryValidator() and svlMVStub_LoadMemoryValidator6432() just loads the Validator DLL. It does not start the Validator profiling the service.

The reason for this change is that we wanted to allow the ability to set a service manager callback so that the service control manager could be informed about a long running instrumentation phase as the Validator is started. This is important because a delay of more than 10 seconds will cause the service control manager to kill the service.

To set the service callback call the svlMVStub_SetServiceCallback() function.

Then start the Validator with svlMVStub_StartMemoryValidator().

Loading, Starting, Unloading the Validator

To control the loading, starting and unloading of the Validator into the service, use these functions.

extern "C" SVL_SERVICE_ERROR svlMVStub_LoadMemoryValidator();

extern "C" SVL_SERVICE_ERROR svlMVStub_LoadMemoryValidator6432();

extern "C" SVL_SERVICE_ERROR svlMVStub_StartMemoryValidator();

extern "C" SVL_SERVICE_ERROR svlMVStub_UnloadMemoryValidator();

extern "C" SVL_SERVICE_ERROR svlMVStub_SetServiceCallback(serviceCallback_FUNC callback,
                                                          void*                userParam);

svlMVStub_LoadMemoryValidator

extern "C" SVL_SERVICE_ERROR svlMVStub_LoadMemoryValidator();

To load the Validator DLL into a 32 bit service when you are working with a 32 bit Validator, or to load the Validator DLL into a 64 bit service when you are working with a 64 bit Validator, use svlMVStub_LoadMemoryValidator().

svlMVStub_LoadMemoryValidator6432

extern "C" SVL_SERVICE_ERROR svlMVStub_LoadMemoryValidator6432();

To load the Validator DLL into a 32 bit service when you are working with a 64 bit Validator, use svlMVStub_LoadMemoryValidator6432().

svlMVStub_SetServiceCallback

extern "C" SVL_SERVICE_ERROR svlMVStub_SetServiceCallback(serviceCallback_FUNC callback,
                                                          void*                userParam);

Once you have successfully loaded the Validator DLL you can setup a service callback so that the service control manager can be kept updated during the process of starting the Validator.

When a service is starting, Windows requires the service to inform the Service Control Manager (SCM) that is starting at least every ten seconds. Failure to do so results in Windows concluding that the service has failed to start, and the service is terminated. Instrumenting your service may take more than 10 seconds, depending on the complexity and size of your service.

The solution is for the Validator to periodically call a user supplied callback from which you can regularly inform the SCM of the appropriate status.

Here is an example callback which ignores the userParam.

   void serviceCallback(void   *userParam)
   {
       static DWORD dwCheckPoint = 1;
   
       ssStatus.dwServiceType = SERVICE_WIN32_OWN_PROCESS;
       ssStatus.dwServiceSpecificExitCode = 0;

       ssStatus.dwControlsAccepted = 0;
      
       ssStatus.dwCurrentState = dwCurrentState;
       ssStatus.dwWin32ExitCode = dwWin32ExitCode;
       ssStatus.dwWaitHint = dwWaitHint;
      
       ssStatus.dwCheckPoint = dwCheckPoint++;
      
       // Report the status of the service to the service control manager.

       return SetServiceStatus(sshStatusHandle, &ssStatus);
   }

svlMVStub_StartMemoryValidator

extern "C" SVL_SERVICE_ERROR svlMVStub_StartMemoryValidator();

Once you have successfully loaded the Memory Validator DLL you can start the Validator inspecting the service by calling svlMVStub_StartMemoryValidator().

svlMVStub_UnloadMemoryValidator

extern "C" SVL_SERVICE_ERROR svlMVStub_UnloadMemoryValidator();

To unload the Validator DLL from your service call svlMVStub_UnloadMemoryValidator(), do not call FreeLibrary(). svlMVStub_UnloadMemoryValidator() shuts down various communications threads and removes any hooks that the Validator may have installed.

Debugging functions

We have added the following debugging functions to the NT Service API.

Before using these functions you must first set the log file name, otherwise the other debugging functions will do nothing.

The log file is opened for each write to the log file then closed afterwards. As such these log file functions are slow. Do not use these functions in any performance sensitive task. They are intended to allow you get debugging information out of a service when you’re trying to work out why something isn’t working, should you have problems.

extern "C" void svlMVStub_setLogFileName(const wchar_t* fileName);

extern "C" void svlMVStub_deleteLogFile();

extern "C" void svlMVStub_writeToLogFileA(const char* text);

extern "C" void svlMVStub_writeToLogFileW(const wchar_t* text);

extern "C" void svlMVStub_writeToLogFile(SVL_SERVICE_ERROR errCode);

extern "C" void svlMVStub_writeToLogFileLastError(DWORD errCode);

extern "C" void svlMVStub_dumpPathToLogFile();

svlMVStub_setLogFileName

To use any of the following logging functions you first need to set the name of the filename used for logging.
Setting this filename also sets the filename used by some of these API functions – you will find additional logging data from those functions that will help debug any issues with the service.

extern "C" void svlMVStub_setLogFileName(const wchar_t* fileName);

svlMVStub_deleteLogFile

extern "C" void svlMVStub_deleteLogFile();

This function deletes the log file.

svlMVStub_writeToLogFileA

extern "C" void svlMVStub_writeToLogFileA(const char* text);

Call this function to write a standard ANSI character string to the log file.

svlMVStub_writeToLogFileW

extern "C" void svlMVStub_writeToLogFileW(const wchar_t* text);

Call this function to write a unicode character string to the log file. The characters are cast to ANSI prior to writing. As such you can’t write Korean to the log file. This is simply a convenience function to write wide characters as simply as ANSI characters.

svlMVStub_writeToLogFile

extern "C" void svlMVStub_writeToLogFile(SVL_SERVICE_ERROR errCode);

Call this function to write a human readable description of the SVL_SERVICE_ERROR error code to the log file.

svlMVStub_writeToLogFileLastError

extern "C" void svlMVStub_writeToLogFileLastError(DWORD errCode);

Call this function to write a human readable description of the Windows error code to the log file.

svlMVStub_dumpPathToLogFile

extern "C" void svlMVStub_dumpPathToLogFile();

Call this function to write the contents of the PATH environment variable to the log file. This can be useful if you want to know what the search path is when trying to debug why a DLL wasn’t found during an attempt to load the Validator DLL.

NT Service API for each Validator

C++ Bug Validator

extern "C" SVL_SERVICE_ERROR svlBVStub_LoadBugValidator();

extern "C" SVL_SERVICE_ERROR svlBVStub_LoadBugValidator6432();

extern "C" SVL_SERVICE_ERROR svlBVStub_StartBugValidator();

extern "C" SVL_SERVICE_ERROR svlBVStub_UnloadBugValidator();

extern "C" SVL_SERVICE_ERROR svlBVStub_SetServiceCallback(serviceCallback_FUNC callback,
                                                          void*                userParam);

// debugging functions

extern "C" void svlBVStub_setLogFileName(const wchar_t* fileName);

extern "C" void svlBVStub_deleteLogFile();

extern "C" void svlBVStub_writeToLogFileA(const char* text);

extern "C" void svlBVStub_writeToLogFileW(const wchar_t* text);

extern "C" void svlBVStub_writeToLogFile(SVL_SERVICE_ERROR errCode);

extern "C" void svlBVStub_writeToLogFileLastError(DWORD errCode);

extern "C" void svlBVStub_dumpPathToLogFile();

C++ Coverage Validator

extern "C" SVL_SERVICE_ERROR svlCVStub_LoadCoverageValidator();

extern "C" SVL_SERVICE_ERROR svlCVStub_LoadCoverageValidator6432();

extern "C" SVL_SERVICE_ERROR svlCVStub_StartCoverageValidator();

extern "C" SVL_SERVICE_ERROR svlCVStub_UnloadCoverageValidator();

extern "C" SVL_SERVICE_ERROR svlCVStub_SetServiceCallback(serviceCallback_FUNC callback,
                                                          void*                userParam);

extern "C" void svlCVStub_setLogFileName(const wchar_t* fileName);

extern "C" void svlCVStub_deleteLogFile();

extern "C" void svlCVStub_writeToLogFileA(const char* text);

extern "C" void svlCVStub_writeToLogFileW(const wchar_t* text);

extern "C" void svlCVStub_writeToLogFile(SVL_SERVICE_ERROR errCode);

extern "C" void svlCVStub_writeToLogFileLastError(DWORD errCode);

extern "C" void svlCVStub_dumpPathToLogFile();

C++ Memory Validator

extern "C" SVL_SERVICE_ERROR svlMVStub_LoadMemoryValidator();

extern "C" SVL_SERVICE_ERROR svlMVStub_LoadMemoryValidator6432();

extern "C" SVL_SERVICE_ERROR svlMVStub_StartMemoryValidator();

extern "C" SVL_SERVICE_ERROR svlMVStub_UnloadMemoryValidator();

extern "C" SVL_SERVICE_ERROR svlMVStub_SetServiceCallback(serviceCallback_FUNC callback,
                                                          void*                userParam);

extern "C" void svlMVStub_setLogFileName(const wchar_t* fileName);

extern "C" void svlMVStub_deleteLogFile();

extern "C" void svlMVStub_writeToLogFileA(const char* text);

extern "C" void svlMVStub_writeToLogFileW(const wchar_t* text);

extern "C" void svlMVStub_writeToLogFile(SVL_SERVICE_ERROR errCode);

extern "C" void svlMVStub_writeToLogFileLastError(DWORD errCode);

extern "C" void svlMVStub_dumpPathToLogFile();

C++ Performance Validator

extern "C" SVL_SERVICE_ERROR svlPVStub_LoadPerformanceValidator();

extern "C" SVL_SERVICE_ERROR svlPVStub_LoadPerformanceValidator6432();

extern "C" SVL_SERVICE_ERROR svlPVStub_StartPerformanceValidator();

extern "C" SVL_SERVICE_ERROR svlPVStub_UnloadPerformanceValidator();

extern "C" SVL_SERVICE_ERROR svlPVStub_SetServiceCallback(serviceCallback_FUNC callback,
                                                          void*                userParam);

extern "C" void svlPVStub_setLogFileName(const wchar_t* fileName);

extern "C" void svlPVStub_deleteLogFile();

extern "C" void svlPVStub_writeToLogFileA(const char* text);

extern "C" void svlPVStub_writeToLogFileW(const wchar_t* text);

extern "C" void svlPVStub_writeToLogFile(SVL_SERVICE_ERROR errCode);

extern "C" void svlPVStub_writeToLogFileLastError(DWORD errCode);

extern "C" void svlPVStub_dumpPathToLogFile();

C++ Thread Validator

extern "C" SVL_SERVICE_ERROR svlTVStub_LoadThreadValidator();

extern "C" SVL_SERVICE_ERROR svlTVStub_LoadThreadValidator6432();

extern "C" SVL_SERVICE_ERROR svlTVStub_StartThreadValidator();

extern "C" SVL_SERVICE_ERROR svlTVStub_UnloadThreadValidator();

extern "C" SVL_SERVICE_ERROR svlTVStub_SetServiceCallback(serviceCallback_FUNC callback,
                                                          void*                userParam);

extern "C" void svlTVStub_setLogFileName(const wchar_t* fileName);

extern "C" void svlTVStub_deleteLogFile();

extern "C" void svlTVStub_writeToLogFileA(const char* text);

extern "C" void svlTVStub_writeToLogFileW(const wchar_t* text);

extern "C" void svlTVStub_writeToLogFile(SVL_SERVICE_ERROR errCode);

extern "C" void svlTVStub_writeToLogFileLastError(DWORD errCode);

extern "C" void svlTVStub_dumpPathToLogFile();
Share

Updated error codes for all Validator tools

By , May 12, 2017 12:01 pm

We’ve just updated our documentation for all our Validator tools to include an up to date list of Exit return codes. You may find these useful if you’re running these tools from the command line.

These error codes apply to C++ Bug Validator, C++ Coverage Validator, C++ Memory Validator, C++ Performance Validator, C++ Thread Validator, .Net Coverage Validator, .Net Memory Validator, .Net Performance Validator and VM Validator.

0 All ok
-1 Unknown error. An unexpected error occurred starting the runtime
-2 Application started ok. You should not see this code returned
-3 Application failed to start. E.g. runtime not present, not an executable or injection dll not present
-4 Target application is not an application
-5 Don’t know what format the executable is, cannot process it
-6 Not a 32 bit application
-7 Not a 64 bit application
-8 Using incorrect MSVCR(8|9).DLL that links to CoreDLL.dll (incorrect DLL is from WinCE)
-9 Win16 app cannot start these because we can’t inject into them
-10 Win32 app – not used
-11 Win64 app – not used
-12 .Net application
-13 User bailed out because app not linked to MSVCRT dynamically
-14 Not found in launch history
-15 DLL to inject was not found
-16 Startup directory does not exist
-17 Symbol server directory does not exist
-18 Could not build a command line
-19 No runtime specified, cannot execute script (or Java) (obsolete)
-20 Java arguments are OK – not an error (obsolete)
-21 Java agentlib supplied that is not allowed because Java Bug Validator uses it (obsolete)
-22 Java xrun supplied that is not allowed because Java Bug Validator uses it (obsolete)
-23 Java cp supplied that is not allowed because Java Bug Validator uses it (obsolete)
-24 Java classpath supplied that is not allowed because Java Bug Validator uses it (obsolete)
-25 Firefox is already running, please close it (obsolete)
-26 Lua runtime DLL version is not known (obsolete)
-27 Not compatible software
-28 InjectUsingCreateProcess, no DLL name supplied
-29 InjectUsingCreateProcess, Unable to open PE File when inspecting DLL
-30 InjectUsingCreateProcess, Invalid PE File when inspecting DLL
-31 InjectUsingCreateProcess, No Kernel32 DLL
-32 InjectUsingCreateProcess, NULL VirtualFree() from GetProcAddress
-33 InjectUsingCreateProcess, NULL GetModuleHandleW() from GetModuleHandleW
-34 InjectUsingCreateProcess, NULL LoadLibraryW() from LoadLibraryW
-35 InjectUsingCreateProcess, NULL FreeLibrary() from FreeLibrary
-36 InjectUsingCreateProcess, NULL VirtualProtect() from GetProcAddress
-37 InjectUsingCreateProcess, NULL VirtualFree() from GetProcAddress
-38 InjectUsingCreateProcess, unable to find DLL load address
-39 InjectUsingCreateProcess, unable to write to remote process’s memory
-40 InjectUsingCreateProcess, unable to read remote process’s memory
-41 InjectUsingCreateProcess, unable to resume a thread
-42 UPX compressed – cannot process such executables
-43 Java class not found in CLASSPATH
-44 Failed to launch the 32 bit svlGetProcAddressHelperUtil.exe
-45 Uknown error with svlGetProcAddressHelperUtil.exe
-46 Couldn’t load specified DLL into svlGetProcAddressHelperUtil.exe
-47 Couldn’t find function in the DLL svlGetProcAddressHelperUtil.exe
-48 Missing DLL argument svlGetProcAddressHelperUtil.exe
-49 Missing function argument svlGetProcAddressHelperUtil.exe
-50 Missing svlGetProcAddressHelperUtil.exe
-51 Target process has a manifest that requires elevation
-52 svlInjectIntoProcessHelper_x64.exe not found
-53 svlInjectIntoProcessHelper_x64.exe failed to start
-54 svlInjectIntoProcessHelper_x64.exe failed to return error code
-55 getImageBase() worked ok
-56 ReadFile() failed in getImageBase()
-57 NULL pointer when trying to allocate memory
-58 CreateFile() failed in getImageBase()
-59 ReadProcessMemory() failed in getImageBase()
-60 VirtualQueryEx() failed in getImageBase()
-61 Bad /appName argument in svlInjectIntoProcessHelper_x64.exe
-62 Bad /dllName argument in svlInjectIntoProcessHelper_x64.exe
-63 Bad /procId argument in svlInjectIntoProcessHelper_x64.exe
-64 Failed to OpenProcess in svlInjectIntoProcessHelper_x64.exe
-65 A DLL that the .exe depends upon cannot be found
Share

Working with Dev C++

By , December 14, 2016 5:57 pm

We’ve had a few people asking how to configure C++ Memory Validator to work with programs built using Dev C++. Dev C++ is an IDE used for developing programs with the MingW compiler.

We tested using this download of Dev C++.

Debug information

Any program built using the default Dev C++ settings will generate a binary image that contains debugging information that is in a format that our tools cannot read. The MingW compiler can create debug information in many formats, including COFF and STABS, both of which our tools support. You can turn these debugging formats on using the -gCoff and -gstabs flags. We recommend using STABS symbols.

Configuring Dev C++

Open the Project Options… dialog from the Project menu.
devcppprojectmenu

Choose the Parameters tab. Add the option -gstabs to all three columns. Click OK.
devcpp_debugoptions

Now that you have configured the debug options all you need to do is to rebuild your project to ensure the debug information is present.

Share

The correct way to determine if a file is a directory.

By , November 30, 2016 1:41 pm

After writing for Microsoft’s Windows platform for 20 years I thought I knew all I could know about GetFileAttributes(). Until I found a rather odd and subtle bug in some code that interacted with data supplied by the user of the software. A call would succeed that I expected to fail. Naturally this meant the software didn’t make the right choices and instead of being presented with an helpful dialog explaining what had failed, the software sat silently in a corner humming to itself waiting for the user to work out what had happened. The failure was that I was presenting incorrect data to GetFileAttributes() assuming that it would always fail for bad input. How wrong I was!

I thought I’d write up what can go wrong with GetFileAttributes().

It’s tempting to test if a file is a directory by writing code like this:

if ((GetFileAttributes(fileName) & FILE_ATTRIBUTE_DIRECTORY) != 0)
{
    // file is a directory
}

The above looks logically correct. But there are problems with it.

First, a refresher on file attribute values…

File Attributes

The list of defined file attributes is in WinNT.h. The values are shown below.

#define FILE_ATTRIBUTE_READONLY             0x00000001  
#define FILE_ATTRIBUTE_HIDDEN               0x00000002  
#define FILE_ATTRIBUTE_SYSTEM               0x00000004  
#define FILE_ATTRIBUTE_DIRECTORY            0x00000010  
#define FILE_ATTRIBUTE_ARCHIVE              0x00000020  
#define FILE_ATTRIBUTE_DEVICE               0x00000040  
#define FILE_ATTRIBUTE_NORMAL               0x00000080  
#define FILE_ATTRIBUTE_TEMPORARY            0x00000100  
#define FILE_ATTRIBUTE_SPARSE_FILE          0x00000200  
#define FILE_ATTRIBUTE_REPARSE_POINT        0x00000400  
#define FILE_ATTRIBUTE_COMPRESSED           0x00000800  
#define FILE_ATTRIBUTE_OFFLINE              0x00001000  
#define FILE_ATTRIBUTE_NOT_CONTENT_INDEXED  0x00002000  
#define FILE_ATTRIBUTE_ENCRYPTED            0x00004000  
#define FILE_ATTRIBUTE_VIRTUAL              0x00010000  

Rather strangely, the invalid attributes flag is defined in a different file, WinBase.h.

#define INVALID_FILE_ATTRIBUTES ((DWORD)-1)

Problem 1

What if GetFileAttributes() fails? If the file doesn’t exist, the call fails. If the filename specifies a computer name, the call fails. See GetFileAttributes() documentation for more informtion. When GetFileAttributes() fails it returns INVALID_FILE_ATTRIBUTES. This error status passes the above test. OK, so add an additional check and the code becomes

DWORD attribs;

attribs = GetFileAttributes(fileName);
if ((attribs != INVALID_FILE_ATTRIBUTES) &&
    ((attribs & FILE_ATTRIBUTE_DIRECTORY) != 0))
{
    // file is a directory
}

Problem 2

Even with the above file-does-not-exist problem solved there is another problem. The file could be a directory, but it could be a directory that you don’t want. For example what if you’ve allowed the user to specify the directory name and they typed _T(“/”), or what if your filename creation code has a bug in it that fails when passed an empty name, resulting in a calculated filename of _T(“\”). What then?

In these cases the following calls all return 0x16.

GetFileAttributes(_T("\\")); 
GetFileAttributes(_T("/"));

0x16 means hidden (0x02), system (0x04), directory (0x10).

It’s a reasonable bet that in your code, any code looking for a directory to use is probably not looking for a hidden directory and almost certainly not intending to use a system directory. OK, time for a new implementation.

DWORD attribs;

attribs = GetFileAttributes(fileName);
if ((attribs != INVALID_FILE_ATTRIBUTES) &&         // check if a valid file
    ((attribs & FILE_ATTRIBUTE_DIRECTORY) != 0) &&  // file is a directory
    ((attribs & FILE_ATTRIBUTE_HIDDEN) == 0) &&     // file is not hidden
    ((attribs & FILE_ATTRIBUTE_SYSTEM) == 0))       // file is not system
{
    // file is a directory that isn't hidden and isn't system
}

What about files, rather than directories?

It’s natural to think about implementing checks for if a filename identifies a file rather than a directory. You test for this in exactly the same way but looking for different attributes. You’ll want to exclude FILE_ATTRIBUTE_DIRECTORY and then depending on the job your code is doing you’ll want to consider excluding files depending upon the following attributes:

FILE_ATTRIBUTE_DEVICE
FILE_ATTRIBUTE_INTEGRITY_STREAM
FILE_ATTRIBUTE_OFFLINE
FILE_ATTRIBUTE_REPARSE_POINT
FILE_ATTRIBUTE_SPARSE_FILE
FILE_ATTRIBUTE_TEMPORARY
FILE_ATTRIBUTE_VIRTUAL

and of course, you might also want to consider FILE_ATTRIBUTE_HIDDEN and FILE_ATTRIBUTE_SYSTEM.

Additional reading

Microsoft documentation on GetFileAttributes().

Why is GetFileAttributes the way old-timers test file existence? Old New Thing.

Share

We’ve been quiet for a while, sorry about that.

By , November 28, 2016 3:29 pm

Hi,

It’s been a while since we posted anything on the blog. If you weren’t a customer, regularly receiving our software update emails you might think we weren’t going anything.

That’s an oversight on our part. We’re hoping to rectify this over the next few months, posting more useful information both here and in the library.

_tempnam and friends

Our most recent update has been to update C++ Memory Validator provide memory tracking for the _tempnam group of functions. These are _tempnam, _tempnam_dbg, _wtempnam, _wtempnam_dbg.

This support is for all supported compilers, from Visual Studio 2015, Visual Studio 2013, Visual Studio 2012, Visual Studio 2010, Visual Studio 2008, Visual Studio 2005, Visual Studio 2003, Visual Studio 2002, Visual Studio 6. Delphi and C++ Builder, Metrowerks compiler, MingW compiler.

.Net support, for the future

Internal versions of C++ Coverage Validator can provide code coverage statistics for .Net (C#, VB.Net, J#, F#, etc) as well as native languages (C++, C, Delphi, Fortran 95, etc).

Internal versions of C++ Performance Validator can provide performance profiling statistics for .Net (C#, VB.Net, J#, F#, etc) as well as native languages (C++, C, Delphi, Fortran 95, etc).

UX improvements

All tools, free and paid, have had the UX for filename and directory editing improved so that if a filename doesn’t exist it is displayed in red and if it does exist it is displayed in it’s normal colour (typically black). See screenshots (from Windows 8.1).

Non existent filename:
filenamebad

Existing filename:
filenamegood

Share

Marmalade game SDK support

By , December 16, 2015 12:02 pm

We’ve recently added support for the Marmalade game SDK to C++ Memory Validator.

This article will show you how to configure a Marmalade project for use with C++ Memory Validator, how to setup C++ Memory Validator for use with Marmalade and how to launch a Marmalade game from C++ Memory Validator.

Configuring your project settings

To work with C++ Memory Validator you need to build the x86 Debug configuration and/or the x86 Release configuration of your Marmalade project using Visual Studio.

These configurations need to be built so that they create debug information and so that a PDB file containing debug information is created. The example projects that ship with Marmalade do not do this – you will need to edit them to make the linker stage create debug information.

Editing the compile stage debug information

marmalade-vs-compile


Editing the link stage debug information

marmalade-vs-linker


You must ensure that both compile and link stages have the correct settings set. If only compile or only link is set you will not get debugging symbols.

Debugging symbols are important for two reasons:

  • Without symbols C++ Memory Validator cannot find the Marmalade memory allocators and will not be able to track the Marmalade memory allocations your game makes.
  • Without symbols C++ Memory Validator will not be able to turn callstack addresses into class names, function names, filenames and line numbers.

Configuring C++ Memory Validator

In order to work correctly with Marmalade we need to make sure we’re only going to track the memory allocation your game makes with Marmalade and not any of the work that the Marmalade game engine is doing itself. We need to make a few simple changes to C++ Memory Validator.

  • Open the settings dialog. Click Reset. This will reset all C++ Memory Validator settings to the default.
  • Go to the Collect tab, disable the check boxes in the top two groups of check boxes, then enable the single Marmalade check box. The settings should look like shown below.
  • Click OK.

settings-collect-marmalade


Launching the game

To launch a Marmalade game with C++ Memory Validator we launch the Marmalade simulator and specify the game to run using the Marmalade -via command line argument.

If Marmalade is installed in c:\marmalade then the path to the simulator is

c:/marmalade/8.0/s3e\win32\s3e_simulator_release.exe

If an example game (shipped with Marmalade) is found at this location

c:\Marmalade\8.0\examples\GameTutorial\Lua\Stage9\build_temp\build_stage9_vc14\Stage9_vc14_release.via

then the command line is

-via:"c:\Marmalade\8.0\examples\GameTutorial\Lua\Stage9\build_temp\build_stage9_vc14\Stage9_vc14_release.via"

and the startup directory is

c:\Marmalade\8.0\examples\GameTutorial\Lua\Stage9\build_temp\build_stage9_vc14\

We leave the Application to monitor unchanged. It should have the same value as Application to launch.

This is how the launch dialog looks when you are launching this game.

launch-marmalade


Click Go! to launch the Marmalade game. C++ Memory Validator will monitor all memory allocations, reallocations and deallocations made via the s3eMalloc, s3eRealloc, s3eFree functions. Run your game as normal, then close your game. Wait for C++ Memory Validator to show you any leaks on the Memory tab. Additional statistics can be views on the Objects, Sizes, Timeline and Hotspots tabs.

Share

How to speed up Memory Validator by changing DbgHelp version

By , September 6, 2015 9:26 am

Recently we’ve had a few customers contact to tell us they have experienced a dramatic reduction in speed when using C++ Memory Validator.

We found this puzzling as we hadn’t really noticed this. We investigate and found that some parts of our code were hitting the disk a bit too much. To address this we implemented a buffered file read/write system so that we hit the disk once rather than many times. For our test case (which was a substantial program being monitored) this worked wonders. Performance improved enormously. Smiles all round.

But our customers still reported problems. This was puzzling. We started logging everything in one particular code path (which we knew was the problem). Nothing obvious. Next step, start timing all the logging. But just before we got to that we did a simple test. We iterated through each version of DbgHelp.dll that C++ Memory Validator can supply – if you remember we let you specify which Visual Studio version you used and we supply a version of DbgHelp.dll that ships and works for that (not all DbgHelp.dll are equal!).

Imagine our surprise when we found that DbgHelp.dll (6.11.1.404) shipped prior to Visual Studio 2013 are blazingly fast and the DbgHelp.dll we supply for use with Visual Studio 2013/2015 (6.3.9431.0) are slow. If you’re paying attention you’ll also notice the DbgHelp.dll version number has decreased rather than increased – ask Microsoft, we have no idea why they decreased the version number with more recent releases.

For now, until we can get a new release out to address this anomaly we recommend that you ignore choosing the Visual Studio you are using and deliberately choose Visual Studio 2012. This will select DbgHelp 6.11.1.404 and you should find the usual blazing speeds you are used to are restored.

To change the version of DbgHelp used, open the settings dialog, go to Symbol Lookup then change the version in the combo box. Click OK.

Any problems, as usual, please contact support.

Share

Panorama Theme by Themocracy