Category: Development

The Command Line Builder

By , April 4, 2022 1:32 pm

Last week we introduced the command line builder to all our Validator tools – Bug Validator, Coverage Validator, Memory Validator, Performance Validator and Thread Validator.

All our tools provide command line support allowing you to use them from your favourite automation tool, be that a set of batch files, or something more comprehensive like Jenkins.

Command lines are great, but writing out your command line can be daunting, even if like me, you’ve done it many times. It’s not that it’s hard, it’s just that you don’t do it very often so the options are never top of mind, you’re going to have to go and look at the dreaded documentation.

Yup, documentation, even when it’s as good as ours is, you’re still reluctant to look at it. I know the feeling. You’ve got so used to using a nice GUI that it seems odd to look at the help. But you’ve got to do that if you’re going to write a command line for a tool.

That’s where the Command Line Builder comes in. To reduce the amount of time spent looking at the documentation, and to reduce the likelihood of you making mistakes when creating your command line.

How do I start the command line builder?

The command line builder can be started from the Launch menu and from any of the Launch dialogs and Launch wizards.

Launch Menu

Launch menu

When the command line builder is started from the launch menu you are presented with a set of choices that allow you to preconfigure the command line builder.

Command line builder wizard

The choices provided aim to cover every possible way of using the Validator.

  • Build my own command line.
  • Use a predefined template that provides the command line arguments for common command line tasks. These templates are configured specific to the validator being used. If the Validator supports .Net and .Net Core as well as native there will be .Net specific templates and .Net Core specific templates. If the Validator can also do specific things like merging sessions or comparing sessions, templates for these actions are also provided.
  • Use an existing command line I use to launch my program. You know what you’d type on a command prompt to launch your program. Type that here and let the command line builder turn that into a command line for the Validator.
  • Use an existing Validator command line. Reuse a command line you already have and tweak it a bit to do the new task.
  • Use an existing Validator command file. Reuse a command file you already have and tweak it a bit to do the new task.

With the exception of the first option that allows you to build your command line from scratch, all of these options are present to enable you to save time when building your command line.

Once you’ve chosen your option (and template if you’re using a predefined template) you need to move to then next stage, with the thoughtfully named Next button.

Launch dialog / wizard

When you start the command line builder from a launch dialog or wizard it’s via the Cmd Line… button.

Launch dialog showing Cmd Line... button

When you start the command line builder this way, a command line is built based on the options you have selected on the launch dialog/wizard, and then passed to the command line builder. This allows you to select a known good command line you’ve previously used then click the Cmd Line… button and instantly be able to configure that command line in the command line builder.

Command Line Builder dialog

Once you’ve started the command line builder you’ll be presented with a dialog that will look very similar to this. This example is using a preconfigured template for Coverage Validator that merges coverage data to a central session, exports HTML and XML coverage data, and saves the coverage session.

The first thing you’ll notice is there are many argument values highlighted in red. These are values that are incorrect. This is because they are from a template and you’ll need to edit them to match your own working practices – in this example you’ll need to edit the program being tested and change c:\myResults to be where you wish to save your test results and edit the filenames to something meaningful for your tests.

Each command line argument is displayed on it’s own line, with a value (for the arguments that have a value) and a description of what this argument/value pair represent. If you need help, the ? button in the dialog border will take you there.

Editing

The buttons on the right hand side provide simple editing of the list of argument values to allow you to Add an argument, Edit an argument or value, Remove an argument and Remove All arguments. You can also start editing any existing argument or value by double clicking it.

Editing an argument presents you with a combo box of valid Validator argument values, preventing you from entering incorrect command line arguments.

Editing a value that is an On/Off, Yes/No, True/False type of argument presents you with those choices.

Editing a value that is an enumeration will only show you valid enumeration values.

Editing a value that needs a directory will open a directory chooser dialog, editing a value that needs a file will open a file chooser dialog set to choose that type of file.

All other editing types use a standard text dialog, with limited text input (for example integer values only allow integer input).

As you can see we are trying to reduce the scope for input error as much as we can.

Additional Customization

Next to the editing controls are some customization controls which allow you to add command line arguments (and associated values) to your command line if they are not already present.

Add Hide adds the -hideUI argument which you will need if you are writing command lines for automated use that close the Validator after the target program finishes.

Add Debug adds options to help debugging failures with the command line. These options ensure the command prompt is displayed, that the arguments you think you are supplying are the arguments you’ve supplied, displays error information on the command prompt and displays error information in a message box.

Add Export adds options for exporting in HTML and XML. For Coverage Validator an additional XML Cobertura export option is available.

A fourth button is displayed for some Validators. This provides options specific for that Validator, for example merging coverage data, or comparing memory leak reports or comparing performance reports. In the image show above you can see Add Merge, which is for Coverage Validator.

The last button allows you to Import an existing command file. Using this destroys all existing arguments and replaces them with the values in the command file.

Creating the command line

Once you’ve finished creating your command line the next thing is to get the command line in a form that you can use. We provide two forms of command line creation. The normal command line that most people expect and a much shorter command line combined with a command file that contains the actual command line instructions for the Validator.

The choice of command line is made using the combo box towards the lower half of the dialog.

Command line with arguments

This option is the typical command line that people expect. It starts the Validator and passes all command line arguments and values to the Validator.

The command line is updated each time the arguments and values are edited.

The Copy button copies the command line to the clipboard.

Command line with command file

This option provides a short command line that loads the command file, with the command file containing all the command line instructions for the Validator. There are two reasons for using the command file:

  • A shorter command line. Current versions of Windows support very long command lines. Earlier versions of Windows supported much shorter command lines – it was quite easy to construct a Validator command line that was too long for Windows to process, resulting in incorrect behaviour when you tried to use the command line. Using the command file allows you to specify long command lines on older operating systems.
  • Simplifying command line management. Storing all your command lines in command files allows you to easily manage the command files, storing them all in one place. It also allows some indirection, allowing you to edit the command line without editing the script/tool that invokes the Validator.

When you choose to use a command line with a command file, if you have not specified a command file to use you will be prompted for a command file name. When a command file name is specified the command file will be updated each time the arguments and values are edited.

The Browse… button allows you to use a file browser to choose a command file name. The View… button allows you to view the command file in your favourite editor.

Testing

Once you have created the command line arguments and values, and chosen the type of command line you want to use, you may want to test the command line. This is done by using the thoughtfully named Test Command Line button.

If all goes well a new Validator instance will be launched with your command line, and some time later the target program will complete.

Unfortunately not all command lines are equal, and some will fail to run correctly. So how do you debug that?

Incorrect program

If you have misconfigured the command line (for example: by specifying a 64 bit program to be tested by a 32 bit Validator) you will not get the behaviour you expect.

If you have specified the startup program correctly but failed to ensure that all dependent DLLs can be found the program won’t start, or it will start and then fail shortly afterwards. Check you can start the program in this directory by double clicking it – if that doesn’t work you’ve probably got a missing dependency. You can check your dependencies using PE File Browser.

If you have specified a startup directory that does not exist, the program won’t start.

Unable to see error dialogs and warning dialogs

If you have specified -hideUI you won’t be able to see what happens, so if you are having problems, the first thing to try is to change -hideUI to -showUI and test again to see if any warning dialogs are displayed.

Show the command prompt

Use -showCommandPrompt to ensure the command prompt can be seen. You may find a helpful error message waiting for you.

Check the arguments supplied

Use -echoArgsToUser to check the arguments supplied are what you think they should be.

Ensure you can see errors

Use -showErrorsWithMessageBox to display any error messages with a message box. Use this when your program has a graphical user interface.

Use -showErrorsOnCommandPrompt to display any error messages on the command prompt. Use this when your program is a console program and doesn’t have a GUI.

Conclusion

The Command Line Builder simplifies the creation of valid command lines for use with the Validator tools. We’ve tried to eliminate errors by making it easier to create valid command lines while also highlighting errors with proposed command line values.

Exceptions Codes you’ve never heard of

By , January 28, 2022 3:33 pm

If you write software for any length of time you’ll have seen your software fail in a variety of ways.

Exceptions are a common cause of application failure.

Exceptions are also used to handle unusual error cases, with the intention that the calling code is aware of this exception and will handle it, ensuring that the application doesn’t fail.

For this article I’m going to be discussing Win32 structured exception handling (SEH) exception codes. I’m also going to assume that you’re using Visual Studio, a Windows SDK or another IDE, such as C++ Builder, or Delphi.

Most of the exception codes you’ve likely encountered in your career can be found declared in ntstatus.h

Depending on which version of Visual Studio you’re using some exception codes may not present.

I’m going to list exception codes that we’ve found during the process of developing software tools at Software Verify.

Many of these exception codes are not documented, or are only present in recent versions of SDKs, which you may not be using because you’re still using an older Visual Studio.

Where possible we’ll indicate additional sources of information about a given exception code. We’ll indicate the actual exception code and what name we use to reference it. We’ll also include a downloadable header file so that you can use these definitions without needing to type them in yourself.

Visual Studio Specific

These exceptions are specific to Microsoft Visual Studio and Microsoft compilers and linkers.

Microsoft Thread Naming Exception

0x406D1388 SVL_THREAD_NAMING_EXCEPTION

This exception is used to communicate to a debugger (or any other monitoring tool) a suggested name to represent the thread. This is technique is discussed in this blog post. It has been replaced in Windows 10 by SetThreadDescription().

Visual Studio C++ Exception

0xE06D7363 SVL_MSVC_EXCEPTION

This exception is used to implement C++’s exception handling.

This is discussed in this blog post.

.Net specific

These exceptions are specific to the Common Language Runtime (CLR), used to implement .Net. Most of these exception codes we found while inspecting .Net Core open source code. A few of them we have found elsewhere.

CLR Fatal JIT Exception

0x02345678 SVL_CLR_NET_FATAL_JIT_EXCEPTION

This definition comes from .Net Core open source code.

CLR Data Checksum Exception

0x31415927 SVL_CLR_NET_CLRDBG_DATA_CHECKSUM_EXCEPTION

This definition comes from .Net Core open source code.

CLR First Chance Exception

0x04242420 SVL_CLRDBG_NOTIFICATION_EXCEPTION_CODE

CLR Bootup COM+ Exception

0xC0020001 SVL_CLR_NET_BOOTUP_COMPLUS_EXCEPTION

This definition comes from .Net Core open source code.

CLR Exception

0xE0434F4D SVL_CLR_NET_EXCEPTION

CLR COM+ Exception

0xE0434352 SVL_CLR_NET_COMPLUS_EXCEPTION

This definition comes from .Net Core open source code.

Many causes of this exception are discussed here.

CLR HIJACK Exception

0xE0434F4E SVL_CLR_NET_HIJACK_EXCEPTION

This definition comes from .Net Core open source code.

CLR Notify Exception

0xE0444143 SVL_CLR_NET_CLRDATA_NOTIFY_EXCEPTION

This definition comes from .Net Core open source code.

CLR EXX Exception

0xE0455858 SVL_CLR_NET_EXX_EXCEPTION

This definition comes from .Net Core open source code.

CLR SEH Verification Exception

0xE0564552 SVL_CLR_NET_SEH_VERIFICATION_EXCEPTION

This definition comes from .Net Core open source code.

CLR Internal ASSERT Exception

0xE0584D4E SVL_CLR_NET_INTERNAL_ASSERT_EXCEPTION

This definition comes from .Net Core open source code.

Embarcadero specific

The exceptions are thrown by software made with tools from Embarcadero (formerly Borland).

Delphi Exception 1

0x0EEDFADE SVL_DELPHI_EXCEPTION_1

This is an exception thrown by Delphi.

Delphi Exception 2

0x0EEDFACE SVL_DELPHI_EXCEPTION_2

This is an exception thrown by Delphi.

C++ Builder Exception

0x0EEFFACE SVL_CPP_BUILDER_EXCEPTION

This is an exception thrown by C++ Builder.

Windows SDK

These exceptions are Windows exceptions that may or may not be defined in the IDE/SDK that you are using.

For more information you can look up error codes and exceptions here and here.

Fatal Memory Exhaustion

0xC00001AD STATUS_FATAL_MEMORY_EXHAUSTION

The process cannot allocate any more memory.

Fail Fast Exception

0xC0000602 STATUS_FAIL_FAST_EXCEPTION

This is used by the Kernel to handle fail fast exceptions. See the list of fail fast codes.

Stack Buffer Overrun

0xC0000409 STATUS_STACK_BUFFER_OVERRUN

This used to be used to indicate that the user mode callstack had had a buffer overrun, corrupting the stack and potentially opening up the application to the a malicious attack.

This exception code has been repurposed. It is now used to handle user mode fail fast exceptions. The first parameter passed with the exception is the fast fail code.

Invalid C Runtime Parameter

0xC0000417 STATUS_INVALID_CRUNTIME_PARAMETER

This is used to indicate that invalid parameters have been passed to a C runtime function. An example of this would be passing too small a string buffer to a _stprintf_s();

Heap Corruption

0xC0000374 STATUS_HEAP_CORRUPTION

A Win32 heap (HeapCreate, HeapDestroy, HeapAlloc, HeapReAlloc, HeapFree) has had it’s internal structures corrupted.

Invalid Exception Handler

0xC00001A5 STATUS_INVALID_EXCEPTION_HANDLER

An exception handler was expected but not found.

You can often see this in 32 bit programs where the stack (containing the stack based exception handler) has been corrupted.

Unwind

0xC0000027 STATUS_UNWIND

Bad Stack

0xC0000028 STATUS_BAD_STACK

During an unwind operation an invalid stack location, or a misaligned stack location was found.

Invalid Unwind Target

0xC0000029 STATUS_INVALID_UNWIND_TARGET

During an unwind operation and invalid unwind target was found.

Fatal User Callback Exception

0xC000041D STATUS_FATAL_USER_CALLBACK_EXCEPTION

An unhandled exception was found while handling a user callback.

Invalid Image Format

0xC000007B STATUS_INVALID_IMAGE_FORMAT

The specified file is not a PE format file, or it is a PE format file for a different processor architecture than the version that Windows is running.

To inspect PE files you can use PE File Browser.

Debugger Inactive

0xC0000354 STATUS_DEBUGGER_INACTIVE

Windows may have been started without Kernel debugging enabled.

Certificate Expired

0x800B0101 CERT_E_EXPIRED

The certificate is not within its validity period when verifying against the current system clock or the timestamp in the signed file.

x86 Breakpoint

0x4000001F STATUS_WX86_BREAKPOINT

Exception code used by the Win32 x86 emulation subsystem.

Debug Reply Later

0x40010001 DBG_REPLY_LATER

Debugger will reply later.

Debug Unable to provide handle

0x40010002 DBG_UNABLE_TO_PROVIDE_HANDLE

Debugger cannot provide the handle requested.

Debug Terminate Thread

0x40010003 DBG_TERMINATE_THREAD

Thread terminated by debugger.

Debug Terminate Process

0x40010004 DBG_TERMINATE_PROCESS

Process terminated by debugger.

Debug Control-C

0x40010005 DBG_CONTROL_C

Debugger received Ctrl-C.

Debug Print Exception Char

0x40010006 DBG_PRINTEXCEPTION_C

Debugger received message from OutputDebugStringA();

Debug RIP Exception

0x40010007 DBG_RIPEXCEPTION

Debugger received RIP exception.

Debug Control Break

0x40010008 DBG_CONTROL_BREAK

Debugger received Ctrl-Break.

Debug Command Exception

0x40010009 DBG_COMMAND_EXCEPTION

Debugger communicating a command.

Debug Print Exception Wide Char

0x4001000A DBG_PRINTEXCEPTION_WIDE_C

Debugger received message from OutputDebugStringW();

Debug Exception Not Handled

0x80010001 DBG_EXCEPTION_NOT_HANDLED

Debugger did not handle the exception.

Assertion Failure

0xC0000420 STATUS_ASSERTION_FAILURE

An assertion failed. See NT_ASSERT() and NT_ASSERTMSG().

Fatal Application Exit

0x40000015 STATUS_FATAL_APP_EXIT

Application exited via a call to abort(); or as a result of a fast fail exception.

Application Hang

0xCFFFFFFF STATUS_APPLICATION_HANG

Application Verifier Stop

0xC0000421 STATUS_VERIFIER_STOP

Application verifier has found an error in the current process

Procedure Not Found

0xC06D007F STATUS_PROCEDURE_NOT_FOUND

This is a Software Verify name for this exception, we can’t find an official name.

Header File

I hope you found the above list useful.

You can download the exceptionCodes.h header file.

Changes to how we display data

By , June 12, 2020 11:59 am

We’ve changed the display of data on all of our tree controls that are used to display callstacks and call trees.

The old method of display, everything was displayed in black text at normal font weight.

The new method of display, different components of the text are displayed in their own colour, with optional bold, italic, underline or strikethrough, with an optional change in font face. In practice, at present we’re only changing the display colour and turning bold on/off.

We’re using this new display technology to leave some items neutral, emphasise some items and de-emphasise other items. This change has made quite a difference in readability of the display. Right now there is no ability for the user of the software to edit these settings to change the prioritisation and colour scheme we’ve chosen. We’ll provide this capability in a future release.

I’m going to show some before and after images so that you can see the difference for each tool.

C++ Memory Validator

For C++ Memory Validator, we’ve changed both function and filename to be displayed in bold. Type, thread name, timestamp, module name, and symbol name are displayed in different colours. Address is displayed in a de-emphasised colour.

Before


After


Before


After


.Net Memory Validator

For .Net Memory Validator, we’ve changed both function and filename to be displayed in bold. Age, Generation, thread name, timestamp, module name, and symbol name are displayed in different colours. Address is displayed in a de-emphasised colour.

Before


After


Before


After


Thread Validator

For Thread Validator, we’ve changed both function and filename to be displayed in bold. Thread name, timestamp, module name, and symbol name are displayed in different colours.

Before


After


Before


After


Performance Validator

For Performance Validator, we’ve changed both function and filename to be displayed in bold. Module name, and symbol name are displayed in different colours.

Before


After


Coverage Validator

For Coverage Validator, the branches, files and directory displays have had a few tweaks to de-emphasis address information.

Before

After

Bug Validator

For Bug Validator, we’ve re-ordered the display information so that the symbol comes before the filename. Both function and filename are displayed in bold. Timestamp, thread name, module name, and symbol name are displayed in different colours.

Before

After

Turbo Debugger Symbols Viewer

By , March 6, 2020 12:48 pm

If you’re using Delphi or 32 bit C++ Builder your compiler/linker produces symbols in TDS format. TDS means Turbo Debugger Symbols – it’s an old naming convention from the days of the Turbo compilers.

There are occasions when it’s useful to know what’s inside the symbol file. Is the symbol name as I expected? Or is it mangled to something else? Is the symbol name in the debug info? If not, then maybe the compiler optimised that function out of existence (it does happen).

We’d already written a symbol file viewer for Visual Studio PDB files. It seemed logical to write a similar tool for TDS symbols. That tool is TDS Browser.

TDS symbols can be stored in the executable to which they relate, or in a separate TDS file. The 64 bit version of Delphi doesn’t provide an option for symbols in an external TDS file, which is odd, as no one wants to ship symbols with their executable. But hey, Embarcadero must have their reasons, right? As a result to view symbols with TDS Browser you can specify the TDS file or the executable file, either way we’ll load the symbols if they exist.

Symbol name mangling is different for 64 bit symbols, so we’ve provided an option to see the raw, unmangled symbol name for the occasions when you need that.

You can sort the symbols by any column, and reverse the sort by clicking the same column again. This is useful for finding symbols with source files – sort by filename. Select a symbol and we’ll show you the source code and line numbers if we can find the source code (not easy for source files that don’t have complete paths).

We provided options for resolving addresses into symbols for the 4 main use cases:

  • Crash at absolute address in DLL.
  • Crash at relative address in DLL.
  • Crash at symbol relative address.
  • Crash data from Windows Event Log in XML format.

The above options are on the Query menu.


If the crash address can be resolved into a symbol, the symbol is selected, and the relevant source code and line number information is displayed. A context menu also provides additional options for highlighing symbols and copying symbol related information to the clipboard.


Here’s a short video showing how to use TDS Browser.

An easier way to view crashes in the Windows Event Log

By , March 6, 2020 11:45 am

In a previous article I wrote about how to identify crashes in the Windows Event Log.

You need to use the Windows Event Viewer, inspect each entry looking for some keywords then decode the XML data to get the information you want. All a bit slow, tedious and error prone.

So we wrote a tool to do that for you: Event Log Crash Browser.

Event Log Crash Browser scans your event log looking for crash events, then picks out only the information that is useful:

  • The executable.
  • The DLL that crashed (if it did crash in a DLL, rather than non-DLL memory).
  • The exception code.
  • The offset into the DLL of the crash location (or the location in memory for non-DLL crashes).
  • We also read the version information from the DLL so that we can identify the company responsible for the DLL that crashed.

You can sort on any column, and filter by exception type, executable and DLL.

It’s a really easy way to see what failures are happening on your machine. A lot more convenient than Windows Event Viewer. Looking at this machine I can see that the Visual Studio compiler, linker and IDE crash from time to time. I can also see that the WMI provider service dies quite often from a heap corruption – this is a core bit of Microsoft technology that has problems. Most of the other failures are related to the software under development and test on this machine.

Identifying crashes with the Windows Event Log

By , February 12, 2020 4:43 pm

It’s an unfortunate and inevitable fact that while developing software sometimes your software will crash. This also happens, sometimes, hopefully very infrequently, in production code. Each time this happens Windows stores some information about each crash in the Windows Event Log, along with a multitude of other event information it logs.

In this article I’m going to explain two event log entry types which encode crashes, and how to read them. Then I’ll also introduce some tools that take the drudgery out of converting this information into symbol, filename and line number.

The Windows Event Log

The Windows event log can be viewed using Microsoft’s Event Viewer. Just type “Event Viewer” in the start menu search box and press return. That should start it. Crash information is stored in the sub category “Application” under “Windows Logs”. The two event sources that describe crashes are Windows Error Reporting and Application Error.


The image above shows a Windows Error Reporting event has been selected. The human readable form is shown below in the General tab. Although I say human readable, it really is unintelligible gibberish. None of the fields are identified and you have nothing to work with. The details tab isn’t any better – the raw data is present in text or XML form. Here’s the XML for the crash shown above.

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Windows Error Reporting" /> 
    <EventID Qualifiers="0">1001</EventID> 
    <Level>4</Level> 
    <Task>0</Task> 
    <Keywords>0x80000000000000</Keywords> 
    <TimeCreated SystemTime="2020-02-12T10:09:34.000000000Z" /> 
    <EventRecordID>260507</EventRecordID> 
    <Channel>Application</Channel> 
    <Computer>hydra</Computer> 
    <Security /> 
  </System>
  <EventData>
    <Data>2023787729086567941</Data> 
    <Data>1</Data> 
    <Data>APPCRASH</Data> 
    <Data>Not available</Data> 
    <Data>0</Data> 
    <Data>testDeliberateCrash.exe</Data> 
    <Data>1.0.0.1</Data> 
    <Data>5e419525</Data> 
    <Data>testDeliberateCrash.exe</Data> 
    <Data>1.0.0.1</Data> 
    <Data>5e419525</Data> 
    <Data>c0000005</Data> 
    <Data>000017b2</Data> 
    <Data /> 
    <Data /> 
    <Data>C:\Users\stephen\AppData\Local\Temp\WERC24C.tmp.WERInternalMetadata.xml</Data> 
  <Data>C:\Users\stephen\AppData\Local\Microsoft\Windows\WER\ReportArchive\AppCrash_testDeliberateCr_c31b903842d94a84d4621dceaac377462674f7a_eb589596_139ec4bd</Data> 
    <Data /> 
    <Data>0</Data> 
    <Data>c3d360b2-4d7f-11ea-83d3-001e4fdb3956</Data> 
    <Data>0</Data> 
    <Data>54756af49aec84f97c15f03794ffd605</Data> 
  </EventData>
</Event>

There’s quite a bit of data in here, the purpose of each field implied, not stated. Towards the end is some information related to minidumps, but if you go searching for it, the minidump will no longer be present.

The format for Application Error crashes is different.

Windows Error Reporting

The event log data for a Windows Error Reporting event contains many fields that we don’t need if we’re just investigating a crash address. Each event starts with an <Event> tag and ends with an </Event> tag.

We need to correctly identify the event. Inside the event is a <System> tag which contains a tag with an attribute “Provider Name” set to “Windows Error Reporting”.

Once the event is identified we need to find the <EventData> tag inside the event. The <EventData> contains 14 <Data> tags. These tags are present:


  • 1. Timestamp.

  • 2. Number of data items.

  • 3. Information Type.

  • 4. Information Status.

  • 5. Unknown.

  • 6. Crashing executable.

  • 7. Executable version.

  • 8. Executable timestamp.

  • 9. Crashing DLL. This will be the same as 6 if the crash is in the .exe.

  • 10. DLL version. This will be the same as 7 if the crash is in the .exe.

  • 11. DLL timestamp. This will be the same as 8 if the crash is in the .exe.

  • 12. Exception code.

  • 13. Fault offset.

  • 14. Class. This may or may not be present

Information Type is normally “APPCRASH”. In this case we’re interested in tags 9, 12 and 13.

If Information Type is “BEX”, the data is different:


  • 1. Timestamp.

  • 2. Number of data items.

  • 3. Information Type.

  • 4. Information Status.

  • 5. Unknown.

  • 6. Crashing executable.

  • 7. Executable version.

  • 8. Executable timestamp.

  • 9. Crashing DLL. This will be the same as 6 if the crash is in the .exe.

  • 10. DLL version. This will be the same as 7 if the crash is in the .exe.

  • 11. DLL timestamp. This will be the same as 8 if the crash is in the .exe.

  • 12. Fault offset.

  • 13. Exception code.

  • 14. Class. This may or may not be present

Note that the order of the fault offset and exception code has been reversed compared to APPCRASH.

Of these tags we’re interested in tags 9, 12 and 13.

If we want to version the crashing DLL we also need tags 10 and 11.

Application Error

The event log data for an Application Error event contains many fields that we don’t need if we’re just investigating a crash address. Each event starts with an <Event> tag and ends with an </Event> tag.

We need to correctly identify the event. Inside the event is a <System> tag which contains a tag with an attribute “Provider Name” set to “Application Error”.

Once the event is identified we need to find the <EventData> tag inside the event. The <EventData> contains at least 12 <Data> tags, some of which may not be present, or which may be empty. These tags are present:


  • 1. Crashing executable.

  • 2. Executable version.

  • 3. Executable timestamp.

  • 4. Crashing DLL. This will be the same as 1 if the crash is in the .exe.

  • 5. DLL version. This will be the same as 2 if the crash is in the .exe.

  • 6. DLL timestamp. This will be the same as 3 if the crash is in the .exe.

  • 7. Exception code.

  • 8. Fault offset.

  • 9. Process id.

  • 10. Application start timestamp.

  • 11. Application path.

  • 12. Module path.

Of these tags we’re interested in tags 7, 8 and 12.

If we want to version the crashing DLL we also need tags 5 and 6. If 12 isn’t available, use 4.

Removing the drudgery

The previous two sections have described which fields to extract data from. If you’re doing this manually this is tedious and error prone. You have to select the correct values from the correct fields and then use another application to turn them into a symbol, filename and line number. Our tools DbgHelpBrowser and MapFileBrowser are designed to take a crash offset inside a DLL and turn it into a human readable symbol, filename and line number. But that still requires you to do the hard work of fishing the correct data out of the XML dump.

Now there is a better way, we’ve added an extra option to these tools that allows you to paste the entire XML data from a crash event and the tool then extracts the data it needs to show you the symbol, filename and line number.

DbgHelpBrowser

Load the crashing exe (or DLL) into DbgHelpBrowser. This will cause the symbols to be loaded for the DLL (assuming symbols have been created and can be found). We’re not covering versioning the DLL as most likely you will have your own methods for this.

Choose the option Find Symbol from Event Viewer XML crash log… on the Query menu. The Event Viewer Crash Data dialog is displayed.


Paste the XML data into the dialog and click OK.


The main display will select the appropriate symbol in the main grid and display the relevant symbol, filename, line number and source code in the source code viewer below.


MapFileBrowser

Load the MAP file for the crashing exe (or DLL) into MapFileBrowser. We’re not covering versioning the DLL as most likely you will have your own methods for this.

Choose the option Find Symbol from Event Viewer XML crash log… on the Query menu. The Event Viewer Crash Data dialog is displayed.


Paste the XML data into the dialog and click OK.


The main display will select the appropriate symbol in the main grid and display the relevant symbol, filename, line number and source code in the source code viewer below.


Conclusion

Windows Event Logs can be hard to read and error prone to use. However when paired with suitable tools you can quickly and easily turn event log crashes into useful symbol, filename and line number information to inform your debugging efforts.

Stdout redirection and capture

By , July 11, 2019 10:34 am

We were recently asked if Memory Validator could handle monitoring a program that took it’s input from a file and wrote its’ output to a file. As shown in the following example.

redirect.exe < input.txt > output.txt

Our tools could handle this, but it wasn’t obvious that they could. Also for interactive use there was no easy way to do this via the launch dialog, unless you were using Coverage Validator. We had to make changes to most of our tools so that they could do what Coverage Validator could do. All tools had a new diagnostic sub-tab added so that stdout data that was captured (an option) could be viewed.

In this article I’m going to explain how to launch a program that reads from stdin and/or writes to stdout using our tools. Although I’m talking about Memory Validator, these techniques apply to all our Validator tools.

There are four ways of doing this.

  1. Start your program from a batch file, putting the redirect of stdin and stdout in the batch file.
  2. Start your program from the launch dialog/wizard, specifying the input and output files on the launch wizard.
  3. Start your program from the command line, specifying the input and output files on the command line.
  4. Use the Memory Validator API to start Memory Validator from inside the target program.

Batch File

Using a batch file to do this is easy. Simply create your batch file in the form shown in the example below.

e:\redirect\release\redirect.exe < e:\test\input.txt > e:\test\output.txt

Save the batch file with a known name. In this example I’ll save the batch file in e:\test\redirectTest.bat. Then launch the batch file from Memory Validator (or another Validator tool). The first program launched by the batch file will be monitored.


Launch Dialog/Wizard

We modified the launch wizard and the launch dialog to include fields for an optional input file and an optional output file. We also added an option that would capture stdout so that you could view the output on the diagnostic tab.

This example shows the program testAppTheReadsFromStdinAndWritesToStdout.exe being launched, reading from e:\test\input.txt, and writing to reading from e:\test\output.txt.


The stdout capture checkbox has been selected. This will mean a copy of stdout will be captured and displayed on the diagnostic sub-tab stdout.


Command Line

For command line operation we need two new options -stdin and -stdout, each of which takes a filename for an argument.

There are two additional arguments that you can supply to tell Memory validator to ignore missing input files and missing output files: -ignoreMissingStdin and -ignoreMissingStdout.

memoryValidator.exe -program e:\redirect\release\redirect.exe 
                    -directory e:\redirect\release 
                    -stdin e:\test\input.txt 
                    -stdout e:\test\output.txt
                    -showErrorsWithMessageBox

API

You can use the Memory Validator API to start Memory Validator each time the target program is run. In that case just running the following command on the command prompt would cause Memory Validator monitor the target program redirect.exe.

e:\redirect\release\redirect.exe < e:\test\input.txt > e:\test\output.txt

To use the Memory Validator API with a particular application, the following steps outline the minimum steps required.

  1. Link to svlMemoryValidatorStub.lib (_x64.lib for x64)
  2. Link to svlMemoryValidatorStubLib.lib (_x64.lib for x64)
  3. #include “stublib.h”
  4. call startProfiler(); at the start of your program
  5. See documentation in the help file for more details.

For other Validator tools the library names will change. See the documentation (topic API) for the particular tool for details.

Conclusion

There are four ways you can work with stdin and stdout with our Validator tools.

You can work with batch files, launch interactively using the launch dialog/wizard, work from the command line, and use the Validator API.

Thread naming

By , June 19, 2019 7:21 pm

Multi-threading is becoming quite common these days. It’s a useful way to provide a responsive user interface while performing work at the same time. Our tools report data per thread where that is warranted (per-thread code coverage doesn’t seem to be a thing – no one has requested it in 17 years). In this article I’m going to discuss thread naming, OS support for thread naming, and additional support for thread naming that our tools automatically provide.

The Default Thread Display

Threads are represented by a thread handle and a thread id. Typically the thread id is what will be used to represent the thread in a user interface reporting thread related data. Thread ids are numeric. For example: 341. For trivial programs you can often infer which thread is which by looking at the data allocated on each thread. However that doesn’t scale very well to more complex applications. You can end up with thread displays like this:


Microsoft Thread Naming Exception

The WIN32 API does not provide any functions to allow you to name a thread. To handle this oversight, Microsoft use a convention that allows a program to communicate a thread name with it’s debugger. This is done via means of an exception that the program throws (and then catches to prevent it’s propagation terminating the application). The debugger also catches this exception and with the help of ReadProcessMemory() can retrieve the exception name from the program. Here’s how that works.

In your program

The exception code that identifies a thread naming exception is 0x406D1388. To pass the thread name to the debugger you need to create a struct of type THREADNAME_INFO (definition shown int the code sample), populate it with the appropriate data then raise an exception with the specified exception code. You’ll need to use SEH __try/__except to surround the RaiseException() call to prevent crashing the program.

typedef struct tagTHREADNAME_INFO
{
	DWORD	dwType;		// must be 0x1000
	LPCSTR	szName;		// pointer to name (in user addr space) buffer must be 8 chars + 1 terminator
	DWORD	dwThreadID;	// thread ID (-1 == caller thread)
	DWORD	dwFlags;	// reserved for future use, must be zero
} THREADNAME_INFO;

#define MS_VC_EXCEPTION 0x406D1388

void nameThread(const DWORD	threadId,
                const char	*name)
{
	// You can name your threads by using the following code. 
	// Thread Validator will intercept the exception and pass it along (so if you are also running
	// under a debugger the debugger will also see the exception and read the thread name

	// NOTE: this is for 'unmanaged' C++ ONLY!

	#define BUFFER_LEN		16

	THREADNAME_INFO	ThreadInfo;
	char		szSafeThreadName[BUFFER_LEN];	// buffer can be any size, just make sure it is large enough!
	
	memset(szSafeThreadName, 0, sizeof(szSafeThreadName));	// ensure all characters are NULL before
	strncpy(szSafeThreadName, name, BUFFER_LEN - 1);	// copying name
	//szSafeThreadName[BUFFER_LEN - 1] = '\0';

	ThreadInfo.dwType = 0x1000;
	ThreadInfo.szName = szSafeThreadName;
	ThreadInfo.dwThreadID = threadId;
	ThreadInfo.dwFlags = 0;

	__try
	{
		RaiseException(MS_VC_EXCEPTION, 0, 
                               sizeof(ThreadInfo) / sizeof(DWORD_PTR), 
                               (DWORD_PTR *)&ThreadInfo); 
	}
	__except(EXCEPTION_EXECUTE_HANDLER)
	{
		// do nothing, just catch the exception so that you don't terminate the application
	}
}

In the debugger

The THREADNAME_INFO struct address is communicated to the debugger in event.u.Exception.ExceptionRecord.ExceptionInformation[0]. Cast this pointer to a THREADNAME_INFO *, check the number of parameters are correct (4 on x86, 3 on x64), check the dwType and dwFlags are correct, then use the pointer in the szName field. This is a pointer inside the other process, which means you can’t read it directly you need to use ReadProcessMemory(). Once the debugger continues and this event goes out of scope, the ability to read the thread name no longer exists. If you want to read the thread name you must read it immediately, storing it for future use.

	if ((de->dwDebugEventCode == EXCEPTION_DEBUG_EVENT) &&
	    (de->u.Exception.ExceptionRecord.ExceptionCode == SVL_THREAD_NAMING_EXCEPTION))
	{
#ifdef _WIN64
		if (de->u.Exception.ExceptionRecord.NumberParameters == 3)
#else	//#ifdef _WIN64
		if (de->u.Exception.ExceptionRecord.NumberParameters == 4)
#endif	//#ifdef _WIN64
		{
			THREADNAME_INFO	*tni;

			tni = (THREADNAME_INFO *)&de->u.Exception.ExceptionRecord.ExceptionInformation[0];
			if (tni->dwType == 4096 &&
			    tni->dwFlags == 0)
			{
				void	*ptr = (void *)tni->szName;
				DWORD	threadId = tni->dwThreadID;

				if (ptr != NULL)
				{
					char	buffer[1000];
					int		bRet;
					SIZE_T	dwRead = 0;

					memset(buffer, 0, 1000);
					bRet = ReadProcessMemory(hProcess,
								 ptr,
								 buffer,
								 1000 - 1,	// remove one so there is always a terminating zero
								 &dwRead);

					if (buffer[0] != '\0')
						setThreadName(threadId, buffer);
				}
			}
		}
	}

Our tools support intercepting this mechanism so that we can name your threads if you use this mechanism.

The problem with this mechanism is that it puts the onus for naming the threads on the creator of the application. If that involves 3rd party components that spawn their own threads, and OS threads then almost certainly not all threads will be named, which results in some threads having a name and other threads being represented by a thread id.


CreateThread() And Symbols

In an attempt to automatically name threads to provide useful naming for threads without having to rely on the author of a program using thread naming exceptions we started monitoring CreateThread() to get the thread start address, then resolve that address into a symbol name (assuming debugging symbols are available) and use that symbol name to represent the thread. After some tests (done with our own tools – which are heavily multithreaded – dog-fooding) we concluded this was a useful way to name threads.

However this doesn’t solve the problem as many threads are now named _threadstartex().


_beginthread(), _beginthreadex() And Symbols

The problem with the CreateThread() approach is that any threads created by calls to the C runtime functions _beginthread() or _beginthreadex() will result in the thread being called _threadstart() or _threadstartex(). This is because these functions pass their own thread function to CreateThread(). The way to solve this problem is to also monitor _beginthread() and _beginthreadex() functions to get the thread start address, then resolve that address into a symbol name.


SetThreadDescription

Starting with Windows 10, a name can be associated with a thread handle via the API SetThreadDescription(). The name associated with a thread handle can be retrieved using GetThreadDescription(). We use these functions to provide names for your threads if you have used these functions.

Un-named Threads

Some threads still end up without names – typically these are transient threads created by the OS to do a short amount of work. If the call to create the thread was internal to Kernel32.dll then CreateThread() will not be called by an IAT call and will not be monitored, resulting in the thread not getting named automatically. This isn’t ideal, but ultimately you don’t control these threads, which means you can’t affect the data reported by these threads, so it’s not that important.

Thread Naming Priority

We’ve made these changes to all our Validator tools and updated the user interfaces to represent threads by both thread id and thread name.

If a thread is named by a thread naming exception, SetThreadDescription(), or via a Software Verify API call that name will take precedence over any automatically named thread. This is useful in cases where the same function is used by many threads – you can if you wish give each thread a unique name to prevent multiple threads having the same name.

Here are some of the displays that have benefited from named threads.

Bug Validator


Memory Validator


Performance Validator


Thread Validator



Support for Menus in Visual Studio 2017, 2015, 2013, 2012, 2010

By , May 12, 2018 10:10 am

Visual Studio menu support

We’ve just added support for integrating tool menus into Visual Studio. This integration supports most of our tools. The menu only shows tools that are installed. Supported versions of Visual Studio are 2017, 2015, 2013, 2012, 2010.

Here’s what it looks like in Visual Studio 2017 (if you have all of our tools installed).


This means that most of our tools can be launched from within Visual Studio. If a solution is loaded, the tool you select will be started with appropriate options to process the startup project that is selected. For tools that support both x64 and x86 installation, on x86 platforms the x86 tool will be selected, on x64 platforms the x64 tool will be selected (if it was installed), otherwise the x86 tool will be selected.

For Thread Lock Checker and OutputDebugString checker the tools are given a list of the source files in the project. The source files are scanned for errors.

For DbgHelp Browser and PE File Browser the tools are given the executable that the project builds. The tools then display data about the executable and/or it’s debug information as appropriate.

For the various C++ Validators and .Net Validators, VM Validator, Exception Tracer and Page Fault Monitor, the tools are passed the same information that the Visual Studio debugger uses to start a program (program name, program arguments, startup directory). The tools then start the target program and monitor data according to which tool has been launched.

Exception Tracer

The eagle eyed will have noticed that in this list is the name of a tool that isn’t on our website (yet). Exception Tracer. This is a tool that has been in genesis for some time. It has transformed from an experimental debugger, into an experimental code coverage tool and now it’s an interesting view on the internal life of an application, from the point of view of the exceptions it throws. We’ve used Exception Tracer to analyse some unusual bugs that were almost impossible to analyse using a traditional debugger. We’ll be releasing this tool later this year. It will add to our growing list of free debugging and analysis tools on our website.

Cleaning up Visual Studio

By , March 19, 2018 12:33 pm

Why would you ever need a tool to clean up Visual Studio? Doesn’t Visual Studio have a built in “clean” mode that cleans your build directories?

Yes, it does, but that only cleans the file created by compiling and linking your software. Visual Studio also creates other files. How do you get rid of those files? Look them up, then search for them manually? That’s going to be time consuming.

Why would you even want to delete these extra files?

We’ve also seen Visual Studio crash from time to time. Sometimes it’s a plugin or extension gone astray, but other times it’s when you do something harmless like load a project or workspace or solution. What we found was that in many cases, just deleting the extra files would prevent future crashes.

So we wrote Visual Studio Project Cleaner to do that. The first version of the tool was written to work with Visual Studio 6, but we’ve updated it to also cleanup so of the files created by the newer versions of Visual Studio, Visual Lint and Visual Studio Project Fixer.


What type of file can we clean?

  • aps. Last Resource editor state.
  • bak. Backup file.
  • bsc. Source browser database.
  • dbg. Debugging information.
  • exp. Exports file.
  • idb. Minimum rebuild linker file.
  • ilk. Incremental linker file.
  • ipch. Incremental precompiled header file.
  • lastbuildstate. Build helper.
  • log. Build log.
  • map. Linker MAP file.
  • ncb. Intellisense database.
  • obj. Object file.
  • opt. Visual Studio options.
  • pch. Pre-compiled header file.
  • res. Compiled resource script.
  • sbr. Source browser information.
  • sdf. Intellisense database.
  • tlog. Depdenency file information.

We can also clean the following file types (these are disabled by default).

  • lib. Library file.
  • dll. Dynamic Link Library.
  • exe. Executable.
  • pdb. Program database.
  • old. Visual Studio upgrade files.
  • Visual Lint files in .visualint directories.
  • Visual Studio Project Fixer backup files (*.vcxproj-YYY-MM-DD-HH-MM-SS)


Clean your Visual Studio

We’ve been using Visual Studio Project Cleaner for years to clean unwanted files. It turns a tedious job into a simple and easy job.

Clean your files today with Visual Studio Project Cleaner.

Panorama Theme by Themocracy