Rss Feed
Tweeter button
Facebook button
Technorati button
Reddit button
Myspace button
Linkedin button
Webonews button
Delicious button
Digg button
Flickr button
Stumbleupon button
Newsvine button

Thread naming

By , June 19, 2019 7:21 pm

Multi-threading is becoming quite common these days. It’s a useful way to provide a responsive user interface while performing work at the same time. Our tools report data per thread where that is warranted (per-thread code coverage doesn’t seem to be a thing – no one has requested it in 17 years). In this article I’m going to discuss thread naming, OS support for thread naming, and additional support for thread naming that our tools automatically provide.

The Default Thread Display

Threads are represented by a thread handle and a thread id. Typically the thread id is what will be used to represent the thread in a user interface reporting thread related data. Thread ids are numeric. For example: 341. For trivial programs you can often infer which thread is which by looking at the data allocated on each thread. However that doesn’t scale very well to more complex applications. You can end up with thread displays like this:


Microsoft Thread Naming Exception

The WIN32 API does not provide any functions to allow you to name a thread. To handle this oversight, Microsoft use a convention that allows a program to communicate a thread name with it’s debugger. This is done via means of an exception that the program throws (and then catches to prevent it’s propagation terminating the application). The debugger also catches this exception and with the help of ReadProcessMemory() can retrieve the exception name from the program. Here’s how that works.

In your program

The exception code that identifies a thread naming exception is 0x406D1388. To pass the thread name to the debugger you need to create a struct of type THREADNAME_INFO (definition shown int the code sample), populate it with the appropriate data then raise an exception with the specified exception code. You’ll need to use SEH __try/__except to surround the RaiseException() call to prevent crashing the program.

void nameThread(const DWORD	threadId,
                const char	*name)
{
	// You can name your threads by using the following code. 
	// Thread Validator will intercept the exception and pass it along (so if you are also running
	// under a debugger the debugger will also see the exception and read the thread name

	// NOTE: this is for 'unmanaged' C++ ONLY!

	#define MS_VC_EXCEPTION 0x406D1388
	#define BUFFER_LEN		16

	typedef struct tagTHREADNAME_INFO
	{
		DWORD	dwType;		// must be 0x1000
		LPCSTR	szName;		// pointer to name (in user addr space) buffer must be 8 chars + 1 terminator
		DWORD	dwThreadID;	// thread ID (-1 == caller thread)
		DWORD	dwFlags;	// reserved for future use, must be zero
	} THREADNAME_INFO;

	THREADNAME_INFO	ThreadInfo;
	char		szSafeThreadName[BUFFER_LEN];	// buffer can be any size, just make sure it is large enough!
	
	memset(szSafeThreadName, 0, sizeof(szSafeThreadName));	// ensure all characters are NULL before
	strncpy(szSafeThreadName, name, BUFFER_LEN - 1);	// copying name
	//szSafeThreadName[BUFFER_LEN - 1] = '\0';

	ThreadInfo.dwType = 0x1000;
	ThreadInfo.szName = szSafeThreadName;
	ThreadInfo.dwThreadID = threadId;
	ThreadInfo.dwFlags = 0;

	__try
	{
		RaiseException(MS_VC_EXCEPTION, 0, 
                               sizeof(ThreadInfo) / sizeof(DWORD_PTR), 
                               (DWORD_PTR *)&ThreadInfo); 
	}
	__except(EXCEPTION_EXECUTE_HANDLER)
	{
		// do nothing, just catch the exception so that you don't terminate the application
	}
}

In the debugger

The thread name is communicated to the debugger in event.u.Exception.ExceptionRecord.ExceptionInformation[1]. This pointer can be read using ReadProcessMemory(). Once the debugger continues and this event goes out of scope, the ability to read the thread name no longer exists. If you want to read the thread name you must read it immediately, storing it for future use.

	if ((de->dwDebugEventCode == EXCEPTION_DEBUG_EVENT) &&
	    (de->u.Exception.ExceptionRecord.ExceptionCode == SVL_THREAD_NAMING_EXCEPTION))
	{
		char	buffer[1000];
		int		bRet;
		SIZE_T	dwRead = 0;

		memset(buffer, 0, 1000);
		bRet = ReadProcessMemory(hProcess,
					 (void *)event.u.Exception.ExceptionRecord.ExceptionInformation[1],
					 buffer,
					 1000 - 1,	// remove one so there is always a terminating zero
					 &dwRead);

		setThreadName((DWORD)event.u.Exception.ExceptionRecord.ExceptionInformation[2], buffer);
	}

Our tools support intercepting this mechanism so that we can name your threads if you use this mechanism.

The problem with this mechanism is that it puts the onus for naming the threads on the creator of the application. If that involves 3rd party components that spawn their own threads, and OS threads then almost certainly not all threads will be named, which results in some threads having a name and other threads being represented by a thread id.


CreateThread() And Symbols

In an attempt to automatically name threads to provide useful naming for threads without having to rely on the author of a program using thread naming exceptions we started monitoring CreateThread() to get the thread start address, then resolve that address into a symbol name (assuming debugging symbols are available) and use that symbol name to represent the thread. After some tests (done with our own tools – which are heavily multithreaded – dog-fooding) we concluded this was a useful way to name threads.

However this doesn’t solve the problem as many threads are now named _threadstartex().


_beginthread(), _beginthreadex() And Symbols

The problem with the CreateThread() approach is that any threads created by calls to the C runtime functions _beginthread() or _beginthreadex() will result in the thread being called _threadstart() or _threadstartex(). This is because these functions pass their own thread function to CreateThread(). The way to solve this problem is to also monitor _beginthread() and _beginthreadex() functions to get the thread start address, then resolve that address into a symbol name.


Un-named Threads

Some threads still end up without names – typically these are transient threads created by the OS to do a short amount of work. If the call to create the thread was internal to Kernel32.dll then CreateThread() will not be called by an IAT call and will not be monitored, resulting in the thread not getting named automatically. This isn’t ideal, but ultimately you don’t control these threads, which means you can’t affect the data reported by these threads, so it’s not that important.

Thread Naming Priority

We’ve made these changes to all our Validator tools and updated the user interfaces to represent threads by both thread id and thread name.

If a thread is named by a thread naming exception or via a Software Verify API call that name will take precedence over any automatically named thread. This is useful in cases where the same function is used by many threads – you can if you wish give each thread a unique name to prevent multiple threads having the same name.

Here are some of the displays that have benefited from named threads.

Bug Validator


Memory Validator


Performance Validator


Thread Validator



Business of Software Europe 2019

By , April 18, 2019 11:00 am

Last week I attended my the Business of Software Europe conference. This year the conference returns to Cambridge after a few years in Dublin and London.

This year the conference logo is the black squirrel, which is apparently a genetic mutation local to Cambridge.

As usual, an excellent conference with a wide range of topics and an interesting mix of speakers. I’m not going to go into detail on the talks, other folks will have already written them up in more detail than my sparse notes will allow. If you’re interested in high level strategic thought about how to run a software business, this is the conference for you.

But I did make a note of every book that a speaker or attendee mentioned.

Let It Go, by Dame Stephanie ‘Steve’ Shirley CH. This book was provided to all attendees by the conference.

Art of Profitability, by Adrian Slywotzky

Thinking, Fast and Slow, by Daniel Kahneman

The Games People Play, by Eric Berne

Rest, by Alex Soojung-Kim Pang

What Got You Here Wont Get You There, by Marshall Goldsmith

Skin In The Game, by Nassim Nicholas Taleb

Let My People Go Surfing, by Yvon Chouinard

This Is Marketing, by Seth Godin

Clean Code, by Robert C. Martin

Building A Story Brand, by Donald Miller

11 Laws Of Showrunning, (PDF) by Javier Grillo-Marxuach . Also a podcast.

Powerful, by Patty McCord

The Art of Product Management, by Rich Mironov

What They Don’t Teach You At Harvard, by
Mark H. McCormack

Chimp Paradox, by Professor Steve Peters

The Manager’s Path, Camille Fournier

Radical Candor, by Kim Scott

Badass, by Kathy Sierra

Purple Cow, by Seth Godin

Film: Free Solo

Video: Steve Jobs marketing talk

Metcalfe’s Law, as the number of people rises, the number of connections gets out of control.

Conway’s Law, your designs are constrained by your communication structures. This graphic captures it all. Note the Microsoft graphic.

There’s more than one way to leak a GDI object

By , September 4, 2018 10:05 am

Working with GDI in Windows, whether you’re using Win32 calls or MFC, you’re concerned with pens, brushs, fonts, bitmaps and regions for drawing. You may also be concerned with Palettes, although with our full colour displays these days working with palettes is something of a rarity.

The typical way to work with a GDI object, for example, a pen, is shown below. Create the pen, select it into the DC, do the drawing, select the original object back into the DC, delete the pen.

void CtestGDISelectObjectDlg::drawSomethingWithARedPen(HDC hDC)
{
	HPEN		hPen;
	HPEN		hPenOld;
	HGDIOBJ		retVal;

	hPen = ::CreatePen(PS_SOLID, 0, RGB(255, 0, 0));

	hPenOld = (HPEN)SelectObject(hDC, hPen);

	doDrawing1(hDC);

	retVal = SelectObject(hDC, hPenOld);

	DeleteObject(hPen);
}

Leaking GDI objects, type 1

The most common way people leak GDI objects is because they simply forget to delete them. Here’s the previous example, modified to leak the pen. (Don’t do this!).

void CtestGDISelectObjectDlg::drawSomethingWithARedPen(HDC hDC)
{
	HPEN		hPen;
	HPEN		hPenOld;
	HGDIOBJ		retVal;

	hPen = ::CreatePen(PS_SOLID, 0, RGB(255, 0, 0));

	hPenOld = (HPEN)SelectObject(hDC, hPen);

	doDrawing1(hDC);

	retVal = SelectObject(hDC, hPenOld);
}

The author of the code forgot to delete the pen using DeleteObject(hPen); This leak looks like this in Memory Validator.


Leaking GDI objects, type 2

There is another way to leak GDI objects, even when you think you’ve deleted all the objects you created. Take a look at this code. Does it leak?

	HPEN		hPen1;
	HPEN		hPen2;
	HPEN		hPenOld;
	HGDIOBJ		retVal;

	hPen1 = ::CreatePen(PS_SOLID, 0, RGB(255, 0, 0));
	hPen2 = ::CreatePen(PS_DASHDOTDOT, 0, RGB(255, 0, 0));

	hPenOld = (HPEN)SelectObject(hDC, hPen1);

	doDrawing1(hDC);

	retVal = SelectObject(hDC, hPen2);
	
	doDrawing2(hDC);

	DeleteObject(hPen1);
	DeleteObject(hPen2);

A quick examination shows that two pens are created, some work is done with each pen, then the pens are deleted.

No leaks, right?

Wrong! hPen2 was selected into the DC for use with doDrawing2() but was not deselected from the DC prior to DeleteObject(hPen2);. This means that the call to DeleteObject() will fail as the pen is still in use. hPen2 has been leaked.

Memory Validator can detect this (as of V7.38, released today). Here’s what that looks like:


Expanding the source and you can easily see the failed DeleteObject() call and the SelectObject() call that mean the object was still in use.


Here’s what non-leaking the code should look like:

	HPEN		hPen1;
	HPEN		hPen2;
	HPEN		hPenOld;
	HGDIOBJ		retVal;

	hPen1 = ::CreatePen(PS_SOLID, 0, RGB(255, 0, 0));
	hPen2 = ::CreatePen(PS_DASHDOTDOT, 0, RGB(255, 0, 0));

	hPenOld = (HPEN)SelectObject(hDC, hPen1);

	doDrawing1(hDC);

	retVal = SelectObject(hDC, hPen2);
	
	doDrawing2(hDC);

	SelectObject(hDC, hPenOld);

	DeleteObject(hPen1);
	DeleteObject(hPen2);

To ensure your object is not still selected into the DC you can select the value that was returned to you by the first call (hPenOld in the code above), or you can select a stock object into the DC instead. For example:

	SelectObject(hDC, GetStockObject(BLACK_PEN));

Conclusion

When working with GDI objects you need to keep track of two things:


  1. Creation and Deletion of GDI objects. For every pen, brush, font, bitmap, palette that you create you must delete those objects when you are finished with them. Delete objects using DeleteObject().

  2. You need to ensure that none of the objects that you create are selected into a DC when you try to delete them.

Detecting memory leaks in Visual Test unit tests

By , June 6, 2018 5:40 pm

Introduction

We recently had a request asking if C++ Memory Validator could detect memory leaks in unit tests managed by Microsoft’s Visual Test and Visual Test Explorer. They told us what they’d tried to do and that it had failed. Our response was to find out what was failing, fix it and then describe what you need to do to use our tools with Visual Test. That’s what this article is about.

There were a few bugs specific to working with Visual Test, plus a very novel environment variable data corruption that can only happen in very unusual circumstances (you almost certainly would not hit these in normal usage of our tools). We fixed these tools, then experimented with several ways of working with Visual Test. We’re only going to talk about C++ Memory Validator here, but this article also applies to our Coverage, Performance and Thread tools.

You can run Visual Test from the command line, and also via Visual Test Explorer, which is a component of Visual Studio.

Monitoring Visual Test from the command line

The full details of how to work with Visual Test from the command line are documented in this article from Microsoft.

Using this information we know that we need to launch vstest.console.exe and pass the unit test DLL to that as an argument. For example:

vstest.console.exe unitTest.dll 

Using this information we can launch vstest.console.exe from C++ Memory Validator and pass it the appropriate DLL. You can set the startup directory to whatever you like. We’ve chose to set it to the same directory as the unit test DLL.

Unit Test Code

namespace UnitTest1
{		
	TEST_CLASS(UnitTest1)
	{
	public:
		
		TEST_METHOD(TestMethod1)
		{
			// pretend this is a real unit test, 
			// exercising a real target class/function
			// and that it leaks some memory

			char *ptr;

			ptr = new char[123];
			strcpy_s(ptr, 123, "Excellent Adventure");

			ptr = (char *)malloc(456);
			strcpy_s(ptr, 456, "Bogus Journey");

			ptr = (char *)malloc(789);
			strcpy_s(ptr, 789, "Face the Music");

			Assert::AreNotEqual("Bill", "Ted");
		}

	};
}

You’ll notice that you don’t need to specify the TEST_MODULE_INITIALIZE(moduleInit) function, or the TEST_MODULE_CLEANUP(moduleCleanup) function. You only need to specify the unit tests you want tested. C++ Memory Validator does the rest.

Monitoring Visual Test Explorer

When working with Visual Test Explorer, Visual Test is launched from Dev Studio, and then loads the unit tests for testing from a DLL. Visual Test then stays running in the background and does not shutdown. This means the “program has ended” signal that C++ Memory Validator needs doesn’t get sent. It also means that subsequent tests run with Visual Test Explorer won’t cause Visual Test to startup, meaning that the “program has started” signal that C++ Memory Validator needs doesn’t get sent.

To get around these problems we need to use the NT Service API (see help file, or online documentation for details) to contact C++ Memory Validator at the start of the unit tests, and also at the end of the unit tests. We do that using the TEST_MODULE_INITIALIZE(moduleInit) function, or the TEST_MODULE_CLEANUP(moduleCleanup) functions.

Functions

We use svlMVStub_LoadMemoryValidator() to load C++ Memory Validator into the unit test, then svlMVStub_StartMemoryValidator() to start it monitoring the unit test and communicating with the user interface. At the end of the tests we use svlMVStub_UnloadMemoryValidatorAndTerminateProcess(1000) to shutdown C++ Memory Validator and set a thread running that will after a delay of 1000ms terminate the vstest.executionengine.x86.exe (Visual Test) process. You may need to experiment with this delay on your machine.

Header Files

We need to include two header files from C++ Memory Validator. svlMVStubService.h and svlServiceError.h. You’ll find these in the svlMVStubService folder in the C++ Memory Validator install directory.

Libraries

We also need to link to svlMVStubService.lib (32 bit builds) or svlMVStubService_x64.lib (64 bit builds). You’ll find versions of these libraries in the svlMVStubService folder (one library per version of Visual Studio).

Unit Test Code

#include "svlMVStubService.h"
#include "svlServiceError.h"

namespace UnitTest1
{		
	TEST_MODULE_INITIALIZE(moduleInit)
	{
		SVL_SERVICE_ERROR	sse;

		sse = svlMVStub_LoadMemoryValidator();

		sse = svlMVStub_StartMemoryValidator();
	}

	TEST_MODULE_CLEANUP(moduleCleanup)
	{
		SVL_SERVICE_ERROR	sse;
		
		sse = svlMVStub_UnloadMemoryValidatorAndTerminateProcess(1000);
	}

	TEST_CLASS(UnitTest1)
	{
	public:
		
		TEST_METHOD(TestMethod1)
		{
			// pretend this is a real unit test, 
			// exercising a real target class/function
			// and that it leaks some memory

			char *ptr;

			ptr = new char[123];
			strcpy_s(ptr, 123, "Excellent Adventure");

			ptr = (char *)malloc(456);
			strcpy_s(ptr, 456, "Bogus Journey");

			ptr = (char *)malloc(789);
			strcpy_s(ptr, 789, "Face the Music");

			Assert::AreNotEqual("Bill", "Ted");
		}

	};
}

Having reworked the unit tests to support the NT Service API, we now need to launch devenv.exe from C++ Memory Validator, but with instructions to ignore devenv.exe and monitor Visual Test (vstest.executionengine.x86.exe). We do that from the launch dialog/wizard.

First choose devenv.exe to monitor using the Browse… button next to the Application to launch field. In this example we chose C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\devenv.exe


Next we need to setup the applications to monitor. Click the Edit… button next to the Application to monitor field. The applications to monitor dialog is displayed.


Choose devenv.exe using the Browse… button next to the Application to launch field.

Click the Add… button and add the vstest.executionenginex86.exe that corresponds with the devenv.exe you selected. In this example we chose C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstestexecutionengine.x86.exe. Be sure to choose the correct one – there are several with similar names to choose from.


Click OK to accept the application to monitor.


Click OK to accept the new definitions.

Now on the launch dialog, change the Application to monitor combo to select the vstest.execution.x86.exe entry.

Enter the name of your unit test dll(s) in the Arguments field, separated by spaces. If you specify the directory containing the DLLs as the startup directory, you can specify the DLL names without paths.

Running Your Tests

Now that we’ve setup C++ Memory Validator to monitor Visual Test when started from devenv, let’s get to work.

To start devenv, click the Go! button. Devenv will start. Load the solution that contains your unit tests. When you choose Run Selected Tests or Debug Selected Tests from Visual Test Explorer, the tests will run and will be monitored by C++ Memory Validator at the same time.

Business of Software Europe 2018

By , May 26, 2018 9:41 am

Last week I attended the Business of Software Europe 2018 conference, in London, United Kingdom. It’s a great conference, for people building sustainable software businesses. If you’re trying to create the next Instagram, this is not the conference for you. If you are trying to make your own mark on the world, carve out a future for yourself, this is probably the place to be.

I’m not going to try to write up a series of actionable points taken from the conference. That would take too long, and anyway, those are points relevant to me, which may not work for you. All I can say is that it’s a great conference, with a good community of attendees and no Jupiter class egos. You do usually need a pair of strong sunglasses to deal with Mark Littlewood’s shirts, but it’s a small price to pay. More women and more non-white people each year. Diversity is a good thing. I wish it wasn’t over so soon.

The following books were recommended by speakers at Business of Software Europe 2018

Collective Genius, by Linda Hill
Deep Work, by Cal Newport
Drive, by Dan Pink
The Progress Principle, by Teresa Amabile and Steven Kramer
Failed It!, by Erik Kessels
Corporate Startup, by Tendayi Viki
The First Game of Business, Jack Stack
The Coaching Habit, by Michael Bungay Stamer
Work Rules, by Laszlo Bock
Turn the Ship Around!, by David Marquet

The following resources were recommended by speakers at Business of Software Europe 2018

Manager Talks podcast
Product Culture newsletter
Project Aristotle Building better teams through psychological safety

Support for Menus in Visual Studio 2017, 2015, 2013, 2012, 2010

By , May 12, 2018 10:10 am

Visual Studio menu support

We’ve just added support for integrating tool menus into Visual Studio. This integration supports most of our tools. The menu only shows tools that are installed. Supported versions of Visual Studio are 2017, 2015, 2013, 2012, 2010.

Here’s what it looks like in Visual Studio 2017 (if you have all of our tools installed).


This means that most of our tools can be launched from within Visual Studio. If a solution is loaded, the tool you select will be started with appropriate options to process the startup project that is selected. For tools that support both x64 and x86 installation, on x86 platforms the x86 tool will be selected, on x64 platforms the x64 tool will be selected (if it was installed), otherwise the x86 tool will be selected.

For Thread Lock Checker and OutputDebugString checker the tools are given a list of the source files in the project. The source files are scanned for errors.

For DbgHelp Browser and PE File Browser the tools are given the executable that the project builds. The tools then display data about the executable and/or it’s debug information as appropriate.

For the various C++ Validators and .Net Validators, VM Validator, Exception Tracer and Page Fault Monitor, the tools are passed the same information that the Visual Studio debugger uses to start a program (program name, program arguments, startup directory). The tools then start the target program and monitor data according to which tool has been launched.

Exception Tracer

The eagle eyed will have noticed that in this list is the name of a tool that isn’t on our website (yet). Exception Tracer. This is a tool that has been in genesis for some time. It has transformed from an experimental debugger, into an experimental code coverage tool and now it’s an interesting view on the internal life of an application, from the point of view of the exceptions it throws. We’ve used Exception Tracer to analyse some unusual bugs that were almost impossible to analyse using a traditional debugger. We’ll be releasing this tool later this year. It will add to our growing list of free debugging and analysis tools on our website.

Cleaning up Visual Studio

By , March 19, 2018 12:33 pm

Why would you ever need a tool to clean up Visual Studio? Doesn’t Visual Studio have a built in “clean” mode that cleans your build directories?

Yes, it does, but that only cleans the file created by compiling and linking your software. Visual Studio also creates other files. How do you get rid of those files? Look them up, then search for them manually? That’s going to be time consuming.

Why would you even want to delete these extra files?

We’ve also seen Visual Studio crash from time to time. Sometimes it’s a plugin or extension gone astray, but other times it’s when you do something harmless like load a project or workspace or solution. What we found was that in many cases, just deleting the extra files would prevent future crashes.

So we wrote Visual Studio Project Cleaner to do that. The first version of the tool was written to work with Visual Studio 6, but we’ve updated it to also cleanup so of the files created by the newer versions of Visual Studio, Visual Lint and Visual Studio Project Fixer.


What type of file can we clean?

  • aps. Last Resource editor state.
  • bak. Backup file.
  • bsc. Source browser database.
  • dbg. Debugging information.
  • exp. Exports file.
  • idb. Minimum rebuild linker file.
  • ilk. Incremental linker file.
  • ipch. Incremental precompiled header file.
  • lastbuildstate. Build helper.
  • log. Build log.
  • map. Linker MAP file.
  • ncb. Intellisense database.
  • obj. Object file.
  • opt. Visual Studio options.
  • pch. Pre-compiled header file.
  • res. Compiled resource script.
  • sbr. Source browser information.
  • sdf. Intellisense database.
  • tlog. Depdenency file information.

We can also clean the following file types (these are disabled by default).

  • lib. Library file.
  • dll. Dynamic Link Library.
  • exe. Executable.
  • pdb. Program database.
  • old. Visual Studio upgrade files.
  • Visual Lint files in .visualint directories.
  • Visual Studio Project Fixer backup files (*.vcxproj-YYY-MM-DD-HH-MM-SS)


Clean your Visual Studio

We’ve been using Visual Studio Project Cleaner for years to clean unwanted files. It turns a tedious job into a simple and easy job.

Clean your files today with Visual Studio Project Cleaner.

Improving how we build projects and solutions

By , March 12, 2018 12:01 pm

The problem with Solutions

If you’re working with just one solution and the projects it contains Microsoft Visual Studio is a good tool for managing the builds. Except that it won’t necessarily build the projects in the right order, forcing you to make multiple build passes to get a built solution.

But what if you’re working with many solutions?

What if some of the projects in the solutions have identical project names, even though they are different projects? You can’t merge these into one mega solution so that you can build everything at once because the project names will clash.

Another problem with solutions is that the projects in the solution do not necessarily get built in the optimum order. A project that depends on another project may be built before that project dependency is satisfied. This forces you to build the solution multiple times until all dependencies are satisfied.

What if there was a better way to manage your builds? What if there was a better way to automate your builds?

Born from a real need

At Software Verify we have a code base of several million lines of C++, plus some C, assembler and C#. Looking at the Visual Studio Project Builder display I can see we have 28 solutions, with 217 projects, giving us a total of 962 different project configurations to complete a build. (The actual number is higher than this because it doesn’t include the legacy support solution and projects for Visual Studio 2002, because that can’t be installed on a 64 bit machine.).

This large number of projects and solutions includes every version of Visual Studio from Visual Studio 2017 to Visual Studio 6.

We needed to have a build tool that would allow us to reliably build a project, or solution, or a specific configuration of a project, regardless of version of Visual Studio or programming language.

All this would be done with one simple to use software tool that could be used interactively with a GUI or from the command line. The tool would provide visual feedback so that you could tell from 50 feet if the build was going pear shaped. The tool would also provide notifications so that you could forget about it and go do something else instead.

Visual Studio Project Builder

I’ve seen various solutions to this problem, lots of scripts, batch files, programs interacting. It works, kind of, but it’s all a bit of a mess, held together with string and duct tape in the best “make it work” seat of the pants “we haven’t got enough time” developer fashion.

That isn’t what I wanted. We’d already finished Visual Studio Project Fixer and I was very pleased with the results from that. I wanted the same “one tool” approach. We had the requirements, we knew what we needed to be able to build. We knew we had a decent test case (build our own tools) and we knew what a good tool would look like. As simple as possible.

That’s how Visual Studio Project Builder was born.

The basic capabilities are:

  • Automatically discover solutions and projects.
  • Optimise the build order.
  • Build, rebuild, clean project for any Visual Studio version from 2017 to 6.
  • Build individual project configurations, or variations based on project, solution and configuration.
  • Easily identify warnings and errors.
  • Easily filter warnings and errors.
  • Interactive GUI with real time build progress updating.
  • Can be run from a command line if desired.
  • Works on any Windows OS from Windows 10 through Windows XP.

A simple user interface


When performing a full build, full rebuild or full clean, a progress dialog is displayed, allowing the progress of the build to assessed at a glance, but with extra detail if you want it. You can tell from the colours on this dialog if a build is good, or bad, or has warnings, from 50 feet.


Behind the progress dialog is the main display, showing the status of every project configuration that can be built. Colours indicate status (good, warning, compiler error, linker error). A context menu provides access to options related to that project configuration, project or solution. For example:

  • Build all projects named “stubNonMFC” that have the configuration “Win32|Debug”.
  • Rebuild all projects that have the configuration “Win32|Release”.
  • Rebuild all projects in this solution that have the configuration “Win32|Release”.
  • Clean all projects named “tabserv”.
  • View build logfile.
  • View build command line passed to Visual Studio.

Some of the build options from the context menu cannot be done with Visual Studio, except by manual intervention for each project. When you span this across many solutions and projects this becomes very powerful and an incredible time saver.

Build notifications

Software developers perform a mentally taxing job. The last thing you want to add to their list is “Is the build done yet?”. To remedy this problem we have provided several (optional) notification methods:

  • Email when build is complete.
  • Text message when build is complete. Uses Twilio.
  • Speech message when build is complete.

My favourite is the last one. I can go and do something else, read a book, investigate a bug, work on something else, talk to someone, and when the build is done my computer talks to me: “Build Complete, Success”, or “Build Failure. 3 Errors”.

The email and text message notifications are intended for use when you are out of range of the computer talking to you.

We have delayed implementing carrier pigeon functionality until there is a proven market for it.

Does this compete with existing build solutions?

Does this compete with existing build solutions such as IncrediBuild?

No. Incredibuild speeds up builds of projects and solutions by distributing them on your local network.

Visual Studio Project Builder is all about getting projects built, in the right order, even though many toolsets may be in use. It all happens on the same machine. We’re hoping that Visual Studio Project Builder can work with build accelerators like IncrediBuild to speed things up.

Build your Visual Studio Projects

We’ve been using Visual Studio Project Builder in house for some time. We thought it was about time everyone else got a chance to use it. We’re making this available for free. Learn more about Visual Studio Project Builder.

Fixing Visual Studio project problems

By , March 5, 2018 2:51 pm

The productivity sink

Visual Studio is a very productive development environment. However, you can get Visual Studio into a state (for C++ and C developers, at least) where the compile and link both succeed, but when you go to run the executable Visual Studio annoyingly tells you that the build is out of date and would you like to build it?

This is a very frustrating experience. You go around and around in circles trying to work out what is wrong with the project that is causing this warning message to be displayed.

After some head scratching you’ll Google it and find solutions on numerous websites that tell you to do this, or that, or the other. The advice on some sites is consistent with the advice on other sites, some advice is not consistent. Then you find that some advice is for one version of Visual Studio, but you need to do something differently for a different version of Visual Studio.

It’s a mess. But you work out what you need to do, set about it and after tweaking a registry setting, running a tool to grab the output from OutputDebugString (or writing one if you couldn’t find a tool!), you now know which file is causing Visual Studio to tell you the build is out of date. Great!

Er, not so fast. The problem with this is Visual Studio only tells you about the first file that causes it to need to be rebuilt. Any other files aren’t listed, so you have to make the change then repeat the whole cycle. Slow!

“Crikey, that was a lot of effort, I hope that doesn’t happen again”, you think, or perhaps something a bit less polite. And then you remember you’ve got 10 more projects that behave like this, but you’d been ignoring that for a long time. But now you know how to fix it, but you can’t imagine putting in the effort for those 10 projects.

That was the situation we were in, except that just about every project we had (over 200 projects) had this problem. After tackling just one project I knew we had a choice:

  • Ignore the warnings.
  • Fix the warnings manually. Very slow. Lots of typing. Boring. Morale killer.
  • Find a way to automate the fixing of these problems.

Our engineering needs drove the decision.

Visual Studio Project Fixer

We decided to write a tool to make the hard bits of the above easy and to automatically discover what was wrong with a project file, then allow the user to choose which items to fix and which to leave alone. Plus an automatic mode that just fixes everything for you.

Things that can affect a project file are:

  • Missing files. The file exists on disk, but is not specified in the project.
  • Incorrect files. The file exists on disk, but not in the location specified in the project
  • Nonexistent files. The file doesn’t exist on disk, but is specified in the project.
  • Obsolete files. Files from older versions of Visual Studio.
  • .tlog files. Visual Studio file dependency files.

Some examples…

Missing file. You have a class defined in class.h and implemented in class.cpp. Only class.cpp is listed in the project file.

Incorrect file. A file is listed in the project as ..\svlCommonUI\class.h but is actually located at ..\svlCommon\class.h.

Nonexistent file. A resource file is specified that doesn’t exist. .\images\nonexistent.bmp.

Obsolete file. An old format help file from an early version of Visual Studio, carried over in Visual Studio project conversion.

A simple user interface


The image above shows the user interface after a scan of many projects has been performed. The summary display is shown, which lists an overview of the state of the projects and provides an option to automatically fix all the projects. If you click an individual project, the display changes to provided detailed information about that project.

There are four ways of scanning projects, providing for maximum flexibility in automating fixes or manually choosing each fix to apply.

  • Open a project.
  • Open a solution.
  • Open all projects in a directory (and sub-directories).
  • Open all projects in solutions in a directory (and sub-directories).

Having scanned some projects you can examine the results and choose to fix individual projects, or to fix all the projects that need fixing.


The default settings are to backup all project files before modifying them.

Diagnostic Information

It is possible that even with all the information provided by Visual Studio Project Fixer you may still wish to view the output from Visual Studio. Visual Studio 2015 and 2017 provide an option to show you this information. However for Visual Studio 2010, 2012 and 2013 you can enable diagnostic information to be sent via OutputDebugString(). This means that Visual Studio Project Fixer can intercept this information. All you need to do is enable it (from the Edit menu).

Then every time Visual Studio checks the status of a build, it’s output can be seen on the Visual Studio OutputDebugString tab.


Fix your Visual Studio Projects

We’ve been using Visual Studio Project Fixer in house for some time. We thought it was about time everyone else got a chance to use it. We’re making this available for free. Learn more about Visual Studio Project Fixer.

Changes to the first run wizard

By , February 15, 2018 2:41 pm

When you run one of our tools for the first time it displays the first run wizard.

The first run wizard takes you through the steps of configuring a few settings that are important to the tool your a using. Most of these settings are the same for each tool, although some of the first run wizards have settings pages that are specific to that tool.

We’ve just add a new page to the first run wizard. We’ve added the ability to configure the behaviour of the launch, inject and wait dialogs. The default behaviour is wizard. The behaviour can be modified from the Settings menu (formerly, Configure menu). Now this setting can also be set on the first run wizard.

The reason for adding this option to the first run wizard is because when we’ve done interactive debugging with customers we often notice that the user still has their Validator configured to use the launch wizard, rather than the more concise launch dialog. We thought there was a chance that some people don’t realise they can change the style of these dialogs/wizards.


Panorama Theme by Themocracy