Rss Feed
Tweeter button
Facebook button
Technorati button
Reddit button
Myspace button
Linkedin button
Webonews button
Delicious button
Digg button
Flickr button
Stumbleupon button
Newsvine button

Category: Development

64 bit tools leaving beta this month

By , December 5, 2013 1:14 pm

For those of you keeping a keen eye on the version numbers of the various C++ 64 bit betas we have will have noticed that the betas now have the same version number as their 32 bit counterparts. The reason for this is that we are getting ready for these tools to leave beta and to be released. We expect this to be during December 2013, if all goes well in the next week or so.

Before we can do this we need to make some modifications to the website, the shopping cart, test it all still works correctly, etc.

Once released the 32 bit and 64 bit C++ tools will have the same version number.

The 32 bit tools will continue to work with 32 bit executables on both 32 bit and 64 bit operating systems.

The 64 bit tools will only work with 64 bit executables on 64 bit operating systems.

The 64 bit tools will come bundled with the 32 bit version of the tool so that you can work with 32 bit or 64 bit executables on 32 bit and 64 bit operating systems. We anticipate a future version of the 64 bit tools that can launch and monitor 32 bit executables. This will allow you to collect more data and examine much larger datasets when testing 32 bit executables. We have prototypes of these tools now, but they are not ready for public release yet.

Why so long?

I know its been a very long time for these tools to have been in beta. We could have released some of them some time ago but we wanted to release all four at the same time so that we could also offer suites of tools and the also the combined 32+64 bundles. The problem was that C++ Memory Validator x64 had some problems with a few products being beta tested and C++ Performance Validator x64 also had some problems with SolidWorks 64 bit. We fixed the C++ Performance Validator x64 problem a few months ago and have been working on nailing all the remaining C++ Memory Validator x64 bugs.

Two beta testers in particular deserve a special mention for really helping us with these last bugs. Ciro Ettorre and Kevin Ernst. Ciro provided many logs of data for me to stare at and Kevin provided us with a machine that we could use to repeatedly try new fixes on.

I guess I didn’t really follow the advice I give to many people: “Get your product out there”. But that’s because these are new versions of existing products and I wanted them to be as good as the existing 32 bit tools. I hope we do not disappoint you.

Recommendation

I’d also like to recommend TeamViewer as a very useful product for remote working. We couldn’t have done the work with Kevin without this excellent tool. Best used with a headset microphone with the Voice over IP part of TeamViewer turned on.

What next?

As always we’ll keep you updated with what is happening with the betas via the email address associated with your beta test account.

Share

DbgHelp search path

By , June 26, 2013 9:54 am

The problem

One of the problems using DbgHelp.dll to read the symbols for your Visual Studio application is that sometimes your symbols do not load.

There are typically three broad classes of failure when trying to load symbols from a PDB file:

  1. Missing PDB file.
  2. PDB file located in the wrong place. You think the PDB is present but DbgHelp.dll can’t find it.
  3. An incorrect PDB file. You think the PDB is correct but it’s actually from a different build.

You think you have the symbols in the right place but the symbols don’t load. You double check, you may even move some files around in desperation, and still the symbols do not load. This is a source of great frustration and a waste of time. What can you do about that?

The solution

With the latest release of our C++ software tools we’ve update the Diagnostic tab to include a combo box that allows you to choose what to display on the diagnostic tab. You can view everything (the default) or just specific types of information. One of those types is DbgHelp debug. When this is selected the diagnostic tab will only show you information which is the debugging output from the DbgHelp.dll. You can see every message that DbgHelp.dll issued, including each path that DbgHelp.dll looked into to find a PDB file and whether it found a PDB, or the PDB was mismatched or the PDB was found but only with COFF symbols or if the PDB was found with public/private symbols and lines.

We display the output for each module in alternating colours. This makes it easier to identify messages for one DLL compared to another DLL.

Here are some examples of symbol load success and failure:

Correct symbol file

DbgHelp.dll search path correct symbols found

DbgHelp searches in various places looking for mvExample.pdb.
Eventually mvExample.pdb is found in e:\om\c\memory32\mvExample\DebugNonLink6_0\mvExample.pdb.
DbgHelp loads private symbols and lines. (The alternate outcome is that DbgHelp loads public symbols).

Outcome: Success. Symbols are loaded.

Missing symbol file

DbgHelp.dll search path no symbols found

DbgHelp has the search path set then searches in various places looking for mvExample.pdb.
mvExample.pdb never gets found on the search path. SymSrv then looks for additional locations for mvExample.pdb. None are found.
DbgHelp does find some COFF symbols in the executable. Unfortunately COFF symbols do not contain filename or line number information.

Outcome: Failure. The PDB file could not be found. Some default symbols are loaded but are not of much use.

When a PDB file can’t be found you can examine the search path used by DbgHelp.dll and then check where your PDB file is to ensure that your PDB file is on the DbgHelp search path. You can either move your PDB file onto the search path or you can update the File Locations part of the settings dialog to include a path to your PDB file.

Mismatched symbol file

DbgHelp.dll search path mismatched symbols found

DbgHelp searches in various places looking for mvExample.pdb.
Eventually mvExample.pdb is found in e:\om\c\memory32\mvExample\DebugNonLink6_0\mvExample.pdb.
DbgHelp attempts to load the symbols but fails because the symbols are for a different build of the software. The checksum inside the PDB file does not match the module.

DbgHelp does find some COFF symbols in the executable. Unfortunately COFF symbols do not contain filename or line number information.

Outcome: Failure. A PDB file was found but it was not the correct PDB file. Some default symbols are loaded but are not of much use,

When an incorrect PDB file is found you can examine the search path used by DbgHelp.dll and then either update the PDB file to be the correct PDB file or if the PDB file is being found because the search path is incorrect you can update the search path in the File Locations part of the settings dialog to prevent the wrong PDB file being found.

Things to check

  • Ensure the PDB file found is the correct PDB file for the build. If you are copying builds from a build server be sure to copy the correct PDB files as well.
  • Check the File Locations PDB paths to ensure that all the possible paths for PDB files are listed in the correct order so that if multiple paths have a PDB file with the same name that the correct PDB file is found first.
Share

New environment variable dialog

By , May 9, 2013 2:57 pm

We recently had some feedback from customers telling us that they were dealing with legacy applications that were configured using large numbers of environment variables and that starting configuring such large number of environment variables via our environment variable dialog was unwieldy.

When I asked “How many environment variables?” I was told “Up to 161, depending on the application”. Ouch. Entering that lot by hand will be tiresome.

To improve the situation for environment variable usage we’ve added three new buttons to the dialog to allow you to acquire environment variables from the Operating System, to import environment variables from an ASCII text file and to export environment variables to an ASCII text file.

Environment Variable Dialog

Of course once you’ve acquired or imported your environment variables you can edit them as you see fit them either export them or click OK use this configuration on the Launch dialog/wizard. Any environment variables used in a given launch configuration will be used again if you re-use a particular configuration.

Share

“cannot open type library file” error on x64 systems

By , May 18, 2012 11:18 am

I’ve just tried building a Visual Studio 2010 helper DLL on Windows 7 x64.

The build failed with “cannot open type library file vsmso.olb : No such file or directory”.

A quick search found them in c:\program files (x86)\microsoft shared\MSEnv. This is the folder for 32 bit applications (Visual Studio is a 32 bit application).

OK, so if the files are present on the machine why does the build fail for this line?

	//The following #import imports VS Command Bars
	#import <vsmso.olb> raw_interfaces_only named_guids

The reason is the Visual Studio project include directories are setup for c:\program files\microsoft shared\MSEnv. You need to edit the properties for Debug and Release for each processor type (in my case Win32 and x64) and change the include directory (or add another include directory) c:\program files (x86)\microsoft shared\MSEnv then rebuild.

This is a potential non-obvious timewaster. Hope I’ve saved you some time.

Problem solved.

Share

x64 Porting gotcha #3. Serializing collections

By , April 22, 2012 1:21 pm

When Microsoft ported MFC to 64 bits they also changed the return type for the GetSize() and GetCount() methods in the collection classes. They changed the return type from the 32 bit DWORD to the 64 bit DWORD_PTR on x64. This has implications if you write your own collection serialization methods. For example if you use a CMap<> to map a thread id to an object you will want to write your own serialization code.

For example (error checking removed for simplification), consider the serialization of this collection.

	CMap		threadObjectStatistics;

Saving

	CSingleLock	lock(&threadObjectStatisticsSect, TRUE);
	POSITION	pos;

	ar << threadObjectStatistics.GetCount();

	pos = threadObjectStatistics.GetStartPosition();
	while(pos != NULL)
	{
		runningObjectManager	*rom = NULL;
		DWORD			threadID;
	
		threadObjectStatistics.GetNextAssoc(pos, threadID, rom);
		ar << threadID;
		rom->save(ar);
	}

Loading

	CSingleLock	lock(&threadObjectStatisticsSect, TRUE);
	DWORD		i, count;

	ar >> count;
	for(i = 0; i < count; i++)
	{
		runningObjectManager	*rom = new runningObjectManager();
		DWORD			threadID;
	
		ar >> threadID;
		rom->load(ar);
		threadObjectStatistics.SetAt(threadID, rom);
	}

In the above code the first item saved/loaded is the number of objects in the CMap. After that the thread id and the complex object associated with the type is saved/loaded for each pair of objects in the CMap. The code above uses a DWORD to load the size. This won’t work for x64 because the count of objects is taken directly from the GetCount() method (or GetSize() for some collection types).

	ar << threadObjectStatistics.GetCount();

x86, return type is DWORD, count is saved as DWORD (32 bit)

x64, return type is DWORD_PTR, count is saved as DWORD_PTR (64 bit)

This is a problem because the loading code is expecting a DWORD.

	DWORD		i, count;

	ar >> count;

Update (23/4/2012): Turns out the same issue affects the STL collections as well. If you are directly serializing the result from the size() method in an STL collection you will be faced with the same problem as I describe for MFC.

Solution 1

One solution is simply to change the type being loaded from a DWORD to a DWORD_PTR.

	DWORD_PTR	i, count;

	ar >> count;

Solution 2

An alternative solution is to always save the size as a DWORD.

	DWORD		count;

	count = (DWORD)threadObjectStatistics.GetCount();
	ar << count;

Conclusion

You may think this is a trivial issue and why write a blog post about it? I agree the actual problem is trivial and the fix for it is also trivial. However the fact that you have a mismatch in datatypes being serialized because you directly serialized the return value from GetCount() is not so obvious. So much so that this particular issue escaped our attention (and got past a static analyser) until today.

So yes, its a trivial problem but its a hidden problem and it will cause all sorts of issues during serialization and when you go looking for it you'll probably look straight past it for a while. Hopefully I've just saved you a few hours of banging your head on a brick wall, or more likely your desk.

Share

Communication

By , March 30, 2012 3:20 pm

Are you any good at communication? Thats a good question. If you’re like me, you probably think you are not good at communication.

You may think that good communicators are slick, polished, don’t make mistakes, can sum things up nicely and exude a certain level of confidence. Well yes, the public speakers that have been around for a while do tend to be that way. But that is more the result of speaking in public many times than natural ability. Eric Ries wasn’t as polished when he started speaking in public. He is the first to admit that. Practice makes perfect.

That isn’t want I’m talking about. I’m asking you if you communicate your ideas to others and if at the end of the conversation do they understand you?

Why am I mentioning this? Years ago I had a performance review by my line manager. The review was done in 3 parts. My manager would fill in his scores for various tasks and abilities, rating me. I would do the same. Then we’d compare the scores we had for each task and discuss the differences/similarities and how to improve any areas that needed improvement and how to make the most of areas I excelled at.

There were lots of categories, none of which I can remember except for “communication”. I don’t tend to award myself a 10 or 1 in anything when I self score, so the sheet had various high scores and a few low scores. I was hitting it out of the park in software development terms (which would come back to bite me a few years later in the form of RSI – my work was 3 months ahead of schedule) but for communication I gave myself a really low mark.

The really low mark for communication confused my line manager and we spent a good chunk of the review just talking about communication. I had interpreted “communication” on the form as “Can I present, Do I speak in public, do I do this, that the other…” all these imagined things that I thought a good speaker should do. I didn’t think I could do them.

My line manager wasn’t interested in that. He was interested in did my team mates, colleagues, people in teams interfacing with our team (in person or via our documented API), senior managers etc, did these people understand the technical work I was doing? Did they understand how to use it, why certain things were the way they were and most importantly if someone came and asked me a question could I answer it with confusing the living daylights out of them?

Turns out I could. I scored quite well on all that. No idea what he wrote down. Not really relevant these days.

All I’m trying to say is communication is making sure the other person understands. Far better to be slightly slower, or more verbose, or elaborate (or whatever) and succeed in communicating than be super concise, uber terse, abrupt and leave the other person feeling bewildered or intimidated by what you’ve just said.

How do you feel about communication now? Think you’re better at it than before reading this?

Share

64 bit porting gotcha #2! x64 Register preservation

By , March 9, 2012 8:01 pm

In a previous article on x64 development I mentioned the problem of aligning the callstack on 16 byte boundaries and what happens if you do not do this.

Why 16 bytes?

At the time it seemed odd to me that the stack had to be 16 byte aligned. Why? All the parameters are 8 bytes (64 bits) wide and the first four are passed in registers. Everything else spills onto the stack and anything larger is passed as a reference. Floating point values are passed in dedicated floating point registers.

And there lies the key. That last sentence. The floating point registers.

Floating point on x64 is not done using the floating point coprocessor instructions. Instead the SSE instruction sets (and its extensions) are used.

If everything floats, whats the point?

If you are just hooking x64 functions and possibly collecting callstack you may never know need to know about floating point register preservation. We managed to get all four of our C++ tools (coverage, memory, profiler and deadlock detector) functional without knowing. Why would we need to know? Floating point preservation was never important for x86 because we could never damage the floating point without trying to do so.

But when we got into the details of the last bugs that were killing us we noticed seemingly random crashes. Closer investigation showed that calls to Win32 API functions had a tendency to wipe out the floating point registers. And that is when we got interested in what happens if we preserved the x64 floating point registers.

How to view the registers?

At first glance this wasn’t obvious to me. The Registers window in Visual Studio just shows the registers from RAX through R15 etc. However if you right click this window there is a helpful context menu that allows you to choose just how much information you display in this window.

Once you have the registers in view things get a lot easier inside the debugger. You can step through your code ensuring that nothing is getting trampled on until Viola! the floating point registers get totally hosed. A bit more investigation and you realise that seemingly innocent call you had in your code contains a call to a Win32 function (for example VirtualProtect) and that that function is responsible for the register death.

OK, so how do we preserve registers on x64? Its nothing like on x86.

x64 floating point preservation

The x64 designers in their infinite wisdom took away two very useful instructions (pushad and popad). As a result x64 hook writers now have to push lots of registers and pop lots of registers at the start and end of each hook. You can even see this in parts of the Windows operating system DLLs. So much simpler just to push everything and pop everything.

However what the Lord taketh away he can give back. And the x64 designers did that by providing two dedicated instructions for saving and restoring floating point. fxsave and fxrstor. These instructions take one operand each. The operand must point to a 512 byte chunk of memory which is 16 byte aligned.

A common usage would be as shown below although you can use any register as the destination location. It just so happens that the stack pointer (rsp) is the most common usage.

	sub	rsp, 200h;
	fxsave	[rsp];

	.. do your task that damages the floating point registers

	fxrstor	[rsp;]
	add	rsp, 200h;

When you see the above usage you can see why there is the requirement for the stack to be 16 byte aligned. Why 16 bytes? I suspect it is because it is the start of a cache line and that makes executing the instruction *SO* much quicker.

Conclusion

So now you know why the x64 callstack is 16 byte aligned. Its all to do with ensuring your code executes as fast as possible, especially when executing a memory intensive register copy when saving and restoring the x64 floating point registers. I’ve also shown you how to preserve and restore the floating point registers.

Share

Unhandled non-continuable exception. What?

By , February 17, 2012 1:13 pm

The Problem

A few days ago I was testing the latest version of C++ Memory Validator x64.

The testing was going fine until I tried testing a particular test tool of ours. C++ Memory Validator x64 would inject into the test tool, start injecting then just die at some point.

No problem, just set the injection flags to inject a breakpoint and that’ll trigger the debugger as soon as we inject. This works with most software and with all our test tools. No joy. Hmmm, puzzling.

OK, try again, but when we get to CreateProcess() I’ll attach the debugger to the paused process myself. That’ll work. Right? Wrong. The debugger does attach and thats great. I resume the process and the the debugger spits out a really impenetrable error message.

“Debugger:: An unhandled non-continuable exception was thrown during process load”.

That is a really useful and useless message all in one.

Useful Unhandled. No one handled it, so in that case it will be a problem.
Useful Non-continuable. Even if someone could handle it, you can’t recover from it. Major problem.
Useless No description of what the exception was, no exception code, nothing. Why?

Next steps

One possible next step would be to repeat this sequence but before resuming the application, go to the debugger and open the Exceptions dialog from the Debug menu.

When the Exceptions dialog is displayed go to the Win32 exceptions part of the settings and expand it so that you can see the various exceptions that will be reacted to. We need to tell the debugger to react to some of these exceptions. I’ve included screenshots with the exceptions highlighted. Feel free to enable other exceptions that you think may be troublesome.

Having enabled the appropriate exceptions you can resume the process and see if the debugger reacts to any of these more obscure exceptions.

Solution

It turned out the problem was a DLL dependency was failing and thus resulting in a module not found exception.

In my case what had tripped me up was that we’ve been doing static analysis of our software recently using PC-Lint from Gimpel Software combined with Visual Lint from Riverblade. PC-Lint does the hard word of analysing your source code and Visual Lint organises the mountain of results into useful and usable information. If you’ve ever seen the log file from PC-Lint you’ll understand the benefits of a tool like PC-Lint to organise the results.

The result of the static analysis is that we’ve changed many APIs. Many objects that were passed in as objects (and implicitly incurred object copying penalties) are now passed as const references. Many char * strings and wchar_t * strings are now passed as const and so on. We’ve done this all over – changing our DLL APIs, everything.

It’s great, we’ve found bugs because of this. Really useful. But I digress. One side effect of this is that the anything dynamically or statically linked against our libraries now fails to work. We had rebuilt all the dynamically linked tests but forgotten the statically linked one. The test I was performing was statically linked.

Rebuilding and relinking the statically linked test meant that the DLL imports/exports now resolved and the DLL would load. Problem solved.

The change to our APIs is a one time change and will be painful for the folks that use our linkable APIs, but the benefits are increased software robustness for everyone, ourselves and customers alike.

Share

Improving your code with Static Analysis tools

By , February 16, 2012 1:58 pm

Update, October 2012. I’ve edited this to include more tools, prices and our experience with each vendor we’ve interacted with.

At Software Verification we create tools for the dynamic analysis of software – analysing your software as it executes. This is an activity that takes places after you’ve compiled your software. There is a complementary method of analysing your software – static analysis. Static analysis is the process of analysing your software before your compile it.

There is a small overlap between the types of bugs that static analysis and dynamic analysis can find. Static analysis will find some of the bugs dynamic analysis finds, but static analysis will also find bugs that dynamic analysis cannot find and dynamic analysis will find bugs that static analysis can’t find. In summary you should have both static analysis tools and dynamic analysis tools in your software tool box.

Tools

If you look around the web you will find a variety of tools, some open source and some commercial. The commercial tools range from the sensibly priced Gimpel PC Lint to tools that are so expensive you need to be sending rockets into space to afford them. For these products where the vendor does not list the price on the website, we have listed POA in the table below. POA means “Price on Application”, which you can think of as code for “very expensive”.

One other thing, most of these POA tools only license you for using the tool for one year. After that you have to buy the software again (not maintenance, but the right to use the software again). So they are even more expensive than you think.

This is not an exhaustive list of tools. There are many more, especially if you look outside of C/C++.

Tool Type Cost Web
Coverity Yearly subscription POA http://www.coverity.com/
Klocwork Yearly subscription POA http://www.klocwork.com/
Mathworks Polyspace Permanent? POA, £20,000/user http://www.mathworks.co.uk/products/polyspace/
Understand Permanent POA, 1003 Euro, Maintenance 18% https://www.scitools.com
CodeSonar Yearly subscription POA http://www.grammatech.com
Pattern Insight Yearly subscription POA http://patterninsight.com/products/code-assurance
Imagix Yearly subscription POA http://www.imagix.com/a/source-code-analysis.html
CheckMarx Yearly subscription POA http://www.checkmarx.com
Parasoft C++ Test Commercial POA http://www.parasoft.com/jsp/products/cpptest.jsp
PVS-Studio Commercial 3500 Euro, maintenance 80%, minimum 5 users http://www.viva64.com/en/pvs-studio/
QA C++ Commercial POA http://www.programmingresearch.com/qacpp_main.html
SEntry Yearly subscription $4,995 US http://www.vigilantsw.com/
Gimpel PC Lint Commercial, permanent license $389 US http://www.gimpel.com/html/pcl.htm
Microsoft PREfast Commercial Free http://msdn.microsoft.com/en-us/windows/hardware/gg487345.aspx
CppCheck Open Source Free http://sourceforge.net/apps/mediawiki/cppcheck
Splint Open Source Free http://lclint.cs.virginia.edu/
Google cpplint.py Open Source Free http://google-styleguide.googlecode.com/svn/trunk/cpplint/cpplint.py
Inspirel Vera++ Commercial Free http://www.inspirel.com/vera/

In addition to the above tools there is also a Visual Studio addin (Visual Lint by Riverblade) that can manage many of the above tools and provide you with a very usable, nice interface to the mountains of data some of these tools can generate. Visual Lint is not actually a lint or static analysis tool. Visual Lint’s job is to organise the output of the static analysis tools.

Riverblade Visual Lint Commercial, permanent license $399 US (varies) http://www.riverblade.co.uk/

Our static analysis history

Several years ago we thought getting a static analysis tool would be a good idea. We approached several tool vendors and drew a blank. The prices were astronomical – Coverity (based on their standard pricing) would want $5.6 million (for a one year, single user license) if I remember correctly. An email exchange with them revealed they would provide a discount but the price was still more than half a million dollars. We approached Klocwork but their reseller was only interested in selling us a five user license when we only wanted a single user license. I think that was $20,000 for a one year license.

Recently we’ve approached more vendors:

PVS-Studio

PVS-Studio insist on selling a minimum order of 5 users, which isn’t very useful when you only have a requirement for one user. But you can evaluate without talking to anyone, just download an evaluation from their website. PVS-Studio also works out of the box with your Visual Studio projects. By far the easiest to setup.

Mathworks

Mathworks make you complete a long account form on the website (which appears to be redundant) then wait for an email, then a telephone call followed by two more emails and er, we’re still waiting. Strikes me as a rather good way to lose sales. We’ve evaluated several other tools just while we waiting to even start with MathWorks. Mathworks is at the expensive end of the pricing scale, but by no means the most expensive. But if you can’t even evaluate, how do you get to purchase?

SciTool

SciTools Understand, like PVS-STudio, allow you to download an evaluation. SciTools appears to be hard to setup – it defeated us.

Imagix 4D

Imagix 4D, like PVS-STudio, allow you to download an interactive demo and then to request an evaluation if you like the demo. From the demo we couldn’t see how this is a static analysis tool, so we passed on the evaluation. It seems to be more of a code understanding tool.

Grammatech CodeSonar

Grammatech require you to fill in a form, then they telephone you to start the evaluation.

Checkmarx

We haven’t got around to evaluating yet but their email response time and quality of answers to questions is excellent.

VSLint

We had previously used Gimpel PC Lint, but found it very hard to configure. Then we found Riverblade’s Visual Lint which is an addon for Microsoft Visual Studio. Visual Lint works with many other analysis tools and provides a nice visual front end for viewing the output of the other tools. For us this was perfect as it worked with Gimpel PC Lint and configured it correctly out of the box. No messing with Gimpel PC Lint to make it work with our Visual Studio builds. We’ve had a few problems (our setup seems unusual) and have found Riverblade very responsive to support requests. We now have x86 and x64 static analysis setup using VSLint.

We have been analysing our software using Visual Lint recently. We haven’t found a lot of bugs (great!), but it has revealed subtle signed/unsigned conversions, some logic errors, redundant code and some code that could (in the right conditions) manifest as bugs. It also reports various issues that you may regard as coding style issues. One of the coding styles I hate is the call a function, assign its return value and test a conditional all in one.

    if (!(fResult = ETPhoneHome()))
    {
        RideBicycleIntoSpace();
    }

How much clearer to write:

    fResult = ETPhoneHome();
    if (!fResult)
    {
        RideBicycleIntoSpace();
    }

The compiler will generate the same code, but the second is easier to read and should you need to debug it, the second version is easier to debug. Disk space is cheap. There is no reason to code in the style of the first version. As such the informational warnings like this can be used a mini-style guides. Fix the issue to remove the informational warning from the tool output.

For any warning you don’t agree with or don’t care about you can filter them out.

I must say that I’m very pleased with the outcome of this exercise. I can recommend the combination of Gimpel PC Lint and Visual Lint. Finding these bugs so easily has paid for the software tools used to find them.

Unusual benefits

I’m going to list a few things the PC-Lint/Visual Lint combination warns about which I’ve found beneficial.

Variables not initialised in constructors and functions.

Easy mistakes to make. Often caused by logic errors rather than forgetfulness.

Unused variables, unused functions and unreachable code.

Variables you once needed, left behind after edits, cluttering the place up. Same for debugging function and historically obsolete code. Useful to know what you can safely remove. A word of caution though – be sure to check for conditional compilation excluding the code for the lint analysis. Lint can get this wrong sometimes.

Incorrectly defined copy constructors and assignment operators.

I’ve found these to be a boon. The previous constructors and operators worked OK, but better to have them in the style that STL will expect.

Lack of const

Before:

void myFunc(char *title);

void myFuncRef(CString &title);

After:

void myFunc(const char *title);

void myFuncRef(const CString &title);

These are minor wins as this change only gets suggested if Lint can determine that the parameter is not modified. However it is a win for the rest of your software as you can now pass const objects into these functions without casting them to none const (ouch!) and also know that the function, if modified to modify the objects will not compile properly. So it is a useful maintenance win.

Change from pass by value to pass by const reference.

Before:

void myFuncRef(CString value);

After:

void myFuncRef(const CString &value);

This is a major win. You get the maintenance wins from the previous example plus you get a CPU saving because there is no implicit object copying happening to pass the value into the function. In the case of complex objects (like CString) which allocate memory on construction and destroy memory on destruction there you save even more CPU cycles because the memory manager is not called. Additionally you are less likely to fragment the heap because the heap is used less. Win, win, win!

You definitely want to be making changes like this. Visual Lint will tell you stuff like this in its results.

The only downside to these changes are that you need to rebuild after the changes. If the changes are in a core header file or a class that many files use you’ll spend a while rebuilding. But this pain is short-lived. You’re making the software more robust in the process and long term that is a worthwhile trade off.

Pricing

I think most static analysis tool vendors are missing a huge opportunity. By pricing themselves so expensively most software houses will not purchase such tools. Then to compound matters they make it a yearly subscription so you have to spend the full price again each year. When I look at purchasing software the first thing I look for is a published price list. The lack of a price list sends the price signal “we are expensive”. This is a mistake – surely the message should be “we offer value” and be confident enough to state your pricing. I can’t help thinking hidden pricing means the price varies depending on what they think you can pay. That’s really not a good message to send. Since the previous version of this article I’ve approached some of the POA vendors and found that some of them are expensive and some are very affordable, but still have unlisted prices “just because that is the industry convention”.

Any company that chooses to price their tools so they are affordable to software developers (in the $200 – $1000) range will have a much better opportunity for these tools. From what I can see only Gimpel and Riverblade have taken this route. ? SciTools Understand is a sensible price, but hard to setup. PVS-Studio come close but lose it with their minimum user limit of 5 users. Who is going to pay for non-existent users?

Conclusion

If you are not using static analysis tools I recommend that you spend some time investigating such tools. The free tools have their pitfalls – incomplete implementation and they don’t always have support. CppCheck seems to be highly thought of (it doesn’t work properly with our code). If you are using Gimpel PC Lint or want to use a lint then I recommend Visual Lint for use with Visual Studio. If you have the budget then check out the POA tools in the table above.

Some people make the mistake of thinking that static analysis tools replace dynamic analysis tools (like C++ Memory Validator, etc). Static analysis tools complement dynamic analysis tools. There is a small middle ground where some bugs will be identified by both a static analysis tool and a dynamic analysis tool, but each group of tools can identify whole classes of bug that the other tool cannot identify.

If you are a tool vendor listed above please let us know your pricing and/or licensing terms if this article lists it incorrectly.

Share

Improving MFC memory performance

By , January 2, 2012 12:16 pm

If you are using MFC arrays it is possible in quite a few cases to improve the speed and memory performance of these arrays. This applies to the standard CStringArray, CUIntArray and similar classes and also to template classes based upon the CArray template.

If you are not using MFC but using another framework or set of classes that provide similar functionality you can often find similar functions in those classes that will allow you to get a similar benefit to what I will describe in this article.

The Problem

The problem is the typical use of populating the array is by calling the Add() method to add something to the array. No actual problem with that, its simple and straightforward enough. However, under the hood, each time you call Add() the array class has to reallocate more memory of the data stored in the class.

This reallocation has a cost. The cost is increased CPU usage as suitable memory space is searched for by the memory allocator and memory is copied and reassigned inside the class. For small arrays this is not really a problem.

However for larger arrays this becomes quite a noticeable issue. In addition you also run into potential memory fragmentation issues, where memory "holes" of an unusuable size are left in the memory managed by the memory allocator. Should enough of these holes occur you can suffer out of memory conditions even when Task Manager tells you you have available memory. Frustrating! I’ll cover Memory Fragmentation in a different article.

Here is a (simplified) example of the type of problem:

// read data from the serialization archive and store in an array

DWORD i, n;

ar >> n; 
for(i = 0; i < n; i++)
{
    someClass *sc;

    sc = new someClass();
    if (sc != NULL)
    {
        sc->load(ar);
        array.Add(sc);
    }
}

The Solution

In the case shown above we know how many objects we require storage for beforehand. This means we can tell the array how many objects to store and only perform one memory allocation to set aside storage for the array. This has CPU benefits and also because there are no repeated calls to reallocate the memory the likelihood of fragmentation occurring diminishes dramatically. In many cases, completely removed from the scenario.

To set the size beforehand we need to call SetSize(size); and to place data in the array we no longer use Add();, but use SetAt(index, data); instead.

Here is the reworked example:

// read data from the serialization archive and store in an array

DWORD i, n;

ar >> n; 
array.SetSize(n);
for(i = 0; i < n; i++)
{
    someClass *sc;

    sc = new someClass();
    if (sc != NULL)
    {
        sc->load(ar);
        array.SetAt(i, sc);
    }
}

For large volumes of data the above implementation can be noticeably faster.

Caveats

When you preallocate memory like this you must be aware that if you don’t fill all locations in the array using SetAt() you may get errors when you call GetSize() to get the array size and GetAt(i) to retrieve data.

// read data from the serialization archive and store in an array
// we won't store all data, leaving some unused memory at the end of
// the array

DWORD i, n, c;

ar >> n; 
c = 0;
array.SetSize(n);
for(i = 0; i < n; i++)
{
    someClass *sc;

    sc = new someClass();
    if (sc != NULL)
    {
        sc->load(ar);
        if (sc->IsEmpty())
        {
            // discard
 
            delete sc;
        }
        else
        {
            array.SetAt(c, sc);
            c++;
        }
    }
}

GetSize() will return the size of the array you set when you called SetSize(), this is not necessarily the number of items in the array – this will come as a surprise to people used to adding data by calling Add().

To fix this, use FreeExtra() to remove any unused items from the end of the array. You can also use GetUpperBound() to find the largest index that is used by the array. The example below shows this.

// read data from the serialization archive and store in an array
// we won't store all data, leaving some unused memory at the end of
// the array

DWORD i, n, c;

ar >> n; 
c = 0;
array.SetSize(n);
for(i = 0; i < n; i++)
{
    someClass *sc;

    sc = new someClass();
    if (sc != NULL)
    {
        sc->load(ar);
        if (sc->IsEmpty())
        {
            // discard
 
            delete sc;
        }
        else
        {
            array.SetAt(c, sc);
            c++;
        }
    }
}

// make sure array.GetSize() returns the max number of items used

array.FreeExtra();
Share

Panorama Theme by Themocracy