Posts tagged: critical section

Detecting Abandoned Critical Sections

By , March 6, 2020 4:24 pm

Multithreading is a powerful way to improve the processing throughput and responsiveness of your software. We use it to great effect at Software Verify. In order to manage multithreading successfully it’s necessary to use some form of synchronization between each thread that wishes to read/write data. Deadlocks can result. The main cause of deadlocks is two or more locks (critical section being an example) accessed in different orders on each thread. This has been the subject of much writing, so for now I won’t repeat that topic here.

There is another cause of deadlock which is less well known. The abandoned critical section.

In this article I’m going to describe how to detect abandoned critical sections. But first I need to describe them to you and explain how abandoned critical sections get created.

What is an Abandoned Critical Section?

An abandoned critical section is a critical section that has been locked but then the thread that owns the lock ends without unlocking the critical section. This creates a critical section that cannot be unlocked, and is thus permanently locked. If any other thread attempts to enter the critical section it will wait forever, in a deadlock caused by an infinite wait.

How does this happen?

There are several ways that a critical section can become abandoned.

  • Incorrect code.
  • Incorrect exception handling.
  • Terminate Thread.

Incorrect code

This is where the thread code enters a critical section to do some work and forgets to unlock the critical section. Then the thread exits. If you use object oriented code (CSingleLock for example) to manage the lifetime of critical section ownership then this problem should never happen. But if you manually control the locking, using say, CCriticalSection::Lock() and CCriticalSection::Unlock(), or EnterCriticalSection(&cs) and LeaveCriticalSection(&cs) then it’s possible for you to forget to leave a locked CS, or for a logic failure to result in a critical section not being locked.

If you’re using object oriented synchronization locking methods you might want to look at Thread Lock Checker to automate checking for some simple and common errors that can happen.

DWORD doThread(void	*param)
{
	EnterCriticalSection(&dataCS);
	
	doWork(data);
	
	return 0;	// forgot to call LeaveCriticalSection(&dataCS);
}

Incorrect exception handling

This is where some code in a thread is protected by an exception handler (you’re calling a 3rd party library, or working with data of unknown integrity) and a critical section is locked when an exception is thrown. In an ideal world the exception handler will leave that locked critical section. Unfortunately the writer of the exception handler may not known about the critical section, or they may have forgotten about it – either way the locked critical section doesn’t get unlocked. As with the previous case, if you use object oriented access to critical sections (CCriticalSection, CSingleLock) the process of unwinding the stack during the exception handling should automatically unlock these locks. This won’t happen if you’re using CRITICAL_SECTIONs with the Win32 API.

DWORD doThread(void	*param)
{
	__try
	{
		EnterCriticalSection(&dataCS);
	
		doWork(data);	// something inside here throws an exception. 
	
		LeaveCriticalSection(&dataCS);
	}
	__except(EXCEPTION_EXECUTE_HANDLER)
	{
		// forgot to call LeaveCriticalSection(&dataCS);
	}
	
	return 0;	
}

Terminate Thread

This is where a thread that is doing some work that has accessed some critical sections is killed by another thread calling TerminateThread(). There are occasions where TerminateThread() can be useful, but this is a last ditch method for dealing with threads. If your code is using TerminateThread() to manage your own threads why not spend some time to work out how not to use TerminateThread and to make your threads end normally (by exiting the thread or calling ExitThread()).

// correctly written thread

DWORD doThread(void	*param)
{
	EnterCriticalSection(&dataCS);
	
	doWork(data);
	
	LeaveCriticalSection(&dataCS);
	
	return 0;
}

void mainThread()
{
	HANDLE	hThread;
	DWORD	threadId;
	
	hThread = CreateThread(NULL, 0, doThread, NULL, 0, &threadId);
	if (hThread != NULL)
	{
		doSomeWork();
		
		TerminateThread(hThread, 0); // this is a bit brutal
		CloseHandle(hThread);
	}
}

How to detect Abandoned Critical Sections?

We have two ways to detect Abandoned Critical Sections.

  • Thread Wait Chain Inspector
  • Thread Validator

Thread Wait Chain Inspector

Thread Wait Chain Inspector is a free software tool that we wrote that uses the Win32 Wait Chain API to identify various wait chain states of the locks and waits in a given application. Just select the application in question and look at the results.


This tool tells you process ids and thread ids, but it can’t give you symbols, filenames and line numbers. It will provide thread names if you’re working on Windows 10 and you’ve named your threads using the SetThreadDescription() API.

Thread Validator

Thread Validator is our thread analysis software tool for analysing thread synchronization problems, deadlocks, busy locks, slow locks, contended locks and recursing locks. We’ve recently added some reporting options to Thread Validator will help you identify the location of abandoned critical sections.

I’ve used the tvExample demonstration application that ships with Thread Validator (you’ll need to build) to deliberately create two abandoned critical sections. From the test menu choose “Exit thread with a locked critical section” and “Terminate thread with a locked critical section”.

The summary display will show an abandoned count of 2 in the Errors panel.


The various locks displays will colour the abandoned thread dark purple and list the Lock status as Abandoned


If you click the Abandoned bar in the Errors panel, the display will move to the Analysis tab and the callstacks for the abandoned critical sections will be displayed.


Expanding each entry reveals the callstacks so that you can see where see where each critical section is abandoned. Note that each entry shows two callstacks. The first is where the critical section was created. The second is where the critical section was abandoned. You can expand any entry on any callstack to see the source code.

Abandoned because of thread exit


Abandoned because of TerminateThread()


Expanding the callstack entries to reveal the source code…


Conclusion

Abandoned Critical Sections are bad news. They cause deadlocks. But they don’t need to be hard to track down when you’ve got the right tools to put to work.

Thread Wait Chain Inspector

By , August 22, 2019 10:53 am

Since Windows Vista the Windows operating system has included functionality to iterate across the waiting objects that form a chain between threads. I’m waiting for thread A, which is waiting for thread B, which is waiting for process Y. That sort of thing. Waits come in the form of EnterCriticalSection, WaitForSingleObject, WaitForMultipleObjects, etc. All documented in Microsoft’s Synchronization API. If you get these waits wrong, you can get deadlocks, or waits that wait forever. Either way, it’s game over for your program if that happens.

The Wait Chain Traversal API was added with windows Vista, but only made public recently. Prior to the API, access to the Wait Chain API was only via Resource Monitor, and more recently via Task Manager. A detailed article by a Microsoft field engineer, faik hakan bilgen, documents the history of the Wait Chain user interfaces and then provides a console program (with source code on github) to provide a wait chain dump to a text file. Unfortunately this isn’t very easy to use as it relies on decoding thread ids and process ids to understand what is happening. Also because it’s a text file, to get an update you need to run the tool again.

We decided to take inspiration from our Thread Status Monitor tool and create a version specifically for wait chains – Thread Wait Chain Inspector.


Select the process you are interested in in the upper window. Wait chains for each thread are shown in the window below. Select a thread and it and any related threads (in the same wait chain) will be highlighted in yellow. Deadlocked threads are shown in red. Process names are displayed and thread names are taken from the GetThreadDescription() API (Windows 10 only).

If you wish to debug a process you can create a minidump for any process in a wait chain. Right click on the process of interest in the wait chain and Create a Minidump….

Thread Wait Chain Inspector is a free tool, complementing our other threading tools, Thread Validator, Thread Lock Checker and Thread Status Monitor.

Improving how you use CSingleLock

By , April 2, 2010 9:44 pm

Thread Lock Checker Logo

This posting covers a brief background:

  • Win32 critical sections.
  • How CCriticalSection and CSingleLock can be used instead of Win32 critical sections.
  • An improved way to use CSingleLock.
  • Some ways CSingleLock can be used that do not have the desired effect.

Critical Sections in Win32

The Win32 API uses InitializeCriticalSection, EnterCriticalSection, LeaveCriticalSection and DeleteCriticalSection to manage critical sections (CRITICAL_SECTION). Using these APIs is not particularly hard, but nonetheless it is possible to use critical sections that have not been initialized or that have been deleted. It is also possible to forget to leave a critical section that has been entered. In addition, any exceptions that get thrown may result in a critical section being left in its locked state.

This can cause serious performance problems as locks are held for too long, or in the case of a lock not being released, it can prevent other threads gaining access to the resource the lock was protecting, possibly resulting in a deadlock.

Example Win32 usage (assume critical section initialized in a different function):

void someFunc()
{
	doWork();

	EnterCriticalSection(&cs);

	doWorkEx();

	LeaveCriticalSection(&cs);
}

Why use CSingleLock and CMultiLock?

When using critical sections in MFC you use the CCriticalSection class instead of CRITICAL_SECTION objects.

You can directly call Lock() and Unlock() on the CCriticalSection, but it is recommended that you use CSingleLock and CMultiLock to manage your CCriticalSection objects.

The benefits of using a class such as CSingleLock (and its related class CMultiLock) are that:

  • The CSingleLock manages the activities of entering and leaving the critical section – you do not have to think about the critical section at all.
  • Any CCriticalSection object used with CSingleLocks will automatically be initialized before the CSingleLock gets to work with it.
  • The CSingleLock is automatically unlocked (if it was locked) when the CSingleLock is deleted and thus the CCriticalSection that was associated with this CSingleLock is not held locked .
  • If an exception is thrown, C++ objects are cleaned up by the exception handling chain, thus automatically deleting any CSingleLock objects and releasing any locks they hold.
  • CSingleLock can be used to lock and unlock critical sections just like the old Win32 methods, allowing for easy conversion of code from Win32 style to CSingleLock style.
  • It is possible to create a CSingleLock that is automatically locked. This is very useful for set-and-forget critical section management. Just put the CSingleLock in the right place and you can ignore it in the rest of the code. Very neat, convenient and elegant.

One way of using CSingleLock

As described above a typical style of using CSingleLocks echoes the Win32 style of using critical sections.

void someFunc()
{
	CSingleLock	lock(&csSect);

	doWork();

	lock.Lock();
	doWorkEx();
	lock.Unlock();
}

As you can see, the CSingleLock lock manager is created, the doWork() function is called outside of the protected area, the lock is locked, doWorkEx() is called, then the lock is unlocked. This is a very similar style of writing to Win32 equivalent.

A better way of using CSingleLock

The problem with the previous way of using CSingleLock is that most of the power and convenience of CSingleLock is ignored. Lock management has been made explicit via calls to Lock() and Unlock(). This means there is potential for forgetting to lock the CSingleLock, or for unlocking the CSingleLock later than desirable.

An improved way of using CSingleLock is to always create CSingleLocks in the locked state and to create CSingleLocks as close to the resource they are need to protect.

The following example shows the same function written using a CSingleLock that is automatically locked, created just before it is required and automatically destroyed at the end of the function.

void someFunc()
{
	doWork();

	CSingleLock	lock(&csSect, TRUE);

	doWorkEx();
}

If I wanted to some more work after doWorkEx() but I didn’t want that protected by the lock I could do it by using C++’s scoping capabilities. I simply create a new scope and place the CSingleLock in there. At the end of the scope the CSingleLock is destroyed and the lock is unlocked.

void someFunc()
{
	doWork();

	{
		CSingleLock	lock(&csSect, TRUE);

		doWorkEx();
	}

	doMoreWork();
}

Some problems we have seen…

During the development of code for the software tools at Software Verification and our private tools we’ve found a few interesting mistakes. Mistakes often made not through poor design, but simply a typing oversight or mistake, possibly due to tiredness of the person working on the code – the type of mistake you can only put down to the fact that humans do make mistakes, not matter how talented they are in any given field.

Where possible we like to use the CSingleLock lock(&csSect, TRUE) automatic locking style coupled with tight scoping to make the lock lifetime short. As a result we are interested in find the following coding constructs which will result in errors in expected behaviour in our software:

  • CSingleLock created without a lock argument. This defaults to an unlocked CSingleLock.
    CSingleLock	lock(&csSect);
  • CSingleLock created with a FALSE lock argument. This is an unlocked CSingleLock.
    CSingleLock	lock(&csSect, FALSE);
  • CSingleLock created with a variable declaration. This compiles but creates a lock that is immediately destroyed. Any of these three variants are interesting as none of the are useful, but all compile OK.
    CSingleLock(&csSect);
    CSingleLock(&csSect, FALSE);
    CSingleLock(&csSect, TRUE);

Thread Lock Checker

Thread Lock Checker

The problem with the examples we show above is that looking for them is hard work because humans often read what they expect to read (this is part of our predictive pattern recognition built into how we process shapes and text). As a result you may be looking right an error and not see it, but you may see the error the next time you come to the code (having forgotten all about it).

To aid in the discovery of these types of lock usage (for both CSingleLock, CMultiLock and any named classes that have the same style of behaviour) we have written a software tool, Thread Lock Checker.

We use Thread Lock Checker before we release any software. We use Thread Lock Checker to scan our codebase looking for any mistakes not identified by our software engineers. Its a very useful tool. We hope that you will also find Thread Lock Checker useful. Please check back next week for your free download.

We will be releasing Thread Lock Checker during the week of 5 April to 9 April.

Panorama Theme by Themocracy