How to prevent re-entrancy of WPF event handler during ActiveX method call?

<tldr> try the code below in Attempt #3. There was nothing on TV when I sat down to write this.</tldr>

Thanks for the clarification! I see that there is only one thread here; and also, since the CloseShelf frame is still on the stack, it looks like the COM call is actually blocking.

From the stack trace, it looks like the com object is calling GetMessage or PeekMessage API (or if it's VB6, DoEvents or similar) which will check the message queue and PROCESS messages on it - irregaurdless of if it will cause re-entrancy. AKA 'pumping the message queue' - but if it uses peekmessage, it won't block if there are no messages, but will still execute the messages. In your stacktrack these calls could be hidden in the invisible native part. The stack trace actually proves that somehow, the COM call is calling back into your code! It looks like you are aware of this. If it's a bit foggy for some reader here are a couple links:

http://discuss.joelonsoftware.com/default.asp?joel.3.456478.15
http://en.wikipedia.org/wiki/Event_loop

Cooperative multitasking (one message loop for the whole OS like win3.0):http://en.wikipedia.org/wiki/Computer_multitasking#Cooperative_multitasking.2Ftime-sharing

The quote 'voluntarily ceded time...' is actually done by these calls. And it STILL happens all the time on the GUI thread of every windows app!

Since the actual call is in the COM, if you can't edit it, you still have to code around it. It is probably just this one method (hopefully).

Sometimes programs and components check the message loop on purpose, to times to allow things to respond, or repaint the screen. Just like this poster is trying to do:

Is there a function like PeekMessage that doesn't process messages?

The quote about 'The system may also process internal events' is buring you here.

note where you say 'the programmer does not expect a method call to be async' - it isn't async, or the stack trace would look different. it's 'recursing' back into your code. As an old Win3.1 programmer this was the hell we dealt with for EVERY app in the system!

I found that link googling for 'check the message queue peekmessage' while trying to see if Get/Peekmessage, etc. could be prevented from processing messages. You can see from this documention...

http://msdn.microsoft.com/en-us/library/windows/desktop/ms644943(v=vs.85).aspx

...that sending something like PM_QS_INPUT | PM_QS_PAINT would prevent the problem. Unfortunately, since you aren't calling it, you can't change it! And I didn't see any way to set a flag to control subsequent message processing from later calls.

If a reader is still confused (if not skip this code) ... For a simple example, make a VB Winforms App and make one button and double click it and paste this code in - (I say VB because application.doevents is the handiest way to invoke this nasty message queue check):

    For i As Integer = 0 To 20        Text = i.ToString        System.Threading.Thread.Sleep(100)        Application.DoEvents()    Next

Now click the button. Note you can enlarge the window and the background repaints - as the doevents allows these events to happen by checking the message queue (REM out the doevents and it will 'wait' untill the count is done; also try clicking 3x and you will get 3 counts in a row).

NOW... the kicker. Click the button w/ Doevents NOT commented out. Click the button 3 times - the countdowns interrupt each other, then resume as the previous one finishes. You can Pause the IDE and see 3 click callstacks.

Yummy!

I also see that you tried a couple things to stop messages or events from being processed. If the code that fires the EventShelfClosed is getting called from the re-entrancy caused by a PeekMessage or 'Doevents', however, that may not effect it.

Note this practice has it's detractors: http://www.codinghorror.com/blog/2005/08/is-doevents-evil-revisited.html

Best would be to change the COM so it doesn't make any API calls that are checking the message loop.

Good luck with that!

Another way to change it would be to move from events controlling EventShelfClosed, and call it explicity after the call to CloseShelf is exited (since the com call is really happening). Unfortunately your program's architechture probably won't allow that without major changes and/or increased cohesion and other stuff that dirties pretty models (not fashion models mind you :-).

A different way would be to make a new thread object pointed to a function that makes the com invoke, then start it, then join it, in hopes that anything like PeekMessage would not find a message pump on the new thread and therefore not interfere with things. It looks like a couple of you attempts involved this type of thing. Unfortunately if the COM Peeks for messages, and there is no message pump on the thread, kaboom. It will probably blow up instead of just ignoring things. Sounds like that's what happened. Further more, if the COM relies on other items that should only be accessed from the GUI/messagepump thread, you're in trouble with cross-thread calls (which would certainly be the case if the COM interacts with the UI or any UI objects).

If you can't stop the message queue from being checked, or prevent EventShelfClosed from firing until later, you have no choice but to let the EventShelfClosed get called. But what you can do is cause it to wait, and then fall thru when the CloseShelf is finished.

So you still have to have a class-level boolean field set/unset by CloseShelf so EventShelfClosed will know that it is running.

Unfortunately just checking this in a while loop, even with a sleep, will block the single thread you have, and freeze the app. You may just try to have EventShelfClosed re-raise itself and exit the function as long as the bool is set; but since RaiseEvent stays inside of managed, runs the code immediately, and does not check the message queue it will in crash with a stackoverflow. If you can figure out how to re-raise EventShelfClosed by calling the PostMessage API (not SendMessage, that runs it right away) - that will keep putting on the GUI thread's message queue as many times as the COM call will make windows check it. Unless said COM waits for the queue to be empty for some stupid reason - another lockup. Not recommending.

Soo... You can fight fire with fire. Here I am implementing another message-checking loop to allow your events to happen while you wait for the bool to clear.

Note this is only going to fix this one problem in this one case. Auditing all calls ... this isn't a magic bullet. My guess is there is none. Very messy and this is a total hack.

Attempt #3

It's not really attempt #3, it's more like possibility #8. But I referenced this in my old answer and am too lazy to edit that.

Boolean InCloseShelffunction CloseShelf(...)    InCloseShelf=True;    try    {         com call and all else     }     finally         InCloseShelf=Falsefunction EventShelfClosed(...    while (InCloseShelf)    {         DoEvents     }

Now of course there is no DoEvents in WPF, it's in winforms' 'application'. This blog has an implementation

http://dedjo.blogspot.com/2007/08/how-to-doevents-in-wpf.html

void DoEvents(){ DispatcherFrame f = new DispatcherFrame(); Dispatcher.CurrentDispatcher.BeginInvoke(DispatcherPriority.Background,  (SendOrPostCallback)delegate(object arg) {     DispatcherFrame fr =  arg as DispatcherFrame;     fr.Continue=True; }, f); Dispatcher.PushFrame(frame); }

Untested, of course! Note this is from the correction in the comments:

static void DoEvents(){DispatcherFrame frame = new DispatcherFrame(true);Dispatcher.CurrentDispatcher.BeginInvoke(DispatcherPriority.Background, (SendOrPostCallback) delegate(object arg){var f = arg as DispatcherFrame; f.Continue = false;}, frame);Dispatcher.PushFrame(frame);} 

Or you could always reference WinForms and call Application.DoEvents.

I'm guessing you already know this, but you're in a bad spot right now. If this doesn't do it, Good luck! If you find a better way please update the post, because, well, evil hacks like this suck, but now you can see why people say to check the message queue sparingly!

I have dealt with similar problems in the past although not from WPF.

In a win32 application the recommended approach was to use IMessageFilter::MessagePending - this could be configured to say what types of messages were allowed to be handled when an outgoing STA call was already in progress. Here you had to be careful to ensure that any callbacks from your callee object were accepted and handled.

http://msdn.microsoft.com/en-us/library/windows/desktop/ms694352(v=vs.85).aspx

In WPF this is not available. I think your approach to using another thread is the correct way to go.

In principle you want your main thread to block on a child thread. The child thread can then make the outgoing COM call. You probably want to make the child thread an STA to avoid introducing other unknown issues. It is important that messages are pumped on the child thread and that any pointers or types are correctly marshalled as the child thread will be in a different COM apartment. Reentrancy is avoided because callbacks are the only thing that would attempt to message the thread that is pumping.

In WPF I believe that a Dispatcher should provide all the functionality that you need.

I'm not sure why your attempt to do this using a Dispatcher failed - it might be related to this known issue:http://support.microsoft.com/kb/926997

You have the answer!

Is it just an old hack? No, it's not, it's standard operating procedure when any type of re-entrancy is involved. This has worked flawlessly for me in more cases than I can remember, from humble single panel VB3 crud popups to huge MVVM/DDD wannable enterprise management apps.

It is what you mentioned: 'use custom locking like 'static bool locked = false; if (!locked) { locked = true; InvokeMethod(); ...; locked = false; }'.

EDIT

Note the comments from the OP. OK so this won't solve the problem! The 2nd event isn't a spurious click; it is a different event, critical to the proper operation of the system.

Please see my next answer for a few more attempts. #3 is the ugliest but should work.