[RESOLVED] Sometimes VirtualDub hangs with no error message

[admin]

My current theory is that there is a race condition such that this desired order:

VD instantiates DGDecodeNV
DGDecodeNV completes init and waits on the first GetFrame() call
VD asks for the first frame and Avisynth calls GetFrame()
DGDecodeNV is awoken by GetFrame() and starts decoding

...becomes:

VD instantiates DGDecodeNV
VD asks for the first frame and Avisynth calls GetFrame()
DGDecodeNV completes init and waits on the first GetFrame() call
DGDecodeNV hangs because it had not yet started waiting before the first GetFrame() call
VD hangs because the previous GetFrame() call never completed

It all depends on how fast VD is in asking for the first frame. My code currently does not exclude this scenario where VD is "too fast". I am testing a fix and will give it to you in a few minutes.

[admin]

Please re-download 2052 and update both DGIndexNV and DGDecodeNV. I am hopeful that there will be no further hangs. Please test and report.

Note that this slipstream includes another change (HEVC 10-bit support) that bumps the DGI file version, so you must re-make your index files.

[jpsdr]

Ok, thanks, i will. It will take a little while before i'll be able to report.

At first glance i found strange that you "leave Instance" before it's fully finished, so somehow i'm not surprise of this kind of things happen. I'm even surprised that it's not happened before...
It's just a little thought that immediatly pops when i've seen this part :

DGDecodeNV completes init and waits on the first GetFrame() call

=> Meaning VD continue in parallel because you've leaved/ended the instance part code, but the init is still not yet finished...
=> Meaning also i may have misunderstood what you've said with this...
But... I don't know how decode pluggins work, and neither if when VD instantiates a pluggin it also "continues" in parallel without waiting the end of instance (which would be strange).
Now, if it's because VD is "too fast", well, it's indeed an i7-6950x overclocked at 4,1GHz So... It increases the chances indeed of being "too fast".
And i've use lot of smiles...

[admin]

"At first glance i found strange that you "leave Instance" before it's fully finished"

To be fully finished I have to execute the wait, but if I wait I can't inform the VD thread to return. If Win32 had a clean way to determine if a thread is waiting I could solve this deterministically (by having the VD thread return when it sees my thread start waiting), but for now I have solved it heuristically. There is a simple managed solution but my code is not managed. And there apparently is a WMI API used by debuggers but it's total overkill.

The full explanation is involved and I'd have to show you the code for your full understanding. Suffice to say I believe I have solved the race condition. Your testing will determine if that is the case.

Cut me some slack; one doesn't always anticipate all race conditions, especially in a complex application.

[jpsdr]

Cut me some slack; one doesn't always anticipate all race conditions, especially in a complex application.

For having created a threadpool for my needs (and very probably very less complex than your application), i know...

[jpsdr]

I've never done decoding avs pluggin, only standard filters. I have the code of rawsource, very simple, but i've been able to check if something was different from a standard filter, and didn't see anything.
I have to assume that your dgdecodenv has a more complex structure than the standard "Create/Constructor/Getframe" calling, because i don't understand the part where "you wait", neither what you're waiting for (probably a WaitForSingleObject, but what is the object ?). In a classic standard "Create/Constructor/Getframe" calling, there is no wait, but obviously we are not in a classic case.
If i take a look at the code of dgindex, is it possible that i can see what you mean ? (I had it once, but i have to retrieve it to check).

After, you said it's solved, i'll take your word.

[admin]

I said I'm hopeful it is resolved.

Yes, DGDecodeNV is more complicated than a simple filter, mainly due to threading needs imposed by CUVID. Therefore, looking at DGIndex won't enlighten you.

Just think about the catch22: I'm not done until I wait, and if I wait I can't release VD to continue, so I have to release VD just before I execute the wait. It's a tiny window but you have been hitting it apparently. We'll see.

[jpsdr]

Just to understand globaly the situation :

A thread (or a process) will call A, start a thread (or a process) call B.
A "stop/wait" for B to finish before continue.
And B has finished only when is begining to wait.
My first idea of course will be to wait for event (With CreateEvent, WaitForSingleObject), and if you don't know it, take a look at what i've discovered yesterday when checking what kind of fuction exist SignalObjectAndWait :
https://msdn.microsoft.com/fr-fr/librar ... s.85).aspx
And i think using this function will prevent the issue, because it's probably "mono...." don't rememeber the word, but basicaly while executing SignalObjectAndWait, no other code can be run in parallel.
So basicaly, A create an event, start thread B, providing to it the handle of the event, and then A wait for this event.
B do its stuff, and when finished, call SignalObjectAndWait.
What do you think ? Unless it's allready what you've done, or you're telling me that B wait for multiple objects...

[admin]

I was not aware of the SignalObjectAndWait() function. Thanks for bringing it to my attention.

Unfortunately, my thread does wait on two events. I may be able to revise it to use a single event with a flag to indicate the invoker.

Strangely, SignalObjectAndWait() is said to not be atomic.

[jpsdr]

Euh... Stupid remark from me. Of course a waiting function can't be atomic... Otherwise it will be a little... troublesome to execute the code the function is waiting for. But probably that the "setting" part is. Anyway, my first guess is that it's probably the most proper way to do what you want to.
After, if your thread is waiting for 2 events with a waitformultipleobject, maybe you can just split it with 2 waitforsingleobject,
transforming waitformultipleobject in :
SignalObjectAndWait(xx,A)
waitforsingleobject(B)
without modifying anything and/or using a flag.
waitformultipleobject on 2 objects or 2 waitforsingleobject will produce the same result, it's just that waitformultipleobject is a little more optimized in performance, but both will produce the exact same synchronisation result. In your situation (or at least what i think i've understood from), just splitting with SignalObjectAndWait and waitforsingleobject will do the trick.

[admin]

I'll experiment with it today. Thank you for the ideas. Obviously a deterministic solution is preferred.

[admin]

I just slipstreamed my latest version of this fix. Please run with this one and let me know if it ever hangs again. Thanks.

[jpsdr]

Thanks for your work
Just out of curiosity, if my guess is roughly correct, according what you've said.
In the constructor (i was hesitating between creator/constructor, but constructeur seems more logical), you start a thread to handle all the CUDA things. This thread init things, and when done, wait for a getframe (and maybe other things). When this "CUDA thread" has finished its init and is waiting, you have to notify the constructor that it's finished and it can itself finish.
In the getframe, you trig an event to tell the "CUDA thread" that you've received a getframe.
So is it, roughly, something like this ?

[jpsdr]

Don't want to patronize or anything, just giving suggestion if needed, in case my guess is correct.
If not needed or don't want it, just ignore, absolutely no hard feelings.
My suggestion :

Code: Select all

constructor
{
  Doing stuff
  Creating E1 Event.
  Starting "CUDA Thread"
  // Waiting "CUDA Thread" finished
  WaitForSingleEvent(E1)
  Destroying E1 event
  (will not be needed anymore, don't waste resources for nothing).
}

Destructor
{
  Destroy E2 to E6
}

CUDA Thread
{
  Doing Init Stuff
  // Init is done, i'm ready, notify constructor
  SetEvent(E1)
  While whateever_exit_or_ending_filter_condition_is
  {
     WaitForMultipleEvent(E2,E3)
     ResetEvent(E2)
     ResetEvent(E3)
     // Don't know what E3 is, don't care, it will follow the same idea than GetFrame
     // Doing CUDA stuff
     // Notify GetFrame 1rst part of CUDA Stuff is done
     SetEvent(E4)
     // Waiting GetFrame notifying its part is done
     WaitForSingleEvent(E5)
     ResetEvent(E5)
     // Doing CUDA stuff
     // Notify GetFrame 2nd and last part of CUDA Stuff is done
     SetEvent(E6)
  }
  Doing Ending stuff
}

GetFrame
{
  Do Things
  // Notify CUDA it can do its stuff 
  SetEvent(E2)
  // Waiting CUDA finishing part of its job.
  WaitForSingleEvent(E4)
  ResetEvent(E4)
  Do things
  // Notify CUDA it can do the 2nd part
  SetEvent(E5)
  // Waiting CUDA finishing part of its job.
  WaitForSingleEvent(E6)
  ResetEvent(E6)
  Do things
}

In that case, even if Getframe is called between the tiny interval, it shouldn't matter, the event was already trigged, there will just be no wait. Ajust according the number of dispatching jobs between both.

[admin]

"If not needed or don't want it, just ignore"

OK.

Please advise the results of testing my last slipstream.

[jpsdr]

I've begin to use it. No issue but no big project also immediately (just a very few files). I'll report after a longer use, so may take a little while.

[admin]

Thanks, I appreciate your assistance.

[admin]

Marking this resolved. If not, please post again.

[jpsdr]

I've worked on several project with a lot more of files recently, and so far, so good.
So, it seems that the issue is indeed probably solved.

[admin]

Thank you, my friend, for the update, and for helping me to improve DGDecNV. In honor of your contributions, I have placed you in my Distinguished Member group.

[jpsdr]

Happened again...
But... DGIndexNV is for now second in suspect list, another is the first suspect.
Nevertheless, i would like to provide the ProcessExplorer stack dump when next hang, like last time.
With it, will you be able, in the first time, just checking if hang occured within DGIndexNV or not ?
Situation is not the same as before, this time, DGIndexNV is not the only one filter used.

[admin]

Probably. Let's see the traces.

[jpsdr]

Thanks, i'll post them when it occurs again. Lot and often issues sunday, unable to reproduce yesterday...

[admin]

Patience is a virtue.

See, even I can virtue signal.

[jpsdr]

It happened again, so, if you can check in the first place what you get from this stack information :

Code: Select all

VDub Process 1

ntoskrnl.exe!KeWaitForMultipleObjects+0xc0a
ntoskrnl.exe!KeAcquireSpinLockAtDpcLevel+0x712
ntoskrnl.exe!KeWaitForSingleObject+0x19f
ntoskrnl.exe!PoStartNextPowerIrp+0xbd4
ntoskrnl.exe!PoStartNextPowerIrp+0x187d
ntoskrnl.exe!KeAcquireSpinLockAtDpcLevel+0x91d
ntoskrnl.exe!KeWaitForMultipleObjects+0x26a
ntoskrnl.exe!NtWaitForSingleObject+0x40f
ntoskrnl.exe!NtWaitForSingleObject+0x77e
ntoskrnl.exe!KeSynchronizeExecution+0x3a23
ntdll.dll!ZwWaitForMultipleObjects+0xa
KERNELBASE.dll!GetCurrentProcess+0x40
kernel32.dll!WaitForMultipleObjects+0xb0
Filtres_JPSDR.vdf!VirtualdubFilterModuleInit2+0x1735
Filtres_JPSDR.vdf!VirtualdubFilterModuleInit2+0x3533
Filtres_JPSDR.vdf+0x16c13d
Filtres_JPSDR.vdf!VirtualdubFilterModuleInit2+0x4654
Veedub64.exe+0x448b5
Veedub64.exe+0x47680
Veedub64.exe+0x47db3
Veedub64.exe+0x5081e
Veedub64.exe+0x80640
Veedub64.exe+0x8340b
Veedub64.exe+0x6cdb5
Veedub64.exe+0x226674
kernel32.dll!BaseThreadInitThunk+0xd
ntdll.dll!RtlUserThreadStart+0x21

VDub Process 2

ntoskrnl.exe!KeWaitForMultipleObjects+0xc0a
ntoskrnl.exe!KeAcquireSpinLockAtDpcLevel+0x712
ntoskrnl.exe!KeWaitForSingleObject+0x19f
ntoskrnl.exe!PoStartNextPowerIrp+0xbd4
ntoskrnl.exe!PoStartNextPowerIrp+0x187d
ntoskrnl.exe!KeAcquireSpinLockAtDpcLevel+0x91d
ntoskrnl.exe!KeWaitForMultipleObjects+0x26a
win32k.sys!memset+0x7cc7
win32k.sys!memset+0x7d89
win32k.sys!W32pArgumentTable+0xab22
ntoskrnl.exe!KeSynchronizeExecution+0x3a23
USER32.dll!WaitMessage+0xa
Veedub64.exe+0x1f07ef
Veedub64.exe+0x13bf8e
Veedub64.exe+0x226ff3
Veedub64.exe+0x227087
kernel32.dll!BaseThreadInitThunk+0xd
ntdll.dll!RtlUserThreadStart+0x21