[RESOLVED] Issue with Jerry Maguire THD stream

Support forum for DGDecNV
User avatar
Boris
Posts: 92
Joined: Sun Nov 10, 2019 2:55 pm

Re: Issue with Jerry Maguire THD stream

Post by Boris »

Rocky wrote:
Wed Jun 24, 2020 8:33 pm
Gonna rip and test tomorrow.
He he he, what if do not wake up tomorrow?
User avatar
Curly
Posts: 712
Joined: Sun Mar 15, 2020 11:05 am

Re: Issue with Jerry Maguire THD stream

Post by Curly »

My mammy always said tomorrow is another day.
Curly Howard
Director of EAC3TO Development
DAE avatar
renols
Posts: 149
Joined: Tue Feb 22, 2011 2:34 am

Re: Issue with Jerry Maguire THD stream

Post by renols »

Rocky wrote:
Wed Jun 24, 2020 8:33 pm
Disk set arrived. Gonna rip and test tomorrow. Long day coaching -- real swimmers in real water -- YAY! I'm really intrigued by this as usually I can figure things out by code inspection. I can't even imagine what is happening. No fear, the debugger reveals all.
Will be interesting to see what your findings will reveal.

renols
User avatar
Rocky
Posts: 3555
Joined: Fri Sep 06, 2019 12:57 pm

Re: Issue with Jerry Maguire THD stream

Post by Rocky »

Ripped and duplicated the issue. The problem is that it's been assumed that each THD frame (major or minor) is a single PES packet and is marked by payload_unit_start_indicator = 1. This has been true for all the streams seen up until this one. This stream packs multiple THD frames into a single PES packet without any apparent rhyme or reason. It does seem to be true that every major starts a new PES packet but that is not enough. As a result, the code fails to properly detect frames as major or minor, and so sometimes falsely detects a double major and deletes frames. Just disabling the mechanism when there is only one M2TS is not good enough because this can still happen to individual M2TS files in the case of multiple M2TS files.

Going to have to buffer and parse at the ES level because the transport level gives no indication of the start of minor frames. The minor frame boundaries are needed to be able to delete minor frames for gaps correction. Also to be considered is whether to return to Bresenham plus safe minor deletion, or stick to the double major idea. That could always be done later if needed.
User avatar
Sherman
Posts: 576
Joined: Mon Jan 06, 2020 10:19 pm

Re: Issue with Jerry Maguire THD stream

Post by Sherman »

Rocky wrote:
Fri Jun 26, 2020 6:12 am
it's been assumed that each THD frame (major or minor) is a single PES packet and is marked by payload_unit_start_indicator = 1
Rocky, that was my assumption. So can I fix this? It hurts after getting my spurs.
Sherman Peabody
Director of Linux Development
User avatar
Rocky
Posts: 3555
Joined: Fri Sep 06, 2019 12:57 pm

Re: Issue with Jerry Maguire THD stream

Post by Rocky »

Go for it, Sherman! And welcome back. What did you learn from Mr. Balzac?
User avatar
Sherman
Posts: 576
Joined: Mon Jan 06, 2020 10:19 pm

Re: Issue with Jerry Maguire THD stream

Post by Sherman »

A lot.
Sherman Peabody
Director of Linux Development
User avatar
Curly
Posts: 712
Joined: Sun Mar 15, 2020 11:05 am

Re: Issue with Jerry Maguire THD stream

Post by Curly »

Sworn to secrecy, eh? Big mystery. Lemme tell you some secrets. Nyuk nyuk nyuk.
Curly Howard
Director of EAC3TO Development
User avatar
Sherman
Posts: 576
Joined: Mon Jan 06, 2020 10:19 pm

Re: Issue with Jerry Maguire THD stream

Post by Sherman »

Hey Rocky. I have fixed up the THD demuxing. But rather than do the THD buffering and ES parsing on-the-fly with demuxing, I instead do a pass at the end on the demuxed THD file to fix up the THD gaps. The little GUI box shows "Fix THD gaps" during this time. I tried to do things on-the-fly but everything got ridiculously complex and slow. My way is clean, simple, and fast. Can you make a test release?
Sherman Peabody
Director of Linux Development
User avatar
Rocky
Posts: 3555
Joined: Fri Sep 06, 2019 12:57 pm

Re: Issue with Jerry Maguire THD stream

Post by Rocky »

Sounds great, Sherman. Gotta love clean, simple, and fast. I figured all that buffering and parsing was going to be a PITA, so I was happy when you offered to fix this. ;) That you found a better way is frosting on the cake. I'll finish my coffee and danish and then make a test release.
User avatar
Rocky
Posts: 3555
Joined: Fri Sep 06, 2019 12:57 pm

Re: Issue with Jerry Maguire THD stream

Post by Rocky »

Here is a test version with MVC and the THD fix.

http://rationalqm.us/dgdemux/binaries/D ... c_test.rar

Feedback will be appreciated.
User avatar
Sherman
Posts: 576
Joined: Mon Jan 06, 2020 10:19 pm

Re: Issue with Jerry Maguire THD stream

Post by Sherman »

Rocky wrote:
Sun Jun 28, 2020 8:48 am
I figured all that buffering and parsing was going to be a PITA, so I was happy when you offered to fix this. ;) That you found a better way is frosting on the cake.
Thanks, Rocky. My way also has the advantage that when gaps processing is not required (it's disabled or there is only one M2TS) the demuxing is somewhat faster. To be honest, buffering and parsing at different levels is a real performance killer. Yes, I know DGIndexNV is designed that way, but it almost has to be that way as the relation between the video and everything else is very important and must be maintained, especially to honor the project range, etc. But for DGDemux we should be able to get rid of all that. I see you started implementing my way and got a giant performance boost (350%). I can finish that off if you like.
Sherman Peabody
Director of Linux Development
User avatar
Rocky
Posts: 3555
Joined: Fri Sep 06, 2019 12:57 pm

Re: Issue with Jerry Maguire THD stream

Post by Rocky »

Go for it, Sherman!

My life has become so easy. :lol:

Bullwinkle: ace system architect and forum enforcer
Sherman: ace coder and apprentice system architect
Rocky: ace debugger and occasional coder
Curly: our conscience
Levi: spiritual and creative foundation
admin: bean counter
DG: founder and benefactor
DAE avatar
Guest

Re: Issue with Jerry Maguire THD stream

Post by Guest »

Quick test
Movie with one M2TS and 0 ms delay between audio and video
video 1 h 48 min 54 s 243 ms
thd... 1 h 48 min 54 s 237 ms
ac3... 1 h 48 min 54 s 240 ms
User avatar
Bullwinkle
Posts: 338
Joined: Thu Sep 05, 2019 6:37 pm

Re: Issue with Jerry Maguire THD stream

Post by Bullwinkle »

Sherman! C'mon, you can't do gaps correction when the stream is not enabled for demuxing. Otherwise, highly impressive.

@gonca

Thank you for your testing. Looking good.
User avatar
Sherman
Posts: 576
Joined: Mon Jan 06, 2020 10:19 pm

Re: Issue with Jerry Maguire THD stream

Post by Sherman »

Bullwinkle wrote:
Sun Jun 28, 2020 10:16 am
you can't do gaps correction when the stream is not enabled for demuxing
Oh, yeah, thanks. I'll give Rocky an update and he can update the test version.
Sherman Peabody
Director of Linux Development
User avatar
Rocky
Posts: 3555
Joined: Fri Sep 06, 2019 12:57 pm

Re: Issue with Jerry Maguire THD stream

Post by Rocky »

Sherman wrote:
Sun Jun 28, 2020 10:18 am
I'll give Rocky an update and he can update the test version.
Test version updated online.
DAE avatar
renols
Posts: 149
Joined: Tue Feb 22, 2011 2:34 am

Re: Issue with Jerry Maguire THD stream

Post by renols »

Hi.

Have given the test version a run.

Jerry Maguire. The thd file is now binary compared the same as the old dgdemux version with gap disabled. So disabling gap is no longer needed to get a good thd file.

Cars 2. The file created by dgdemux test version and dgindexnv (210) is binary compared the same, and comparing flac audio run time to video runtime shows only a difference of 9 ms.

On Cars 2 the dgdemux test version spends some time on "THD Gap fixing", but guess that is how it is supposed to be, when doing the fixing at the end rather than on the fly. Wether it is faster than what dgindexnv (210) does I don't know.

renols
User avatar
Sherman
Posts: 576
Joined: Mon Jan 06, 2020 10:19 pm

Re: Issue with Jerry Maguire THD stream

Post by Sherman »

Hello Mr renols. We haven't met but I have heard wonderful things about you. Great that you found no issues. Theoretically, the new way is faster in all use cases. However, that is moot as we move to the new 350% faster architecture.
Sherman Peabody
Director of Linux Development
User avatar
Rocky
Posts: 3555
Joined: Fri Sep 06, 2019 12:57 pm

Re: [RESOLVED] Issue with Jerry Maguire THD stream

Post by Rocky »

Was reviewing old threads for marking RESOLVED. Just wanted to add here that renols was correct about the fixing process being very slow. Fortunately, Sherman repaired that not too long ago. The process for a typical disk was reduced from 60 seconds to 3 seconds. He's a good boy.
Post Reply