[RESOLVED] Issue with Jerry Maguire THD stream

[renols]

Hi.

I was not sure wether to post here or in dgdemux area, since this is not seamless branching related, but still show the same problem with thd.

I am having issues with the Jerry Maguire 4k disk from the box set here:

https://www.amazon.com/Columbia-Classic ... 233&sr=8-3

I am not sure if the disc is available on its own.

What I have done is this.

Demux using dgindexnv. Result was that the audio for the thd file gets progressively out of sync. The AC3 audio is in perfect sync.

I then used makemkv to create a mkv file. Playing this mkv file shows perfect sync on both thd and AC3 audio.

I demuxed the two mkv files to get the raw thd files, and I then created a FLAC file from each of them.

The thd file from dgindexnv showed a lot of stuff that looked like the thd file was in several parts, while the thd file from makemkv just went through without any extra output. I used eac3to to create the FLAC files.

The playlist contains just one m2ts file, so seamless branching should not be the problem.

I then loaded the two FLAC files into Audacity and what I got was this.

Video frames from the dgi file: 199344

Runtime based on frames: 8314.314 s

dgindexnv runtime: 8313.792 s

makemkv runtime: 8314.307 s

The dgindexnv file is out of sync by more than 0.5 s.

Playing the mkv file with the dgindexnv thd file and putting in a delay of 520 ms at the end the audio plays in perfect sync. Not sure whats up with the audio on this one.

Sorry to be the bearer of bad news, but looks like we need another round of thd stuff

If you want me to send the thd files, the FLAC files or whatever just let me know. Will be glad to help.

renols

[Rocky]

I split this to its own thread.

I have ordered the disk set as it seems a great addition to my collection and donations have been robust. It should arrive in a few days. Meanwhile...

Demux using dgindexnv. Result was that the audio for the thd file gets progressively out of sync. The AC3 audio is in perfect sync.

Hard to believe, but OK. Only thing I can think of is a frame rate issue.

The thd file from dgindexnv showed a lot of stuff that looked like the thd file was in several parts, while the thd file from makemkv just went through without any extra output. I used eac3to to create the FLAC files.

What does that mean exactly? Can you show the EAC3TO output? I can't get much information from "a lot of stuff".

Video frames from the dgi file: 199344
Runtime based on frames: 8314.314 s
dgindexnv runtime: 8313.792 s

I don't know what you mean by dgindexnv runtime and where you got it. Can you post a link to the DGI file?

The dgindexnv file is out of sync by more than 0.5 s.

You mean at the end of the file? You said it got progressively out of sync.

[Rocky]

Had another theory. Perhaps the THD gaps correction is erroneously firing when it should not (no seamless branching here). If the file has double majors it could happen. I've supposed that only happens at file boundaries but could it happen in a single file other than at the start of the file? Theoretically, no. I may have to return to the Bresenham method together with safe last minor deletion.

Can you give me the THD file from MakeMKV? If you need my FTP access let me know.

Also, and this is the biggie, please try DGDemux but with gaps processing disabled.

[renols]

Hi.

Sorry about the "lot of stuff" statement

eac3to output looks like this:

TrueHD (Atmos), 7.1 channels, 48kHz
Decoding with libav/ffmpeg...
Encoding FLAC with libFlac...
Creating file "audio.flac"...
[truehd @ 025e2be0] Lossless check failed - expected 84, calculated c9.
[truehd @ 025e2be0] Lossless check failed - expected 0d, calculated 55.
-[truehd @ 025e2be0] Lossless check failed - expected 72, calculated e4.
-[truehd @ 025e2be0] Lossless check failed - expected 24, calculated ef.
---[truehd @ 025e2be0] Lossless check failed - expected 8e, calculated 12.
[truehd @ 025e2be0] Lossless check failed - expected b3, calculated 3f.

And then it just continues. Not sure how many of those there actually is.

At the start it is in sync, and at the end it is 0,5 s out of sync. Thats why I wrote progressively, since I think it gets more and more out of sync as the movie plays.

The dgi file can be downloaded from here: https://filebin.net/q9samkizhjgx6mzi

The dgindexnv runtime is the time shown in Audacity when I opened the flac file in it.

I will try to demux with the gaps processing disabled and get back to you.

I have the makemkv thd file, and the thd file from dgindexnv. I can send one or both, but will need the FTP info to do so.

renols

[Rocky]

I will try to demux with the gaps processing disabled and get back to you.

This is the most important thing!

I have the makemkv thd file, and the thd file from dgindexnv. I can send one or both, but will need the FTP info to do so.

I'll PM you he details. I'd like both.

[renols]

Hi.

Just did the demux without gap processing. Couldn't find the setting in dgindexnv, so I used dgdemux.

The result is that after converting it to FLAC using eac3to, which didn't show a single error, the runtime of the audio is now
100% the same as the makemkv file.

makemkv 2 h 18 m 34,307 s = 8314.307 s

dgdemux with no_gaps_processing = 8314.307 s

Looks like your suspission about gaps processing being triggered eventhough there is only one file, is probably correct.

The FLAC file created from the makemkv thd file and the FLAC file created from the new dgdemux thd file are bit identical. At least running an FC /B on the two files, say that they are identical.

Guess there is no idea in sending you the makemkv thd file, since it is identical to the one created by dgdemux without gap processing.

The speed to the FTP seems to be quite good. So I am sending the dgindexnv and the makemkv thd files.

renols

[Rocky]

Great, thanks for the debugging. I can start fixing this right away but will confirm things with the files you upload.

[renols]

Great, thanks for the debugging. I can start fixing this right away but will confirm things with the files you upload.

They should both be on the FTP now.

[Rocky]

Thank you.

The good THD file has only one double major at the start of the file, which is legal. So theoretically nothing should get dropped. But when I diff the two THD files I find I am periodically dropping frames throughout the stream. I can't see why from code inspection so I'll have to wait until the disk arrives to debug this.

[Rocky]

Disk set arrived. Gonna rip and test tomorrow. Long day coaching -- real swimmers in real water -- YAY! I'm really intrigued by this as usually I can figure things out by code inspection. I can't even imagine what is happening. No fear, the debugger reveals all.

[Boris]

Gonna rip and test tomorrow.

He he he, what if do not wake up tomorrow?

[Curly]

My mammy always said tomorrow is another day.

[renols]

Disk set arrived. Gonna rip and test tomorrow. Long day coaching -- real swimmers in real water -- YAY! I'm really intrigued by this as usually I can figure things out by code inspection. I can't even imagine what is happening. No fear, the debugger reveals all.

Will be interesting to see what your findings will reveal.

renols

[Rocky]

Ripped and duplicated the issue. The problem is that it's been assumed that each THD frame (major or minor) is a single PES packet and is marked by payload_unit_start_indicator = 1. This has been true for all the streams seen up until this one. This stream packs multiple THD frames into a single PES packet without any apparent rhyme or reason. It does seem to be true that every major starts a new PES packet but that is not enough. As a result, the code fails to properly detect frames as major or minor, and so sometimes falsely detects a double major and deletes frames. Just disabling the mechanism when there is only one M2TS is not good enough because this can still happen to individual M2TS files in the case of multiple M2TS files.

Going to have to buffer and parse at the ES level because the transport level gives no indication of the start of minor frames. The minor frame boundaries are needed to be able to delete minor frames for gaps correction. Also to be considered is whether to return to Bresenham plus safe minor deletion, or stick to the double major idea. That could always be done later if needed.

[Sherman]

it's been assumed that each THD frame (major or minor) is a single PES packet and is marked by payload_unit_start_indicator = 1

Rocky, that was my assumption. So can I fix this? It hurts after getting my spurs.

[Rocky]

Go for it, Sherman! And welcome back. What did you learn from Mr. Balzac?

[Sherman]

A lot.

[Curly]

Sworn to secrecy, eh? Big mystery. Lemme tell you some secrets. Nyuk nyuk nyuk.

[Sherman]

Hey Rocky. I have fixed up the THD demuxing. But rather than do the THD buffering and ES parsing on-the-fly with demuxing, I instead do a pass at the end on the demuxed THD file to fix up the THD gaps. The little GUI box shows "Fix THD gaps" during this time. I tried to do things on-the-fly but everything got ridiculously complex and slow. My way is clean, simple, and fast. Can you make a test release?

[Rocky]

Sounds great, Sherman. Gotta love clean, simple, and fast. I figured all that buffering and parsing was going to be a PITA, so I was happy when you offered to fix this. That you found a better way is frosting on the cake. I'll finish my coffee and danish and then make a test release.

[Rocky]

Here is a test version with MVC and the THD fix.

http://rationalqm.us/dgdemux/binaries/D ... c_test.rar

Feedback will be appreciated.

[Sherman]

I figured all that buffering and parsing was going to be a PITA, so I was happy when you offered to fix this. That you found a better way is frosting on the cake.

Thanks, Rocky. My way also has the advantage that when gaps processing is not required (it's disabled or there is only one M2TS) the demuxing is somewhat faster. To be honest, buffering and parsing at different levels is a real performance killer. Yes, I know DGIndexNV is designed that way, but it almost has to be that way as the relation between the video and everything else is very important and must be maintained, especially to honor the project range, etc. But for DGDemux we should be able to get rid of all that. I see you started implementing my way and got a giant performance boost (350%). I can finish that off if you like.

[Rocky]

Go for it, Sherman!

My life has become so easy.

Bullwinkle: ace system architect and forum enforcer
Sherman: ace coder and apprentice system architect
Rocky: ace debugger and occasional coder
Curly: our conscience
Levi: spiritual and creative foundation
admin: bean counter
DG: founder and benefactor

[Guest]

Quick test
Movie with one M2TS and 0 ms delay between audio and video
video 1 h 48 min 54 s 243 ms
thd... 1 h 48 min 54 s 237 ms
ac3... 1 h 48 min 54 s 240 ms

[Bullwinkle]

Sherman! C'mon, you can't do gaps correction when the stream is not enabled for demuxing. Otherwise, highly impressive.

@gonca

Thank you for your testing. Looking good.