DGDemux development

User avatar
Curly
Posts: 712
Joined: Sun Mar 15, 2020 11:05 am

Re: DGDemux development

Post by Curly »

hydra3333 wrote:
Sat Nov 16, 2019 10:15 pm
If there was a like button on the forum, I'd click it :)
It seems you got your wish.

viewtopic.php?f=5&t=859
Curly Howard
Director of EAC3TO Development
User avatar
Sherman
Posts: 576
Joined: Mon Jan 06, 2020 10:19 pm

Re: DGDemux development

Post by Sherman »

Levi is Moose Approved with only 3 posts. Is that fair?
Sherman Peabody
Director of Linux Development
User avatar
Boris
Posts: 92
Joined: Sun Nov 10, 2019 2:55 pm

Re: DGDemux development

Post by Boris »

You'll understand one day...if you grow up. Bwahahahahaha!
DAE avatar
Guest

Re: DGDemux development

Post by Guest »

Hey Natasha
squirrel attack.jpg
User avatar
Bullwinkle
Posts: 338
Joined: Thu Sep 05, 2019 6:37 pm

Re: DGDemux development

Post by Bullwinkle »

Sherman wrote:
Fri Apr 24, 2020 3:46 pm
Is that fair?
It's about being a strong contributor/generally awesome. So far it's not looking good for you. Levi, on the other hand, is way beyond awesome. He is a God. He may well be the reincarnation of Apollo.

Image

Great image there, gonca. Love it! :salute: Here's what I would say to Natasha: Does it hurt?
DAE avatar
Guest

Re: DGDemux development

Post by Guest »

I would remind her that Rocky is a flying squirrel, that could easily be her nose
flying squirrel attack.jpg
User avatar
Bullwinkle
Posts: 338
Joined: Thu Sep 05, 2019 6:37 pm

Re: DGDemux development

Post by Bullwinkle »

When Rocky gets in a bad mood...

Image

Does it hurt? :lol:
User avatar
Natasha
Posts: 150
Joined: Wed Nov 20, 2019 11:11 am

Re: DGDemux development

Post by Natasha »

You guys are pushing your luck.
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGDemux development

Post by Rocky »

Hmm. Looks like when DGDemux does gaps correction it mangles the major_sync_info() field, which makes the CRC wrong and explains why ffmpeg says invalid data. Weird. OK, let's find out why this is happening.

I don't think tsmuxer is doing anything special; it's just not mangling the major_sync_info(). ;)
User avatar
Natasha
Posts: 150
Joined: Wed Nov 20, 2019 11:11 am

Re: DGDemux development

Post by Natasha »

Rocky wrote:
Fri Apr 24, 2020 8:40 pm
OK, let's find out why this is happening.
Great plan. Who'd a thunk? :roll:
User avatar
Sherman
Posts: 576
Joined: Mon Jan 06, 2020 10:19 pm

Re: DGDemux development

Post by Sherman »

Shut up, Natasha. gonca is right, you're a nobody.
Sherman Peabody
Director of Linux Development
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGDemux development

Post by Rocky »

Be nice to the ladies, Sherman. One day you will understand why.
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGDemux development

Post by Rocky »

Here are two things I just discovered that threw me for loops.

1. When stripping the AC3 core with eac3to you have to give the -keepDialnorm option. This is why I thought I was mangling the major sync info. When you do this they are the same again. domy has asked for an option to remove it in DGDemux but it's not there yet.

2. This one is huge! If you demux only the THD stream with tsmuxer, gaps processing is not performed. You have to also demux the video at the same time to get gaps processing. Without video, it says it is deleting frames but it apparently isn't. To see that just compare the two THD files, one demuxed with video and one without. They are different! EDIT: See the next post for the real reason the files are different.

Hopefully knowing all these things now my debugging can get back on track.
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGDemux development

Post by Rocky »

Oh...My...God!

tsmuxer is not doing proper gaps processing at all. It is just truncating the audio when the video runs out! That is why the THD streams differ depending on if you also demux video.

I demuxed the raw THD with DGDemux (no gaps processing and with splitting the THD and AC3). Resulting THD duration is 6375.219167. Then I demuxed using tsmuxer with video and stripped the AC3 core keeping dialnorm. Resulting THD duration is 6375.160000. It thus looks like tsmuxer is doing gaps processing. Comparison of these two files should show all the frame deletions made by tsmuxer. But when you compare them (you can use fc /b) the only difference is that the tsmuxer one has truncated all the audio beyond the end of the video. There are no individual frame deletions! So tsmuxer is producing equal duration video and audio, but it is not actually doing any sync correction. And looking at the tsmuxer source code, it claims to only delete a single transport packet, which would be wrong because a THD frame spans many packets. tsmuxer is a fraud (at least as far as sync correction is concerned)!

So tsmuxer is useless for our purposes. I'll have to go back to MakeMKV for clues about how to delete individual frames in the compressed THD domain. I can make an MKV and then demux the THD for comparison to DGDemux. Let's hope MakeMKV is not also a fraud. :)

Oh what a tangled web!
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGDemux development

Post by Rocky »

Using MakeMKV to rip CARS_2 to MKV...
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGDemux development

Post by Rocky »

Aha! Everything is clear now.

MakeMKV is not a fraud. It is deleting only major frames and not performing any other adjustments, e.g., to the timestamps. Thus, I would expect ffmpeg to bomb when converting the THD stream demuxed from the MKV to FLAC. And it does! It fails exactly as does DGDemux. I suppose it is not surprising as there was never an answer to Mike's post asking how to properly adjust timestamps (last post here: https://github.com/Nevcairiel/LAVFilters/issues/282). Thus, MakeMKV is implementing option 2 below. tsmuxer is implementing option 1 but truncating the audio to the length of the video when the video is also demuxed along with the audio.

Therefore, unless one of our esteemed members finds something to challenge this, it looks like we will have to go with our solution for creating a clean in-sync flac file. To remind you, the options are proposed to be:

1. Demux the stream raw. Keeps ATMOS but no sync correction and does not bomb ffmpeg FLAC conversion. Use when not too many M2TS files are in the playlist.

2. Demux with gaps correction. Keeps ATMOS and does sync correction but bombs ffmpeg FLAC conversion. Use when you have too many M2TS files. You can use the THD file directly in (say) mkvtoolnix, as players don't care about the spec violation.

3. Demux raw and correct sync in the uncompressed domain. Does sync correction and does not bomb ffmpeg but loses ATMOS. Not a big deal because any conversion to FLAC would lose ATMOS. Use when you want a sync-corrected FLAC file.

I do not see any way to make a valid sync-corrected THD file that retains the ATMOS. Maybe it could be possible by re-encoding the clean FLAC from 3 to THD and re-adding the ATMOS metadata. Not something that is going to be feasible absent a full-featured THD encoder and some technology to save and restore ATMOS metadata. Forget about it!

I conclude that method 3 (successfully concept tested) is the ONLY way to make a sync-corrected FLAC file. Your feedback on this conclusion and proposal will be greatly appreciated. domy? renols? gonca?
DAE avatar
Guest

Re: DGDemux development

Post by Guest »

Personally I just keep the THD file + Atmos, but for those that transcode to other formats, option 3 is the way to go.
The movies I do don't seem to have to many m2ts files, normally.
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGDemux development

Post by Rocky »

gonca wrote:
Sat Apr 25, 2020 11:06 am
Personally I just keep the THD file + Atmos, but for those that transcode to other formats, option 3 is the way to go.
The movies I do don't seem to have to many m2ts files, normally.
So option 1 for you. And if you do have too many M2TS files then you can use option 2. If you need FLAC, then option 3.

Even so, there is no guarantee that option 2 files will not be found to be problematic somewhere, although tested players don't seem to mind it. And for archiving, you wouldn't want option 2.
DAE avatar
Guest

Re: DGDemux development

Post by Guest »

Correct.
As long as DGDemux retains the 3 options all bases are covered.
Until someone figures out how to correct the timestamps this is as close to perfection as we are going to get
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGDemux development

Post by Rocky »

Yes, but it's not even clear that correcting timestamps will be enough, because the buffering model will be broken, as explained earlier. We would have to become a THD encoder in effect.

Alright, we have gonca's endorsement. One down! Let's see if the others want to weigh in before going full steam ahead.
DAE avatar
Guest

Re: DGDemux development

Post by Guest »

have to become a THD encoder in effect
which is way past the scope of DGDemux.
As long as we have the 3 options all bases are covered.
DGDemux is probably one of the few solutions out there for this issue, without using Dolby's encoding tools ($$$)
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGDemux development

Post by Rocky »

probably? I would say definitely the only solution for clean, transcodable sync correction without THD re-encoding. OK, quibbling on your language, I know.
User avatar
Bullwinkle
Posts: 338
Joined: Thu Sep 05, 2019 6:37 pm

Re: DGDemux development

Post by Bullwinkle »

Rocky and Bullwinkle rule!

Gonna Moose Approve some more members that I missed yesterday.
DAE avatar
renols
Posts: 149
Joined: Tue Feb 22, 2011 2:34 am

Re: DGDemux development

Post by renols »

Hi.

I am also totally fine with the three options.

One thing with ragard to makemkv. the version you have used is the most recent one? (1.15.1). I ask because someone wrote this in an earlier post:

"MAKEMKV does not produce errors in FFMPEG now in the latest version, whereas the previous version did. I guess this is like you suspected Rocky, they remove the major frames what we see comparing DGDEMUX and MAKEMKV output."

If no software, other than some TrueHD encoder can produce a thd file without "errors", then so be it. I guess up to 25 m2ts files, it would still be OK to use the thd file without gap removal, as that would give a 20 ms delay, according to the 0,8 ms per m2ts file you gave us earlier.

So, three options it is, until some other solution may come along.

Mike might also want to keep his cards close, so maybe he isn't interested in sharing what he does, if he does something different. Don't know if you have communicated with him in the past.

Again thanks for the effort put into this. Just makes a great program even better.

renols
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGDemux development

Post by Rocky »

renols wrote:
Sat Apr 25, 2020 1:51 pm
"MAKEMKV does not produce errors in FFMPEG now in the latest version, whereas the previous version did.
Oh, thank you for pointing out my oversight! I have been using 1.14.7. Maybe Mike figured something out. Need to repeat the testing ASAP. After my lunch.

I had read of seamless branching issues with 1.15.1 so didn't upgrade. You can google it.
Post Reply