DGDemux development

User avatar
Britney
Curly Approved
Posts: 18
Joined: Sun Aug 09, 2020 3:24 pm

Re: DGDemux development

Post by Britney » Mon Oct 05, 2020 9:22 pm

Sherman wrote:
Mon Oct 05, 2020 8:06 pm
I don't see any problem.
:bow:

Kisses, Brit

User avatar
Mr. Peabody
Posts: 20
Joined: Tue Dec 24, 2019 10:20 am

Re: DGDemux development

Post by Mr. Peabody » Mon Oct 05, 2020 9:57 pm

Propping up straw men is a not-so-subtle form of rhetoric to be eschewed. I refer of course to the quantum measurement straw man.

User avatar
Sherman
Moose Approved
Posts: 121
Joined: Mon Jan 06, 2020 11:19 pm

Re: DGDemux development

Post by Sherman » Tue Oct 06, 2020 2:29 pm

Mr. Peabody wrote:
Mon Oct 05, 2020 9:57 pm
eschewed
Does that mean hard to swallow?

@Rocky

Got PGS demuxing working in fast mode. The fast and slow demuxed SUP files are binary identical. Now have to do THD and embedded AC3. Then a little cleanup and we can give it out.

There is enough supported to fully demux La Notte (Criterion). Here are the timings for demuxing everything:

slow: 5:00
fast: 1:19

That's a 3.8 times speedup!

User avatar
Natasha
Mosquito
Posts: 58
Joined: Wed Nov 20, 2019 12:11 pm

Re: DGDemux development

Post by Natasha » Tue Oct 06, 2020 7:59 pm

Sherman wrote:
Tue Oct 06, 2020 2:29 pm
That's a 3.8 times speedup!
Is that the best you can do? Pathetic.

User avatar
Curly
Moose Approved
Posts: 69
Joined: Sun Mar 15, 2020 11:05 am

Re: DGDemux development

Post by Curly » Tue Oct 06, 2020 8:01 pm

Give the kid a break. He's doing his best.

User avatar
Bullwinkle
Moose Approved
Posts: 159
Joined: Thu Sep 05, 2019 6:37 pm

Re: DGDemux development

Post by Bullwinkle » Tue Oct 06, 2020 8:19 pm

If you can do better, Nattie, bring it. Otherwise, stand down.

You're doing great Sherman. Moose Approved!

User avatar
Curly
Moose Approved
Posts: 69
Joined: Sun Mar 15, 2020 11:05 am

Re: DGDemux development

Post by Curly » Wed Oct 07, 2020 7:22 am

Sabine Hossenfelder wrote:

"The two biggest problems with superdeterminism at the moment are (a) the lack of a generally applicable fundamental theory and (b) the lack of experiment." *

From which we easily conclude that it is metaphysical mumbo-jumbo. :roll:

* https://arxiv.org/pdf/2010.01324.pdf

User avatar
Levi
Moose Approved
Posts: 10
Joined: Sat Apr 18, 2020 6:12 pm

Re: DGDemux development

Post by Levi » Wed Oct 07, 2020 8:27 pm

Somebody here said you cannot blame a mosquito for being a mosquito. Makes sense to me.

User avatar
Sherman
Moose Approved
Posts: 121
Joined: Mon Jan 06, 2020 11:19 pm

Re: DGDemux development

Post by Sherman » Thu Oct 08, 2020 11:34 am

Thank you Bullwinkle. I really appreciate your support! And Levi too.

I have the first half of THD working, that is, the do-not-split-THD is working. Now working on splitting out the AC3 stream.

User avatar
Sherman
Moose Approved
Posts: 121
Joined: Mon Jan 06, 2020 11:19 pm

Re: DGDemux development

Post by Sherman » Thu Oct 08, 2020 12:24 pm

Now I have THD splitting working. All stream types are now working in fast mode. Still gotta do:

* preallocation
* garbage stripping option
* check that all DELAY values are correct

I see the light at the end of the tunnel!

Nattie, how is your implementation coming along?
:belly-laugh:

User avatar
Bullwinkle
Moose Approved
Posts: 159
Joined: Thu Sep 05, 2019 6:37 pm

Re: DGDemux development

Post by Bullwinkle » Thu Oct 08, 2020 7:53 pm

Sherman wrote: ...laying floors...
Ya gotta earn your keep, kid.
Sherman wrote:
Thu Oct 08, 2020 12:24 pm
I see the light at the end of the tunnel!
I can almost taste it.
Sherman wrote:
Thu Oct 08, 2020 12:24 pm
Nattie, how is your implementation coming along?
That's a pretty fine stomping for a hoomin. Snort!

User avatar
Sherman
Moose Approved
Posts: 121
Joined: Mon Jan 06, 2020 11:19 pm

Re: DGDemux development

Post by Sherman » Fri Oct 09, 2020 12:33 pm

Girls and boys! Rocky gave me permission to release an alpha test version of the fast mode implementation, so here it is:

http://rationalqm.us/dgdemux/binaries/D ... t_test.rar

Notes:

0) To run in fast mode, hold down the left SHIFT key when pressing the Show or Demux buttons. If you use DGDemux.exe directly, include the option '-fast'.

1) I haven't found an efficient way yet to do filler NALU stripping for AVC video, so you will see a difference (which is inconsequential) between the demuxed video in slow and fast mode in this case. The fast code simply processes transport packets one-by-one and has no concept of anything that spans multiple packets (such as NALUs); that's what makes it so fast.

2) All other streams should be bit identical in slow and fast mode.

3) All file names should be the same between slow and fast modes, so change the output name if you want to do both ways to compare (without leaving the GUI).

4) Garbage stripping is not yet implemented.

This is the hardest thing I ever did, including laying floors and installing appliances, so I mighta goofed something up. Your testing and feedback will be greatly appreciated. Enjoy!

Mr Peabody and I are off to visit Albert Einstein again, as there are some loose ends regarding Lorenz invariance I want to clear up, and I want to persuade him to join the forum. See ya in a while!

User avatar
Bullwinkle
Moose Approved
Posts: 159
Joined: Thu Sep 05, 2019 6:37 pm

Re: DGDemux development

Post by Bullwinkle » Fri Oct 09, 2020 12:34 pm

Tastes great, less filling!

gonca
Moose Approved/Curly Approved
Posts: 907
Joined: Sun Apr 08, 2012 6:12 pm

Re: DGDemux development

Post by gonca » Sat Oct 10, 2020 10:59 am

Tested using 4K version of Wonder Woman

external HDD >> internal SSD
normal 16:00
fast 9:31

suspected that HDD was a bottleneck so I copied the movie to an internal SSD

internal SSD >> internal SSD
normal 8:57
fast 2:59

edit
folders containing each set of the demux were identical in size

User avatar
Rocky
Moose Approved
Posts: 1052
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGDemux development

Post by Rocky » Sat Oct 10, 2020 12:10 pm

Thank you for the testing, gonca. The ideal situation is SSD->SSD with enough free space to ensure all the preallocations succeed. That's where we see 3-4 times speedup.

The directories will not be exactly the same size when there is AVC video containing filler NALUs (not your case). I have been demuxing into two subfolders (fast and slow) by changing the output path. Then get in the folder holding those two and open a DOS prompt. Execute:

fc /b slow\* fast\*

That will compare all the files. If you have the AVC video with filler get that out before running the compare, or you will be inundated with output.

I have tested two disks so far and all is well. Will test more today.

User avatar
Bullwinkle
Moose Approved
Posts: 159
Joined: Thu Sep 05, 2019 6:37 pm

Re: DGDemux development

Post by Bullwinkle » Sat Oct 10, 2020 3:55 pm

Nice work. I found some issues doing MONSTERS_UNIVERSITY:

1. The fast video stream is missing a single byte at the end (weird!).

2. The THD embedded AC3 doesn't match.

3. The SUPs don't match.

BTW, I tested slow mode in the test version versus build 43 and everything matched.

User avatar
Sherman
Moose Approved
Posts: 121
Joined: Mon Jan 06, 2020 11:19 pm

Re: DGDemux development

Post by Sherman » Sat Oct 10, 2020 3:56 pm

Thanks, Bullwinkle. I'll fix that up ASAP.

User avatar
Sherman
Moose Approved
Posts: 121
Joined: Mon Jan 06, 2020 11:19 pm

Re: DGDemux development

Post by Sherman » Mon Oct 12, 2020 7:45 am

I fixed the SUP and embedded AC3 demuxing in fast mode. The video discrepancy is actually the old architecture outputing an extra spurious byte at the end of the stream. The fast mode is correct and EAC3TO also doesn't output that byte. I'll do some testing and then give an update for the fast mode test build.

I also fixed a problem with DGDemuxGUI. It was not showing the 'Fix THD gaps' message.

User avatar
Bullwinkle
Moose Approved
Posts: 159
Joined: Thu Sep 05, 2019 6:37 pm

Re: DGDemux development

Post by Bullwinkle » Mon Oct 12, 2020 8:59 am

Keep it up Sherman, you're doing great. Apart from the one-byte video difference MONSTERS demuxes identically between slow and fast modes.

Can you check a few more things, please:

* Proper operation of the skip options in fast mode.
* The FixTHDGaps() function seems slow but maybe it is what it is.

Also, don't forget to implement garbage stripping, and AVC filler NALU stripping (if possible).

User avatar
Sherman
Moose Approved
Posts: 121
Joined: Mon Jan 06, 2020 11:19 pm

Re: DGDemux development

Post by Sherman » Mon Oct 12, 2020 9:10 am

Will do, Bullwinkle. Thank you for your testing.

Meanwhile, here is an updated fast mode build with the aforementioned fixes:

http://rationalqm.us/dgdemux/binaries/D ... t_test.rar

* Fixed SUP demuxing (it was broken only for multiple M2TS files in the playlist).
* Fixed THD embedded AC3 demuxing (gaps correction was not being performed).
* Fixed missing 'Fix THD gaps' message in the GUI.
Last edited by Sherman on Mon Oct 12, 2020 9:31 am, edited 1 time in total.

gonca
Moose Approved/Curly Approved
Posts: 907
Joined: Sun Apr 08, 2012 6:12 pm

Re: DGDemux development

Post by gonca » Mon Oct 12, 2020 9:26 am

Gonna copy some movies from my archive and will try to test today

User avatar
Sherman
Moose Approved
Posts: 121
Joined: Mon Jan 06, 2020 11:19 pm

Re: DGDemux development

Post by Sherman » Mon Oct 12, 2020 9:38 am

Sweet. Thank you, gonca. Would be great if you can test a 3D disk, demuxing all streams and just the MVC stream.

gonca
Moose Approved/Curly Approved
Posts: 907
Joined: Sun Apr 08, 2012 6:12 pm

Re: DGDemux development

Post by gonca » Mon Oct 12, 2020 10:09 am

Copying one of those as well

gonca
Moose Approved/Curly Approved
Posts: 907
Joined: Sun Apr 08, 2012 6:12 pm

Re: DGDemux development

Post by gonca » Mon Oct 12, 2020 11:44 am

Gonna report one disc at a time so my head doesn't explode from the data

VENOM 4K
Primary video in slow is one byte bigger than fast
All other tracks, including DV and PGS, are identical

BUMBLEBEE 4K
Had to use no preallocation
Too many streams and preallocation was using up all free space and then some

One byte issue (I realize it might no be an issue but still reporting)
DV matches
Embedded AC3 matches
all other files match

VENOM 1080p
Size mismatch on AVC file
ac3 files match
PGS files match
can't open dts files

BUMBLEBEE 1080p
AVC made file compare go nuts
all other files match

AGE OF EXTINCTION 3D
AVC file made it go nuts again
*.tag files could not be opened
everything else matches

MVC only
match
Last edited by gonca on Mon Oct 12, 2020 4:07 pm, edited 4 times in total.

User avatar
Rocky
Moose Approved
Posts: 1052
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGDemux development

Post by Rocky » Mon Oct 12, 2020 11:57 am

That's fine. Maybe edit the post instead of making a new post for every disk?

Post Reply