DGIndexNV versus DGDemux

Support forum for DGDecNV
User avatar
Rocky
Moose Approved
Posts: 1052
Joined: Fri Sep 06, 2019 12:57 pm

DGIndexNV versus DGDemux

Post by Rocky » Sun Nov 15, 2020 3:09 pm

[Created this thread based on a PM from renols.]

Didn't quite know where to put this in the forum.

Will any of the speed gains made in DGDemux be possible to put into DGIndexNV?

Just tried to demux a bluray. With DGDemux it took 28 seconds, and with DGIndexNV it took 3m 30s.

I know that DGIndexNV creates the dgi file, and probably can't be compared to DGDemux.

But would still be nice if some of the speed optimizations from DGDemux could be ported to DGIndexNV.

Oh, btw. Just noticed that the two chapter files aren't identical. Looks like the chapter file for DGDemux has a chapter where the movie ends, and DGIndexNV doesn't.

Movie playtime is 1h 22m 41s

DGDemux 1.0.0.45:

CHAPTER01=00:00:00.000
CHAPTER01NAME=
CHAPTER02=00:09:06.713
CHAPTER02NAME=
CHAPTER03=00:19:54.693
CHAPTER03NAME=
CHAPTER04=00:29:13.669
CHAPTER04NAME=
CHAPTER05=00:40:26.341
CHAPTER05NAME=
CHAPTER06=00:50:25.397
CHAPTER06NAME=
CHAPTER07=00:58:46.732
CHAPTER07NAME=
CHAPTER08=01:09:57.026
CHAPTER08NAME=
CHAPTER09=01:19:44.363
CHAPTER09NAME=
CHAPTER10=01:22:40.873
CHAPTER10NAME=

DGIndexNV 217:

CHAPTER01=00:00:00.000
CHAPTER01NAME=
CHAPTER02=00:09:06.713
CHAPTER02NAME=
CHAPTER03=00:19:54.693
CHAPTER03NAME=
CHAPTER04=00:29:13.669
CHAPTER04NAME=
CHAPTER05=00:40:26.341
CHAPTER05NAME=
CHAPTER06=00:50:25.397
CHAPTER06NAME=
CHAPTER07=00:58:46.732
CHAPTER07NAME=
CHAPTER08=01:09:57.026
CHAPTER08NAME=
CHAPTER09=01:19:44.363
CHAPTER09NAME=

I am not quite sure what is correct.

Mediainfo says this when checking the m2ts file:

00:00:00.000 : Chapter 1
00:09:06.712 : Chapter 2
00:19:54.693 : Chapter 3
00:29:13.668 : Chapter 4
00:40:26.340 : Chapter 5
00:50:25.397 : Chapter 6
00:58:46.731 : Chapter 7
01:09:57.026 : Chapter 8
01:19:44.362 : Chapter 9
01:22:40.872 : Chapter 10

Could it be that DGIndexNV is missing a chapter at the end?

User avatar
Rocky
Moose Approved
Posts: 1052
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGIndexNV versus DGDemux

Post by Rocky » Sun Nov 15, 2020 3:13 pm

Performance: Yes, DGIndexNV is slower because it requires the nested architecture to be able to support project ranges, NALU parsing (for decoding), non-transport-packet streams, and some other things. DGDemux uses Sherman's transport-packet-based architecture. That is possible because bluray disks are exclusively transport-packet based, we're not doing any decoding, there is no project range, etc. Maybe we can get some gains from fwrite() versus _write(), however. I'll look into it. Or maybe assign Sherman to it...after he finishes the baseboard and trim. ;)

Chapters: I forgot to port a change from DGDemux, which I will do. Thank you for pointing it out, renols.

User avatar
Rocky
Moose Approved
Posts: 1052
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGIndexNV versus DGDemux

Post by Rocky » Tue Nov 17, 2020 1:08 pm

I snuck the chapters fix into slipstream 221 so just re-download.

User avatar
Sherman
Moose Approved
Posts: 121
Joined: Mon Jan 06, 2020 11:19 pm

DGIndexNV versus DGDemux

Post by Sherman » Fri Nov 20, 2020 5:58 pm

Rock star and Winkie, get ready for the Shame Box!

I made some optimizations to DGIndexNV and obtained the following times for indexing and demuxing the video and first THD from the 1917 bluray:

old way: 4:41
my way: 1:44

That is a 270% performance improvement. The runs vary quite a bit. On one run I got 1:19 but never over 2:00. Not sure why that variability exists. Could be stuff I'm doing at the same time on the PC.

Here is what I did:

1. Replaced _write() with buffered fwrite() for the video and audio demux.

2. Increased the internal BUFFER_SIZE for reading the input stream to 256MB (from 64K). This still uses _read() because the large internal buffer makes it irrelevant. This doesn't have a large effect but it is needed for item 4 below.

3. Inlined some functions.

4. This is the biggie. The function _telli64() is dreadfully slow and was being called at each NALU. I re-engineered things to call _telli64() only when the buffer is filled, which is not so often with the larger BUFFER_SIZE.

5. I optimized the EBSPtoRBSP() call.

I don't see too much more to optimize. I'll tie up loose ends and regression test, and then give you a build to release.

Can I be a moderator now?
Last edited by Sherman on Fri Nov 20, 2020 7:02 pm, edited 1 time in total.

User avatar
Bullwinkle
Moose Approved
Posts: 159
Joined: Thu Sep 05, 2019 6:37 pm

DGIndexNV versus DGDemux

Post by Bullwinkle » Fri Nov 20, 2020 6:01 pm

Way to go, Sherm. And good job on the stable. The rose petals were a nice touch, although I'm thinking maybe you were being snarky.

I'm a system architect. All that stuff is on the coder. No Shame Box for me.

Sorry, you're too young to be a moderator. Legal problems and all that. Give it 11 years.

User avatar
Rocky
Moose Approved
Posts: 1052
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGIndexNV versus DGDemux

Post by Rocky » Fri Nov 20, 2020 6:05 pm

Image

gonca
Moose Approved/Curly Approved
Posts: 907
Joined: Sun Apr 08, 2012 6:12 pm

Re: DGIndexNV versus DGDemux

Post by gonca » Fri Nov 20, 2020 6:25 pm

Sherman wrote:
Fri Nov 20, 2020 5:58 pm
Rock star and Winkie, get ready for the Shame Box!
Don't get cocky, it's a long way down when you fall

User avatar
Sherman
Moose Approved
Posts: 121
Joined: Mon Jan 06, 2020 11:19 pm

Re: DGIndexNV versus DGDemux

Post by Sherman » Fri Nov 20, 2020 6:59 pm

Yes Sir, Mr. gonca. Rocky can fly and I can't!

User avatar
Rocky
Moose Approved
Posts: 1052
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGIndexNV versus DGDemux

Post by Rocky » Sat Nov 21, 2020 3:19 pm

Guys, here is a test version for you. Changes were extensive so it's gotta be tested thoroughly. Please report any problems here.

http://rationalqm.us/misc/DGIndexNV_fast.exe

MKV and MP4 may not see such massive gains. Sherman is working on optimizing them.

gonca
Moose Approved/Curly Approved
Posts: 907
Joined: Sun Apr 08, 2012 6:12 pm

Re: DGIndexNV versus DGDemux

Post by gonca » Sat Nov 21, 2020 6:12 pm

Tested 2 4K movies (also demuxed with DGDemux for comparison)

First movie the mpls points to a single m2ts.
One byte difference in size of demuxed video

Second movie is a seemless branching disc
Opened the mpls with DGIndexNV_test and the demuxed video was 0 bytes

Used VDub2 to check index and created scripts, and they play fine

User avatar
Rocky
Moose Approved
Posts: 1052
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGIndexNV versus DGDemux

Post by Rocky » Sat Nov 21, 2020 6:36 pm

I just checked CARS_2 (HEVC) and CC_SAKURA (AVC). Both worked fine. Are you sure you have Video checked in the Demux list? Do the other streams demux correctly?

What is the disk? What is the video? Can you screenshot your Settings dialog? Does your destination drive have enough space? Please give your exact sequence of steps.

gonca
Moose Approved/Curly Approved
Posts: 907
Joined: Sun Apr 08, 2012 6:12 pm

Re: DGIndexNV versus DGDemux

Post by gonca » Sat Nov 21, 2020 7:20 pm

False alarm (Got used to using DGDemux)
It seems I selected the streams to demux but I did not check the enable demux processing box
1.png
1.png (55.44 KiB) Viewed 276 times
Should it generate a 0 byte video file
2.png
2.png (9.8 KiB) Viewed 275 times

User avatar
Rocky
Moose Approved
Posts: 1052
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGIndexNV versus DGDemux

Post by Rocky » Sat Nov 21, 2020 8:33 pm

OK, I duplicated the wrong creation of an empty video file. We'll get that fixed. Thank you for the report. Any data on timings?

EDIT: I locally fixed the bug that created the empty video file. Please continue testing other things.

BTW, Sherman is doing this in VS C++ 2019 and upgraded the compiler toolset to the latest so we gain great developer tools (like an integrated profiler and static code analysis) but lose Windows XP. Too bad, so sad. I am busy trying to eliminate all warnings/errors reported by the static code analysis. Gotta get out of the Box of Shame!

gonca
Moose Approved/Curly Approved
Posts: 907
Joined: Sun Apr 08, 2012 6:12 pm

Re: DGIndexNV versus DGDemux

Post by gonca » Sat Nov 21, 2020 9:09 pm

Re data on timings
So far the only thing is that there is a one byte difference in the hevc video when compared to DGDemux
Tomorrow I'll do some comparisons

User avatar
Rocky
Moose Approved
Posts: 1052
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGIndexNV versus DGDemux

Post by Rocky » Sat Nov 21, 2020 9:12 pm

gonca wrote:
Sat Nov 21, 2020 9:09 pm
So far the only thing is that there is a one byte difference in the hevc video when compared to DGDemux
Yeah, that is already on my list to fix. Maybe tomorrow. Thank you.

User avatar
Rocky
Moose Approved
Posts: 1052
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGIndexNV versus DGDemux

Post by Rocky » Sun Nov 22, 2020 12:36 am

I fixed the bad extra byte on demuxing video. Off to bed now.

renols
Moose Approved
Posts: 106
Joined: Tue Feb 22, 2011 3:34 am

Re: DGIndexNV versus DGDemux

Post by renols » Sun Nov 22, 2020 4:34 am

Hi.

This looks good.

I have tested the new fast against version 221 and can report the following.

Demuxing has been done with all tracks enabled, including video.

BD1:

221: 3m 32s
fast: 1m 10s

BD2:

221: 5m 13s
fast: 1m 36

4k (90GB size)

221: 11m 14s
fast: 3m 39s

I have not binary compared the files, but they all have the same size. Also the video file. Maybe you updated the download link to the version that had the 1 byte difference fixed.

Tests were done using two SSD disks, demuxing from one to the other.

Impressive results I must say :bravo:

renols

User avatar
Rocky
Moose Approved
Posts: 1052
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGIndexNV versus DGDemux

Post by Rocky » Sun Nov 22, 2020 6:03 am

Thank you for your test results, renols.
renols wrote:
Sun Nov 22, 2020 4:34 am
Impressive results I must say
You inspired the work, so right back at ya: :hat:

gonca
Moose Approved/Curly Approved
Posts: 907
Joined: Sun Apr 08, 2012 6:12 pm

Re: DGIndexNV versus DGDemux

Post by gonca » Sun Nov 22, 2020 12:44 pm

Binary compared 3 4K discs
Disc 1 is seemless branching with THD, gaps processing enabled
Discs 2 & 3 are single m2ts types
All files show no difference (script and dgi excluded)

Do you want me to compare to DGDemux as well?

User avatar
Rocky
Moose Approved
Posts: 1052
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGIndexNV versus DGDemux

Post by Rocky » Sun Nov 22, 2020 3:40 pm

Thank you. That's great. No need to compare to DGDemux.

Stand by for a release candidate with another big performace bump coming in minutes...

User avatar
Rocky
Moose Approved
Posts: 1052
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGIndexNV versus DGDemux

Post by Rocky » Sun Nov 22, 2020 3:49 pm

Here is a release candidate for the high-speed DGIndexNV (includes 32 and 64 bit):

http://rationalqm.us/misc/DGIndexNV_rc.rar

Changes:

* Built with VS 2019 and latest tools. WinXP is no longer supported.
* Major performance improvement for indexing and demux.
* Removed the process priority option. Everything is now high priority, including the indexing thread.
* Fixed several potential crashes exposed by static code analysis.
* Don't drop the last chapter for MPLS.
* Fixed lost demux settings after Save Project with MPLS open.
* Fixed bug in report progress for CLI.
* Fixed bug in detection of CLI invocation.
* Fixed extra byte at the end of demuxed video.

No need to modify your INI file: the line for priority in the INI file will be automatically removed.

Your feeback will be greatly appreciated. I think my contributions get me out of the Box of Shame. ;)

:salute: Sherman
:salute: Rocky
:salute: renols
:salute: gonca

User avatar
Sherman
Moose Approved
Posts: 121
Joined: Mon Jan 06, 2020 11:19 pm

Re: DGIndexNV versus DGDemux

Post by Sherman » Sun Nov 22, 2020 4:00 pm

Rocky wrote:
Sun Nov 22, 2020 3:49 pm
I think my contributions get me out of the Box of Shame.
For sure! I did only the first two items while Rocky did the rest. I couldn't get 32-bit to compile but Rocky figured that out too. It was a wrong solution configuration. I checked only the project configurations. Who'da thought it would be at the solution level? :oops:

This is what you call teamwork.

User avatar
Rocky
Moose Approved
Posts: 1052
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGIndexNV versus DGDemux

Post by Rocky » Sun Nov 22, 2020 5:47 pm

Sorry, guys. Please re-download. Fix THD gaps processing was broken.

http://rationalqm.us/misc/DGIndexNV_rc.rar

gonca
Moose Approved/Curly Approved
Posts: 907
Joined: Sun Apr 08, 2012 6:12 pm

Re: DGIndexNV versus DGDemux

Post by gonca » Sun Nov 22, 2020 6:13 pm

Ran a couple of index jobs just to see how quick the RC is, and to my surprise it is 50% slower than the the old/stable DGIndexNV.
Tried elementary stream from ssd to ssd
and disc from hdd to ssd
No demuxing done, just indexing

User avatar
Rocky
Moose Approved
Posts: 1052
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGIndexNV versus DGDemux

Post by Rocky » Sun Nov 22, 2020 6:36 pm

?????

Investigating...

Post Reply