Support forum for DG Tools

[Created this thread based on a PM from renols.]

Didn't quite know where to put this in the forum.

Will any of the speed gains made in DGDemux be possible to put into DGIndexNV?

Just tried to demux a bluray. With DGDemux it took 28 seconds, and with DGIndexNV it took 3m 30s.

I know that DGIndexNV creates the dgi file, and probably can't be compared to DGDemux.

But would still be nice if some of the speed optimizations from DGDemux could be ported to DGIndexNV.

Oh, btw. Just noticed that the two chapter files aren't identical. Looks like the chapter file for DGDemux has a chapter where the movie ends, and DGIndexNV doesn't.

Movie playtime is 1h 22m 41s

DGDemux 1.0.0.45:

CHAPTER01=00:00:00.000
CHAPTER01NAME=
CHAPTER02=00:09:06.713
CHAPTER02NAME=
CHAPTER03=00:19:54.693
CHAPTER03NAME=
CHAPTER04=00:29:13.669
CHAPTER04NAME=
CHAPTER05=00:40:26.341
CHAPTER05NAME=
CHAPTER06=00:50:25.397
CHAPTER06NAME=
CHAPTER07=00:58:46.732
CHAPTER07NAME=
CHAPTER08=01:09:57.026
CHAPTER08NAME=
CHAPTER09=01:19:44.363
CHAPTER09NAME=
CHAPTER10=01:22:40.873
CHAPTER10NAME=

DGIndexNV 217:

CHAPTER01=00:00:00.000
CHAPTER01NAME=
CHAPTER02=00:09:06.713
CHAPTER02NAME=
CHAPTER03=00:19:54.693
CHAPTER03NAME=
CHAPTER04=00:29:13.669
CHAPTER04NAME=
CHAPTER05=00:40:26.341
CHAPTER05NAME=
CHAPTER06=00:50:25.397
CHAPTER06NAME=
CHAPTER07=00:58:46.732
CHAPTER07NAME=
CHAPTER08=01:09:57.026
CHAPTER08NAME=
CHAPTER09=01:19:44.363
CHAPTER09NAME=

I am not quite sure what is correct.

Mediainfo says this when checking the m2ts file:

00:00:00.000 : Chapter 1
00:09:06.712 : Chapter 2
00:19:54.693 : Chapter 3
00:29:13.668 : Chapter 4
00:40:26.340 : Chapter 5
00:50:25.397 : Chapter 6
00:58:46.731 : Chapter 7
01:09:57.026 : Chapter 8
01:19:44.362 : Chapter 9
01:22:40.872 : Chapter 10

Could it be that DGIndexNV is missing a chapter at the end?

Performance: Yes, DGIndexNV is slower because it requires the nested architecture to be able to support project ranges, NALU parsing (for decoding), non-transport-packet streams, and some other things. DGDemux uses Sherman's transport-packet-based architecture. That is possible because bluray disks are exclusively transport-packet based, we're not doing any decoding, there is no project range, etc. Maybe we can get some gains from fwrite() versus _write(), however. I'll look into it. Or maybe assign Sherman to it...after he finishes the baseboard and trim.

Chapters: I forgot to port a change from DGDemux, which I will do. Thank you for pointing it out, renols.

I snuck the chapters fix into slipstream 221 so just re-download.

Rock star and Winkie, get ready for the Shame Box!

I made some optimizations to DGIndexNV and obtained the following times for indexing and demuxing the video and first THD from the 1917 bluray:

old way: 4:41
my way: 1:44

That is a 270% performance improvement. The runs vary quite a bit. On one run I got 1:19 but never over 2:00. Not sure why that variability exists. Could be stuff I'm doing at the same time on the PC.

Here is what I did:

1. Replaced _write() with buffered fwrite() for the video and audio demux.

2. Increased the internal BUFFER_SIZE for reading the input stream to 256MB (from 64K). This still uses _read() because the large internal buffer makes it irrelevant. This doesn't have a large effect but it is needed for item 4 below.

3. Inlined some functions.

4. This is the biggie. The function _telli64() is dreadfully slow and was being called at each NALU. I re-engineered things to call _telli64() only when the buffer is filled, which is not so often with the larger BUFFER_SIZE.

5. I optimized the EBSPtoRBSP() call.

I don't see too much more to optimize. I'll tie up loose ends and regression test, and then give you a build to release.

Can I be a moderator now?

Way to go, Sherm. And good job on the stable. The rose petals were a nice touch, although I'm thinking maybe you were being snarky.

I'm a system architect. All that stuff is on the coder. No Shame Box for me.

Sorry, you're too young to be a moderator. Legal problems and all that. Give it 11 years.

Sherman wrote: ↑

Fri Nov 20, 2020 4:58 pm

Rock star and Winkie, get ready for the Shame Box!

Don't get cocky, it's a long way down when you fall

Yes Sir, Mr. gonca. Rocky can fly and I can't!

Guys, here is a test version for you. Changes were extensive so it's gotta be tested thoroughly. Please report any problems here.

http://rationalqm.us/misc/DGIndexNV_fast.exe

MKV and MP4 may not see such massive gains. Sherman is working on optimizing them.

Tested 2 4K movies (also demuxed with DGDemux for comparison)

First movie the mpls points to a single m2ts.
One byte difference in size of demuxed video

Second movie is a seemless branching disc
Opened the mpls with DGIndexNV_test and the demuxed video was 0 bytes

Used VDub2 to check index and created scripts, and they play fine

I just checked CARS_2 (HEVC) and CC_SAKURA (AVC). Both worked fine. Are you sure you have Video checked in the Demux list? Do the other streams demux correctly?

What is the disk? What is the video? Can you screenshot your Settings dialog? Does your destination drive have enough space? Please give your exact sequence of steps.

False alarm (Got used to using DGDemux)
It seems I selected the streams to demux but I did not check the enable demux processing box Should it generate a 0 byte video file

OK, I duplicated the wrong creation of an empty video file. We'll get that fixed. Thank you for the report. Any data on timings?

EDIT: I locally fixed the bug that created the empty video file. Please continue testing other things.

BTW, Sherman is doing this in VS C++ 2019 and upgraded the compiler toolset to the latest so we gain great developer tools (like an integrated profiler and static code analysis) but lose Windows XP. Too bad, so sad. I am busy trying to eliminate all warnings/errors reported by the static code analysis. Gotta get out of the Box of Shame!

Re data on timings
So far the only thing is that there is a one byte difference in the hevc video when compared to DGDemux
Tomorrow I'll do some comparisons

gonca wrote: ↑

Sat Nov 21, 2020 8:09 pm

So far the only thing is that there is a one byte difference in the hevc video when compared to DGDemux

Yeah, that is already on my list to fix. Maybe tomorrow. Thank you.

I fixed the bad extra byte on demuxing video. Off to bed now.

Hi.

This looks good.

I have tested the new fast against version 221 and can report the following.

Demuxing has been done with all tracks enabled, including video.

BD1:

221: 3m 32s
fast: 1m 10s

BD2:

221: 5m 13s
fast: 1m 36

4k (90GB size)

221: 11m 14s
fast: 3m 39s

I have not binary compared the files, but they all have the same size. Also the video file. Maybe you updated the download link to the version that had the 1 byte difference fixed.

Tests were done using two SSD disks, demuxing from one to the other.

Impressive results I must say

renols

Thank you for your test results, renols.

renols wrote: ↑

Sun Nov 22, 2020 3:34 am

Impressive results I must say

You inspired the work, so right back at ya:

Binary compared 3 4K discs
Disc 1 is seemless branching with THD, gaps processing enabled
Discs 2 & 3 are single m2ts types
All files show no difference (script and dgi excluded)

Do you want me to compare to DGDemux as well?

Thank you. That's great. No need to compare to DGDemux.

Stand by for a release candidate with another big performace bump coming in minutes...

Here is a release candidate for the high-speed DGIndexNV (includes 32 and 64 bit):

http://rationalqm.us/misc/DGIndexNV_rc.rar

Changes:

* Built with VS 2019 and latest tools. WinXP is no longer supported.
* Major performance improvement for indexing and demux.
* Removed the process priority option. Everything is now high priority, including the indexing thread.
* Fixed several potential crashes exposed by static code analysis.
* Don't drop the last chapter for MPLS.
* Fixed lost demux settings after Save Project with MPLS open.
* Fixed bug in report progress for CLI.
* Fixed bug in detection of CLI invocation.
* Fixed extra byte at the end of demuxed video.

No need to modify your INI file: the line for priority in the INI file will be automatically removed.

Your feeback will be greatly appreciated. I think my contributions get me out of the Box of Shame.

Sherman
Rocky
renols
gonca

Rocky wrote: ↑

Sun Nov 22, 2020 2:49 pm

I think my contributions get me out of the Box of Shame.

For sure! I did only the first two items while Rocky did the rest. I couldn't get 32-bit to compile but Rocky figured that out too. It was a wrong solution configuration. I checked only the project configurations. Who'da thought it would be at the solution level?

This is what you call teamwork.

Sorry, guys. Please re-download. Fix THD gaps processing was broken.

http://rationalqm.us/misc/DGIndexNV_rc.rar

Ran a couple of index jobs just to see how quick the RC is, and to my surprise it is 50% slower than the the old/stable DGIndexNV.
Tried elementary stream from ssd to ssd
and disc from hdd to ssd
No demuxing done, just indexing

?????

Investigating...