[RESOLVED] DGIndexNV versus DGDemux

[Rocky]

Be careful that you don't run the fast one and the slow one from the same directory as the INI file will be wrong. Do everything in separate directories with fresh INI files.

I should leave that line in for backwards compatibility and just ignore it in the new version.

I am testing that now.

[Rocky]

My results on bluray 1917 indexing only. Ran in separate directories with valid INI files for each.

old: 3:36
new: 1:02

If you are doing everything separately and you still see the issue please give your INI files for both.

[Guest]

Repeated my tests
Source is a 4K HEVC elementary stream
Stable DGIndexNV is located at C:\Program Files (Portable)
RC is on desktop, ini deleted and then recreated by DGIndexNV_rc
stable 4m 14s
rc 6m 31s

stable.ini

(1.02 KiB) Downloaded 438 times

rc.ini

(806 Bytes) Downloaded 451 times

[Rocky]

What is the disk? Can you try on other disks? Try an AVC 1080p bluray.

I just can't believe this.

[Rocky]

Your INI files show you loaded a .265 file. Please clarify everything you did.

EDIT: I see you said that. Let me test that.

[Guest]

Tried a regular BD (AVC) 1080p
This time it was HDD to SSD
Stable took 2m 35s
RC took 4m 10s

[Rocky]

Is there is something weird with your system?

I demuxed the .265 from CARS_2 and then did it both ways with no demuxing. I got:

rc: 1:22
stable: 2:07

So why are you getting 6 and 4 minutes? That's terrible for both. Can you try with CARS_2 and give the timings?

Also, please make sure you use the very latest RC. Maybe you use an old version by accident. Should be 1407488 bytes in size.

How much memory does your machine have? Could you be paging my enlarged read buffer?

Can you take everything off your Windows system partition?

Finally, please test with the MPLS's as input.

This sucks and totally ruins my day.

[Guest]

Don't have CARS 2
Maybe my system is weird
I'll try again tomorrow after a reboot, just in case that cleans up whatever my bottleneck is

[Rocky]

Doubtful but worth a try. But go through everything I mentioned in my previous post, trying the things and answering my queries. It's the only hope because if I can't duplicate something, I can't fix it by debugging directly. renols had good results.

[Guest]

Might be my system
I looked at task manager while each version was in use
RC had disk activity of 240 MB/s
Stable was 190 MB/s
I also have 32GB of ram
How can I tell if the read buffer is being paged?

[Rocky]

I'll see what my rates are.

32GB is fine. You won't page.

I'd like you to get stuff off your windows partition.

I know your CPU kicks mine's ass so I'm baffled at this point.

[Guest]

I'm moving stuff to different drives and gonna start clean

[Rocky]

My process is going drive d: to drive d: (an SSD). I see 0% activity on D: during processing! But I have 64GB so the entire source file could be in windows memory from previous runs. I have to find a way to flush it out before running. Hmmm? Simple way is to reboot but that is draconian and a PITA. I'll try it once to see if it changes anything. I could pull 32GB out but that is pretty draconian too.

BTW, if you are only indexing, it's OK to output to the same drive because it only writes the index file.

Gonna reboot now and test. Paradoxically it should make the first run slower for me.

[Guest]

Time for you to relax
I moved the RC to D:\ (HDD)
Put the source on I:\ and output to W:\
Cut the time on the index by half
It has to be my configuration, my I/O never impressed me on this mobo
Oh well, next year I will be building a new system and hopefully these issues will go away

[Rocky]

That's good but not time to relax (see below). It's great that we are exposing all these things.

Here are my shocking results:

After reboot:

rc: 2:08 with read data rate 260 MB/sec
stable 1:23 with read data rate 410 MB/sec

That seems to duplicate your original report.

It doesn't matter where the rc exe is. But what are your I: and W: ?
Just use the same non-system SSD partition for source and output, preferably a separate physical SSD entirely from Windows.

Really scratching my head now. I have no idea how to explain the reduced read data rate for rc. Maybe my internal buffer is too big. I'll try reducing it. Constantly reboot or pull memory. What a pain.

Gonna sleep on it. Let's resume tomorrow. Thanks muchly for your testing.

[Guest]

Ok
I: and W: are SSDs in my computer

[Rocky]

Nah, I won't be able to sleep.

OK, I reduced the BUFFER_SIZE to 32K from 16MB. Now, after a reboot, I get:

rc 1:22 at 410 MB/sec.

That is the same as stable now.

So the question arises, is it only an improvement in demuxing that we have, or is it no improvement at all? I'm so confused. Have to test with demuxing for stable and rc after reboots. Then I'll know for sure what is going on.

We definitely want stuff off the windows partition. Maybe that was your issue all along. Still, it's good to get to the bottom of everything.

[Rocky]

OK, I get it all now.

rc with demux after reboot: 2:37
stable with demux after reboot: 7:24

First we need the reduced buffer. I'll update the rc and post when that is done.
Second we need to stay off the windows partition.

I think that will make everything right but let's see. I'm surprised that there aren't gains just for indexing for me (you said you saw gains) but the demuxing gain is massive. The fixing THD gaps is much faster too.

[Rocky]

The rc build is updated so re-download.

* The BUFFER_SIZE is reduced to optimize IO.

* The priority is restored to the INI file for compatibility, so delete INI files before
re-testing. rc and stable can now be in the same directory and share the same INI file.

http://rationalqm.us/misc/DGIndexNV_rc.rar

Off to bed now. Fingers crossed!

[Guest]

(you said you saw gains)

The gains in the RC were when I changed the source and output drives

On to the good news (BD used for test) (index only)
The RC is on W:\ (SSD), so is the source and output
After reboot
RC_old 1m 41s
RC_new 53s
Moved RC-new to C:\Program Files (Portable)
RC_new 53s

Stable is on C:\Program Files (Portable)
1m 15s

Gotta go to work soon so I will check in when I get home

[Rocky]

That's great gonca, thank you. So it looks like there are significant gains for indexing but really massive ones for demuxing. But I may be able to improve the indexing further now that I know everything that is going on.

I'll fix a few bugs while you are at work. For example, the default save path option is broken. So when you get home check for a new rc.

[DJATOM]

Hi, Rocky. Since you decided to poke the code, it will be nice to have this fixed - https://i.imgur.com/beQEYc2.png
On indexing it shows meaningful frame counting, but on completion those fields becomes "1".

[Rocky]

Good idea DJATOM. I'll do that.

Meanwhile, here is how to flush memory so you don't have to reboot:

https://chadaustin.me/2009/04/flushing-disk-cache/

Makes my life a lot easier! Now we need to find a way to flush the SSD hardware read cache.

[Rocky]

Here is rc3:

* Further small performance improvement.
* Removed the obsolete decode modes option. The line is retained in the INI file and ignored.
* Fixed bugs in default save project path handling.
* No longer clear the Info dialog after an indexing ends.

http://rationalqm.us/misc/DGIndexNV_rc3.rar

gonca
renols
DJATOM

Should be in pretty good shape overall now.

[Rocky]

Here's a little challenge for coders. Following is the biggest CPU bottleneck function during indexing. If you think you can speed it up please submit your changes. You'll have to run it (say) 500000000 times in a loop to get good timings with timeGetTime(). This is the best I was able to do. It's faster than the JM reference code.

Code: Select all

int __forceinline EBSPtoRBSP(byte *streamBuffer, int end_bytepos)
{
	int i, j;
	int count = 0;

	for (i = j = 0; i < end_bytepos; i++)
	{
		if (count == 2 && streamBuffer[i] == 0x03)
		{
			i++;
			count = 0;
		}
		if (!(streamBuffer[j] = streamBuffer[i]))
			count++;
		else
			count = 0;
		j++;
	}
	return j;
}