DGDemux development

Post Reply
User avatar
redbtn
Posts: 44
Joined: Sat Nov 16, 2019 2:35 pm

Re: DGDemux development

Post by redbtn »

Rocky wrote:
Sun Dec 01, 2019 6:00 pm
@redbtn

No need for that stuff. Re-download DGDemux_test.rar. It's built with 100/10/1 prealloc. I'm looking into whether we can get a callback when the allocated size is exceeded, with the idea of enlarging the alloc. Then we could start small and increase only when needed.
Good! Nice idea!
PS: It seems minimize issue is not only my OS issue
DAE avatar
zqslzwzw
Posts: 18
Joined: Fri Nov 29, 2019 8:35 am

Re: DGDemux development

Post by zqslzwzw »

Bullwinkle wrote:
Sun Dec 01, 2019 12:54 pm
Can't duplicate that. What OS etc.? C'mon, windows can't minimize and restore a dialog? Don't hand us some WinXP/Win7 nonsense. We use latest libraries and runtimes.

Gonna tell Rocky to go with video 100 GB, audio 5 GB each, sups 1 GB.
I do not think it's necessary to reduce the fragment count to zero. Slight fragments should be allowed since they do not influence the performance. 100G preallocation is too large for my poor SSD or usb hdd. Maybe .sup can be assigned with 64M block, audio 1G and video 10G.
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGDemux development

Post by Rocky »

gonca wrote:
Sun Dec 01, 2019 6:05 pm
As per Task Manager
DGDemuxGUI 8.0 MB
DGDemux 10.1 MB

Re GUI distortion
Load a mpls, select output, etc
Minimize and the restore
Stream list is now on top

Open DGDemuxGUI
Minimize and then restore
GUI comes up normal
but using BROWSE button makes it go white
Wow. :wow: That's freaky. I'm guessing the checked list component is getting some unexpected messages. Investigating...
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGDemux development

Post by Rocky »

zqslzwzw wrote:
Sun Dec 01, 2019 6:20 pm
I do not think it's necessary to reduce the fragment count to zero. Slight fragments should be allowed since they do not influence the performance. 100G preallocation is too large for my poor SSD or usb hdd. Maybe .sup can be assigned with 64M block, audio 1G and video 10G.
Maybe best to make it configurable then. I ripped CC_SAKURA and got the entire video in one file fragment.

The thing is if you exceed the prealloc, you're going to get lots of little fragments after that. It's not that you will just get a couple large fragments. That's why I'm looking into the dynamic resizing.
User avatar
redbtn
Posts: 44
Joined: Sat Nov 16, 2019 2:35 pm

Re: DGDemux development

Post by redbtn »

It seems like files have no fragmentation, but while they writes to disk, they are still writes in small pieces, because hdd speed drops from 150mbps to 40mbps. I think it happens because hdd writing small pieces and have to move heads very often between files.
I have to check it, but I think if demux only video it will be ok, but in parallel with audio and subs disk IO performance is dropping. And demux takes about 2 times more than it can be.
I'm talking about demux process. After demux is finished, work with files is ok now, because they are not fragmented now.
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGDemux development

Post by Rocky »

Yeah, some internal buffering would be good. I make one pass through the source files, writing things as I find them. Making multiple passes would be awful. Internal buffering (your -abuf and -vbuf) plus prealloc may be the ticket to nirvana. But first, gotta fix the GUI.

If you are using spinning platters you deserve what you get! We just bought four 2TB SSDs at good Black Friday prices. Is that racist?

Moose and squirrel just finished off a massive fatty ribeye with generous quantities of chardonnay. Not much will happen until tomorrow. :lol:
User avatar
redbtn
Posts: 44
Joined: Sat Nov 16, 2019 2:35 pm

Re: DGDemux development

Post by redbtn »

Rocky wrote:
Sun Dec 01, 2019 6:51 pm
Yeah, some internal buffering would be good. I make one pass through the source files, writing things as I find them. Making multiple passes would be awful. Internal buffering plus prealloc may be the ticket to nirvana. But first, gotta fix the GUI.

Moose and squirrel just finished off a massive fatty ribeye with chardonnay. Not much will happen until tomorrow. :lol:
I have to say that DGDemux do job with file fragmentation already better than eac3to. If it be possible to have internal buffer, it will be awesome. I'm going to sleep now, so things will continue tomorrow.
DAE avatar
zqslzwzw
Posts: 18
Joined: Fri Nov 29, 2019 8:35 am

Re: DGDemux development

Post by zqslzwzw »

redbtn wrote:
Sun Dec 01, 2019 6:33 pm
It seems like files have no fragmentation, but while they writes to disk, they are still writes in small pieces, because hdd speed drops from 150mbps to 40mbps. I think it happens because hdd writing small pieces and have to move heads very often between files.
I have to check it, but I think if demux only video it will be ok, but in parallel with audio and subs disk IO performance is dropping. And demux takes about 2 times more than it can be.
I'm talking about demux process. After demux is finished, work with files is ok now, because they are not fragmented now.
It seems that the preallocation is not enough. May be cache with the size of tens Megabytes for each stream in app level is still in demand.
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGDemux development

Post by Rocky »

Sleep well. Check your neck in the morning. Natasha is sneaking around.
User avatar
redbtn
Posts: 44
Joined: Sat Nov 16, 2019 2:35 pm

Re: DGDemux development

Post by redbtn »

zqslzwzw wrote:
Sun Dec 01, 2019 6:58 pm
It seems that the preallocation is not enough. May be cache with the size of tens Megabytes for each stream in app level is still in demand.
Like I already said, at least 1GB will be perfect. Hoomins who demuxing UHD Bluray have to have at least 16GB RAM, so there is no problem I think.
User avatar
redbtn
Posts: 44
Joined: Sat Nov 16, 2019 2:35 pm

Re: DGDemux development

Post by redbtn »

Rocky wrote:
Sun Dec 01, 2019 6:58 pm
Sleep well. Check your neck in the morning. Natasha is sneaking around.
Sure! You too!
Good job for today!
User avatar
Bullwinkle
Posts: 338
Joined: Thu Sep 05, 2019 6:37 pm

Re: DGDemux development

Post by Bullwinkle »

Hoomins are paranoid. If prealloc fails nothing changes. There is simply no prealloc. You get just what you wanted. Could let user know about it but most are dumbasses so why bother?
User avatar
redbtn
Posts: 44
Joined: Sat Nov 16, 2019 2:35 pm

Re: DGDemux development

Post by redbtn »

Bullwinkle wrote:
Sun Dec 01, 2019 7:04 pm
Hoomins are paranoid. If prealloc fails nothing changes. There is simply no prealloc. You get just what you wanted. Could let user know about it but most are dumbasses so why bother?
DGDemux users should be smarter than usual users. Others can use other one-click stuff.
DAE avatar
Guest

Re: DGDemux development

Post by Guest »

Bullwinkle wrote:
Sun Dec 01, 2019 7:04 pm
Hoomins are paranoid. If prealloc fails nothing changes. There is simply no prealloc. You get just what you wanted. Could let user know about it but most are dumbasses so why bother?
Or Hoomins could just use SSDs and have 32 GB of RAM, no problem then
User avatar
redbtn
Posts: 44
Joined: Sat Nov 16, 2019 2:35 pm

Re: DGDemux development

Post by redbtn »

gonca wrote:
Sun Dec 01, 2019 7:21 pm
Or Hoomins could just use SSDs and have 32 GB of RAM, no problem then
I have 32gb of RAM, but it's not helping while we don't have an internal buffer. SSD, yeah, but when I'm working with BD's I need a lot of space, but my SSD only 256gb
User avatar
Natasha
Posts: 150
Joined: Wed Nov 20, 2019 11:11 am

Re: DGDemux development

Post by Natasha »

You are so brilliant in your analyses, guys! You could teach me a lot, with a firm hand, and I could be a great pupil. Thirsting for you.
DAE avatar
Guest

Re: DGDemux development

Post by Guest »

redbtn wrote:
Sun Dec 01, 2019 7:27 pm
gonca wrote:
Sun Dec 01, 2019 7:21 pm
Or Hoomins could just use SSDs and have 32 GB of RAM, no problem then
I have 32gb of RAM, but it's not helping while we don't have an internal buffer. SSD, yeah, but when I'm working with BD's I need a lot of space, but my SSD only 256gb
I work a bit with UHD, but my workflow is such that I use 4 SSDs
User avatar
Boris
Posts: 92
Joined: Sun Nov 10, 2019 2:55 pm

Re: DGDemux development

Post by Boris »

So easy to make OS do file buffering. Don't miss obvious! Special on gold is expiring, don't miss out. And you can get date with Natasha. She will melt your heart and steal your soul.
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGDemux development

Post by Rocky »

Found the minimize bug. Now add file buffering and...whee!
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGDemux development

Post by Rocky »

Here is version 1.0.0.9:

* The finished sound is now an external WAV. If you want it, keep it with DGDemuxGUI.exe. If you don't
want it, you can delete it, rename it, or replace it with something else.

* The minimize/maximize bug was fixed.

* The progress box now shows "Done!" upon completion.

* The stream listing now shows the video type.

* Preallocation and file buffering were implemented to reduce fragmentation of the output
files. If you want minimally fragmented files, be sure to have masses of free space on the
destination drive. Preallocation amounts are 75GB for video, 10GB for each audio, and 500MB
for each sup. File buffering is set at 1MB per file (increasing beyond that has minimal effect).

http://rationalqm.us/dgdemux/DGDemux_1009.zip
User avatar
redbtn
Posts: 44
Joined: Sat Nov 16, 2019 2:35 pm

Re: DGDemux development

Post by redbtn »

Umm. I don't know how to test it properly yet. But I don't see much difference vs 1.0.0.8. Can you do 500mb RAM buffer for each stream for test?

PS: Minimize bug is fixed now, thank you! Thanks for other improvements too.
User avatar
Bullwinkle
Posts: 338
Joined: Thu Sep 05, 2019 6:37 pm

Re: DGDemux development

Post by Bullwinkle »

Nothing like that can help. The OS manages physical writes to the disk from the buffers. Many have benchmarked this and found increasing beyond 1MB file buffer brings you nothing; me too. Are you chasing unicorns? What problem do you have?

"Umm. I don't know how to test it properly yet."

Let us know when you figure it out.
User avatar
redbtn
Posts: 44
Joined: Sat Nov 16, 2019 2:35 pm

Re: DGDemux development

Post by redbtn »

Bullwinkle wrote:
Mon Dec 02, 2019 1:40 pm
Nothing like that can help. The OS manages physical writes to the disk from the buffers. Many have benchmarked this and found increasing beyond 1MB file buffer brings you nothing; me too. Are you chasing unicorns? What problem do you have?
Actually I don't have any issues for now. All looks great. I thought about improving things which works good, you are right, maybe I'm chasing unicorns.
User avatar
Bullwinkle
Posts: 338
Joined: Thu Sep 05, 2019 6:37 pm

Re: DGDemux development

Post by Bullwinkle »

It's called hoomin error, hoomin paranoia. Try to learn from it.

Now we have to focus on the demuxed streams, checking basic functionality, AV sync, etc.

Waiting on an important upload right now...
DAE avatar
rack04
Posts: 32
Joined: Tue Sep 21, 2010 6:45 am

Re: DGDemux development

Post by rack04 »

Nit pick comment but could you change the try icon? I would like to have both DGDecNV and DGDemux in my taskbar without having to guess which is which. Thanks and great work.
Post Reply