DGDemux development

Post Reply
User avatar
Rocky
Posts: 3555
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGDemux development

Post by Rocky »

Yeah, some internal buffering would be good. I make one pass through the source files, writing things as I find them. Making multiple passes would be awful. Internal buffering (your -abuf and -vbuf) plus prealloc may be the ticket to nirvana. But first, gotta fix the GUI.

If you are using spinning platters you deserve what you get! We just bought four 2TB SSDs at good Black Friday prices. Is that racist?

Moose and squirrel just finished off a massive fatty ribeye with generous quantities of chardonnay. Not much will happen until tomorrow. :lol:
User avatar
redbtn
Posts: 44
Joined: Sat Nov 16, 2019 2:35 pm

Re: DGDemux development

Post by redbtn »

Rocky wrote:
Sun Dec 01, 2019 6:51 pm
Yeah, some internal buffering would be good. I make one pass through the source files, writing things as I find them. Making multiple passes would be awful. Internal buffering plus prealloc may be the ticket to nirvana. But first, gotta fix the GUI.

Moose and squirrel just finished off a massive fatty ribeye with chardonnay. Not much will happen until tomorrow. :lol:
I have to say that DGDemux do job with file fragmentation already better than eac3to. If it be possible to have internal buffer, it will be awesome. I'm going to sleep now, so things will continue tomorrow.
DAE avatar
zqslzwzw
Posts: 18
Joined: Fri Nov 29, 2019 8:35 am

Re: DGDemux development

Post by zqslzwzw »

redbtn wrote:
Sun Dec 01, 2019 6:33 pm
It seems like files have no fragmentation, but while they writes to disk, they are still writes in small pieces, because hdd speed drops from 150mbps to 40mbps. I think it happens because hdd writing small pieces and have to move heads very often between files.
I have to check it, but I think if demux only video it will be ok, but in parallel with audio and subs disk IO performance is dropping. And demux takes about 2 times more than it can be.
I'm talking about demux process. After demux is finished, work with files is ok now, because they are not fragmented now.
It seems that the preallocation is not enough. May be cache with the size of tens Megabytes for each stream in app level is still in demand.
User avatar
Rocky
Posts: 3555
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGDemux development

Post by Rocky »

Sleep well. Check your neck in the morning. Natasha is sneaking around.
User avatar
redbtn
Posts: 44
Joined: Sat Nov 16, 2019 2:35 pm

Re: DGDemux development

Post by redbtn »

zqslzwzw wrote:
Sun Dec 01, 2019 6:58 pm
It seems that the preallocation is not enough. May be cache with the size of tens Megabytes for each stream in app level is still in demand.
Like I already said, at least 1GB will be perfect. Hoomins who demuxing UHD Bluray have to have at least 16GB RAM, so there is no problem I think.
User avatar
redbtn
Posts: 44
Joined: Sat Nov 16, 2019 2:35 pm

Re: DGDemux development

Post by redbtn »

Rocky wrote:
Sun Dec 01, 2019 6:58 pm
Sleep well. Check your neck in the morning. Natasha is sneaking around.
Sure! You too!
Good job for today!
User avatar
Bullwinkle
Posts: 338
Joined: Thu Sep 05, 2019 6:37 pm

Re: DGDemux development

Post by Bullwinkle »

Hoomins are paranoid. If prealloc fails nothing changes. There is simply no prealloc. You get just what you wanted. Could let user know about it but most are dumbasses so why bother?
User avatar
redbtn
Posts: 44
Joined: Sat Nov 16, 2019 2:35 pm

Re: DGDemux development

Post by redbtn »

Bullwinkle wrote:
Sun Dec 01, 2019 7:04 pm
Hoomins are paranoid. If prealloc fails nothing changes. There is simply no prealloc. You get just what you wanted. Could let user know about it but most are dumbasses so why bother?
DGDemux users should be smarter than usual users. Others can use other one-click stuff.
DAE avatar
Guest

Re: DGDemux development

Post by Guest »

Bullwinkle wrote:
Sun Dec 01, 2019 7:04 pm
Hoomins are paranoid. If prealloc fails nothing changes. There is simply no prealloc. You get just what you wanted. Could let user know about it but most are dumbasses so why bother?
Or Hoomins could just use SSDs and have 32 GB of RAM, no problem then
User avatar
redbtn
Posts: 44
Joined: Sat Nov 16, 2019 2:35 pm

Re: DGDemux development

Post by redbtn »

gonca wrote:
Sun Dec 01, 2019 7:21 pm
Or Hoomins could just use SSDs and have 32 GB of RAM, no problem then
I have 32gb of RAM, but it's not helping while we don't have an internal buffer. SSD, yeah, but when I'm working with BD's I need a lot of space, but my SSD only 256gb
User avatar
Natasha
Posts: 150
Joined: Wed Nov 20, 2019 11:11 am

Re: DGDemux development

Post by Natasha »

You are so brilliant in your analyses, guys! You could teach me a lot, with a firm hand, and I could be a great pupil. Thirsting for you.
DAE avatar
Guest

Re: DGDemux development

Post by Guest »

redbtn wrote:
Sun Dec 01, 2019 7:27 pm
gonca wrote:
Sun Dec 01, 2019 7:21 pm
Or Hoomins could just use SSDs and have 32 GB of RAM, no problem then
I have 32gb of RAM, but it's not helping while we don't have an internal buffer. SSD, yeah, but when I'm working with BD's I need a lot of space, but my SSD only 256gb
I work a bit with UHD, but my workflow is such that I use 4 SSDs
User avatar
Boris
Posts: 92
Joined: Sun Nov 10, 2019 2:55 pm

Re: DGDemux development

Post by Boris »

So easy to make OS do file buffering. Don't miss obvious! Special on gold is expiring, don't miss out. And you can get date with Natasha. She will melt your heart and steal your soul.
User avatar
Rocky
Posts: 3555
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGDemux development

Post by Rocky »

Found the minimize bug. Now add file buffering and...whee!
User avatar
Rocky
Posts: 3555
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGDemux development

Post by Rocky »

Here is version 1.0.0.9:

* The finished sound is now an external WAV. If you want it, keep it with DGDemuxGUI.exe. If you don't
want it, you can delete it, rename it, or replace it with something else.

* The minimize/maximize bug was fixed.

* The progress box now shows "Done!" upon completion.

* The stream listing now shows the video type.

* Preallocation and file buffering were implemented to reduce fragmentation of the output
files. If you want minimally fragmented files, be sure to have masses of free space on the
destination drive. Preallocation amounts are 75GB for video, 10GB for each audio, and 500MB
for each sup. File buffering is set at 1MB per file (increasing beyond that has minimal effect).

http://rationalqm.us/dgdemux/DGDemux_1009.zip
User avatar
redbtn
Posts: 44
Joined: Sat Nov 16, 2019 2:35 pm

Re: DGDemux development

Post by redbtn »

Umm. I don't know how to test it properly yet. But I don't see much difference vs 1.0.0.8. Can you do 500mb RAM buffer for each stream for test?

PS: Minimize bug is fixed now, thank you! Thanks for other improvements too.
User avatar
Bullwinkle
Posts: 338
Joined: Thu Sep 05, 2019 6:37 pm

Re: DGDemux development

Post by Bullwinkle »

Nothing like that can help. The OS manages physical writes to the disk from the buffers. Many have benchmarked this and found increasing beyond 1MB file buffer brings you nothing; me too. Are you chasing unicorns? What problem do you have?

"Umm. I don't know how to test it properly yet."

Let us know when you figure it out.
User avatar
redbtn
Posts: 44
Joined: Sat Nov 16, 2019 2:35 pm

Re: DGDemux development

Post by redbtn »

Bullwinkle wrote:
Mon Dec 02, 2019 1:40 pm
Nothing like that can help. The OS manages physical writes to the disk from the buffers. Many have benchmarked this and found increasing beyond 1MB file buffer brings you nothing; me too. Are you chasing unicorns? What problem do you have?
Actually I don't have any issues for now. All looks great. I thought about improving things which works good, you are right, maybe I'm chasing unicorns.
User avatar
Bullwinkle
Posts: 338
Joined: Thu Sep 05, 2019 6:37 pm

Re: DGDemux development

Post by Bullwinkle »

It's called hoomin error, hoomin paranoia. Try to learn from it.

Now we have to focus on the demuxed streams, checking basic functionality, AV sync, etc.

Waiting on an important upload right now...
DAE avatar
rack04
Posts: 32
Joined: Tue Sep 21, 2010 6:45 am

Re: DGDemux development

Post by rack04 »

Nit pick comment but could you change the try icon? I would like to have both DGDecNV and DGDemux in my taskbar without having to guess which is which. Thanks and great work.
DAE avatar
zqslzwzw
Posts: 18
Joined: Fri Nov 29, 2019 8:35 am

Re: DGDemux development

Post by zqslzwzw »

Rocky wrote:
Sun Dec 01, 2019 8:53 am
zqslzwzw wrote:
Sun Dec 01, 2019 5:26 am
Could you go to that private track to obtain this source? If you do not have its account, I am pleased to invite you as long as you are willing to be and agree with its rule.
We've been looking for DTS HDHR streams so this would help both of us. If I gave you an FTP would you be able to upload it there? We don't do torrents.
The reported BD disk has been uploaded to this FTP. And an explanation of duplication procedure is placed aside.
Thank you for effort!
User avatar
redbtn
Posts: 44
Joined: Sat Nov 16, 2019 2:35 pm

Re: DGDemux development

Post by redbtn »

Another "Browse" bug
Attachments
Video_2019-12-03_175929-2019-12-03.rar
(1015.36 KiB) Downloaded 346 times
User avatar
Rocky
Posts: 3555
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGDemux development

Post by Rocky »

Thanks, redbtn, for pointing that out.

I really need to know which MPLS you used for that 10ms thing.
User avatar
Rocky
Posts: 3555
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGDemux development

Post by Rocky »

zqslzwzw wrote:
Mon Dec 02, 2019 9:45 pm
The reported BD disk has been uploaded to this FTP. And an explanation of duplication procedure is placed aside.
Thank you for effort!
Thank you. Duplicated and under investigation.
User avatar
Rocky
Posts: 3555
Joined: Fri Sep 06, 2019 12:57 pm

Re: DGDemux development

Post by Rocky »

OK, zqslzwzw, I have this fixed. I want to research a few corner cases versus the DTS spec before releasing a fix.

@redbtn

Fixed your new browse bug, too. MPLS for your 10ms thing? Third time asking!
Post Reply