DGDemux development

[Bullwinkle]

Broccoli, yech! Anyway, Mooses are so big that we can't easily get down to munch that low stuff. Tree bark and leaves, now you're talking my language.

I'm a system architect. I don't dirty my hooves with coding. Students and apprentices, OK?

[Curly]

U pretend real good

Best coder evah!

[Rocky]

Just for fun I implemented the Sherman method for the main demuxing (not just for the MVC stream). So far I have completed AC3 demuxing. So my operation was to demux all 5 AC3 streams from TOY_STORY_4_3D. Here are the timings:

DGDemux old way: 2mins 47 seconds
DGDemux new way: 48 seconds

As you can see, the new way is 3.5 times faster! Let's continue with this effort.

[admin]

By unanimous decision of the forum team, Sherman has been awarded his spurs. Congratulations!

[Levi]

Way to go, Sherman. You are a maestro of coding!

[Rocky]

Here is a test version with all the fixes so far:

MVC support
THD gaps fix
mono LPCM fix

http://rationalqm.us/dgdemux/binaries/DGDemux_test.rar

Feedback will be appreciated as always. DGIndexNV will get the same fix. Hope to slipstream both tomorrow if no issues are found.

[Guest 2]

Is it possible to add to the todo list the support of single m2ts?

[asowaboc]

So, this software doesn't leverage eac3to at all? I'm considering trying it, since eac3to is giving me sync errors and other problems with HDR discs...

[Rocky]

Correct. Totally new application and nothing to do with EAC3TO.

[Rocky]

Is it possible to add to the todo list the support of single m2ts?

It's already on there, that is, to accept a list of m2ts files, said list of course possibly having a single member. However, realistically, it probably won't happen. Too much has been engineered around the MPLS playlist. You can use DGIndexNV for that case.

[Guest]

DGIndexNV allows more than one running instance
Any plans to add that capability to DGDemux?

[Rocky]

Never thought about that. We'll see. Right now everything is focused on the speedup.

[Guest]

Thanks for considering it
It would as a sort of batch/queue system

[Sherman]

Thought I'd give an update on the speedup project.

I have all the video (main, secondary, and MVC), AC3 audio, and LPCM audio converted and working. Should be able to knock out the rest of the formats and PGS subtitles pretty quick. Then some odds and ends like garbage stripping, preallocation, DELAY values, filler NALU stripping, episode demuxing. I think that's it. Still seeing 300-350% speedups.

I found one thing that had me all hot and bothered for a while. I found that everything is being demuxed with _write() instead of fwrite(). That is unbuffered and so is theoretically quite slower. I erroneously thought at first it was just these unbuffered _write()'s responsible for everything but testing refuted that. You see, in practice, for video, it's written by NALU and most of the NALUs are very big, so the penalty is small. For the audio the total demuxed sizes are fairly small compared to video so the penalty again is not great. The case of uncompressed audio is significant though. Some of those can get up to the 10GB level on some disks. So I converted everything to fwrite(). We'll get a speedup from that and, of course, from the new architecture which avoids all the nested buffering and parsing.

[Rocky]

Awesome, Sherman. Really looking forward to the final product.

_write() instead of fwrite()... Wondering how that happened. The original DVD2AVI and DGIndex both have fwrite()'s so what made me change that? Here is what I think happened. When I made DGSplit I found _write() faster (and DGSplit is blazingly fast) but in that use there are very large write sizes, so the lack of buffering was inconsequential. I must have wrongly got into my head that _write() is therefore all-around better. Another hang-my-head-in-shame moment. Cut me some slack. Way back then I was still learning to code competently. The upside of course is that it is easily fixable, as you have discovered. And as you point out, most of the gains are coming from the new architecture.

In case you are wondering, we can use _read() on the input side because reads are buffered internally. And of course DGSource() is not affected because it does not write any files.

[Natasha]

Excuses, excuses. What a loser.

[Rocky]

It can't be too bad, Natty. DG tools have made me a very well-off little squirrel. It's not just about acorns.

[Guest]

I found one thing that had me all hot and bothered

You're to young for that sort of thing Sherman

[Rocky]

Pretty sure he meant:

2. (idiomatic, colloquial, slang) aggravated or irritated

[asowaboc]

...there's no progress bar? Also, DGDemux seems to be creating .sup files that are invalid and can't be read by MKVToolNix/BDSup2Sub.

[BDgeek2]

Please bear with me, this is my first post and it may sound like a very newbie question.

I've been using DGDemux for a couple of months now and I'm loving it. Thanks Rocky, fantastic software!

Now, on the updates/fixes log for slipstream 30, it's stated:
"Fixed file gaps processing for THD audio"

What does it mean in pratical terms?
For exampled, I've demuxed Toy Story 4 UHD that has a DTHD Atmos track with slipstream 29 and muxed it back with MKVToolnix to include the UHD video, saind DTHD Atmos track, plus the BD DTS-HD MA track.
Does it mean this THD track on my mux is incorrect somehow? Does it have any sync issues?

Thanks a lot!

[Rocky]

I've demuxed Toy Story 4 UHD that has a DTHD Atmos track with slipstream 29 and muxed it back with MKVToolnix to include the UHD video, saind DTHD Atmos track, plus the BD DTS-HD MA track.
Does it mean this THD track on my mux is incorrect somehow? Does it have any sync issues?

No, it is fine. That fix was for uncommon streams that did not start all major frames with payload_unit_start_indicator=1. TOY_STORY does not do that and it was one of the disks that I had tested with.

[Rocky]

...there's no progress bar? Also, DGDemux seems to be creating .sup files that are invalid and can't be read by MKVToolNix/BDSup2Sub.

There is a marque progress bar and the current m2ts is shown.

Regarding your subs issue, please tell the disk and PID and specify your exact process to show the issue. Also state the error that you encounter.

[Rocky]

DGDemux seems to be creating .sup files that are invalid and can't be read by MKVToolNix/BDSup2Sub.

I just did some spot checks and cannot find any problems. The files are accepted by SubtitleEdit and MKVToolNix without any issues. Please specify the disk, stream, and your process that creates your issue. You don't want Bullwinkle thinking you are FUD'ing us.

[Bullwinkle]

You don't want Bullwinkle thinking you are FUD'ing us.

Too late now. Sayonara asowaboc!