DGDemux development
- Bullwinkle
- Posts: 338
- Joined: Thu Sep 05, 2019 6:37 pm
Re: DGDemux development
Broccoli, yech! Anyway, Mooses are so big that we can't easily get down to munch that low stuff. Tree bark and leaves, now you're talking my language.
I'm a system architect. I don't dirty my hooves with coding. Students and apprentices, OK?
I'm a system architect. I don't dirty my hooves with coding. Students and apprentices, OK?
Re: DGDemux development
Curly Howard
Director of EAC3TO Development
Director of EAC3TO Development
Re: DGDemux development
Just for fun I implemented the Sherman method for the main demuxing (not just for the MVC stream). So far I have completed AC3 demuxing. So my operation was to demux all 5 AC3 streams from TOY_STORY_4_3D. Here are the timings:
DGDemux old way: 2mins 47 seconds
DGDemux new way: 48 seconds
As you can see, the new way is 3.5 times faster! Let's continue with this effort.
DGDemux old way: 2mins 47 seconds
DGDemux new way: 48 seconds
As you can see, the new way is 3.5 times faster! Let's continue with this effort.
Re: DGDemux development
By unanimous decision of the forum team, Sherman has been awarded his spurs. Congratulations!
Re: DGDemux development
Way to go, Sherman. You are a maestro of coding!
Re: DGDemux development
Here is a test version with all the fixes so far:
MVC support
THD gaps fix
mono LPCM fix
http://rationalqm.us/dgdemux/binaries/DGDemux_test.rar
Feedback will be appreciated as always. DGIndexNV will get the same fix. Hope to slipstream both tomorrow if no issues are found.
MVC support
THD gaps fix
mono LPCM fix
http://rationalqm.us/dgdemux/binaries/DGDemux_test.rar
Feedback will be appreciated as always. DGIndexNV will get the same fix. Hope to slipstream both tomorrow if no issues are found.
Re: DGDemux development
Is it possible to add to the todo list the support of single m2ts?
Re: DGDemux development
So, this software doesn't leverage eac3to at all? I'm considering trying it, since eac3to is giving me sync errors and other problems with HDR discs...
Re: DGDemux development
Correct. Totally new application and nothing to do with EAC3TO.
Re: DGDemux development
It's already on there, that is, to accept a list of m2ts files, said list of course possibly having a single member. However, realistically, it probably won't happen. Too much has been engineered around the MPLS playlist. You can use DGIndexNV for that case.
Re: DGDemux development
DGIndexNV allows more than one running instance
Any plans to add that capability to DGDemux?
Any plans to add that capability to DGDemux?
Re: DGDemux development
Never thought about that. We'll see. Right now everything is focused on the speedup.
Re: DGDemux development
Thanks for considering it
It would as a sort of batch/queue system
It would as a sort of batch/queue system
Re: DGDemux development
Thought I'd give an update on the speedup project.
I have all the video (main, secondary, and MVC), AC3 audio, and LPCM audio converted and working. Should be able to knock out the rest of the formats and PGS subtitles pretty quick. Then some odds and ends like garbage stripping, preallocation, DELAY values, filler NALU stripping, episode demuxing. I think that's it. Still seeing 300-350% speedups.
I found one thing that had me all hot and bothered for a while. I found that everything is being demuxed with _write() instead of fwrite(). That is unbuffered and so is theoretically quite slower. I erroneously thought at first it was just these unbuffered _write()'s responsible for everything but testing refuted that. You see, in practice, for video, it's written by NALU and most of the NALUs are very big, so the penalty is small. For the audio the total demuxed sizes are fairly small compared to video so the penalty again is not great. The case of uncompressed audio is significant though. Some of those can get up to the 10GB level on some disks. So I converted everything to fwrite(). We'll get a speedup from that and, of course, from the new architecture which avoids all the nested buffering and parsing.
I have all the video (main, secondary, and MVC), AC3 audio, and LPCM audio converted and working. Should be able to knock out the rest of the formats and PGS subtitles pretty quick. Then some odds and ends like garbage stripping, preallocation, DELAY values, filler NALU stripping, episode demuxing. I think that's it. Still seeing 300-350% speedups.
I found one thing that had me all hot and bothered for a while. I found that everything is being demuxed with _write() instead of fwrite(). That is unbuffered and so is theoretically quite slower. I erroneously thought at first it was just these unbuffered _write()'s responsible for everything but testing refuted that. You see, in practice, for video, it's written by NALU and most of the NALUs are very big, so the penalty is small. For the audio the total demuxed sizes are fairly small compared to video so the penalty again is not great. The case of uncompressed audio is significant though. Some of those can get up to the 10GB level on some disks. So I converted everything to fwrite(). We'll get a speedup from that and, of course, from the new architecture which avoids all the nested buffering and parsing.
Sherman Peabody
Director of Linux Development
Director of Linux Development
Re: DGDemux development
Awesome, Sherman. Really looking forward to the final product.
_write() instead of fwrite()... Wondering how that happened. The original DVD2AVI and DGIndex both have fwrite()'s so what made me change that? Here is what I think happened. When I made DGSplit I found _write() faster (and DGSplit is blazingly fast) but in that use there are very large write sizes, so the lack of buffering was inconsequential. I must have wrongly got into my head that _write() is therefore all-around better. Another hang-my-head-in-shame moment. Cut me some slack. Way back then I was still learning to code competently. The upside of course is that it is easily fixable, as you have discovered. And as you point out, most of the gains are coming from the new architecture.
In case you are wondering, we can use _read() on the input side because reads are buffered internally. And of course DGSource() is not affected because it does not write any files.
_write() instead of fwrite()... Wondering how that happened. The original DVD2AVI and DGIndex both have fwrite()'s so what made me change that? Here is what I think happened. When I made DGSplit I found _write() faster (and DGSplit is blazingly fast) but in that use there are very large write sizes, so the lack of buffering was inconsequential. I must have wrongly got into my head that _write() is therefore all-around better. Another hang-my-head-in-shame moment. Cut me some slack. Way back then I was still learning to code competently. The upside of course is that it is easily fixable, as you have discovered. And as you point out, most of the gains are coming from the new architecture.
In case you are wondering, we can use _read() on the input side because reads are buffered internally. And of course DGSource() is not affected because it does not write any files.
Re: DGDemux development
Excuses, excuses. What a loser.
Re: DGDemux development
It can't be too bad, Natty. DG tools have made me a very well-off little squirrel. It's not just about acorns.
Re: DGDemux development
You're to young for that sort of thing ShermanI found one thing that had me all hot and bothered
Re: DGDemux development
Pretty sure he meant:
2. (idiomatic, colloquial, slang) aggravated or irritated
2. (idiomatic, colloquial, slang) aggravated or irritated
Re: DGDemux development
...there's no progress bar? Also, DGDemux seems to be creating .sup files that are invalid and can't be read by MKVToolNix/BDSup2Sub.
Re: DGDemux development
Please bear with me, this is my first post and it may sound like a very newbie question.
I've been using DGDemux for a couple of months now and I'm loving it. Thanks Rocky, fantastic software!
Now, on the updates/fixes log for slipstream 30, it's stated:
"Fixed file gaps processing for THD audio"
What does it mean in pratical terms?
For exampled, I've demuxed Toy Story 4 UHD that has a DTHD Atmos track with slipstream 29 and muxed it back with MKVToolnix to include the UHD video, saind DTHD Atmos track, plus the BD DTS-HD MA track.
Does it mean this THD track on my mux is incorrect somehow? Does it have any sync issues?
Thanks a lot!
I've been using DGDemux for a couple of months now and I'm loving it. Thanks Rocky, fantastic software!
Now, on the updates/fixes log for slipstream 30, it's stated:
"Fixed file gaps processing for THD audio"
What does it mean in pratical terms?
For exampled, I've demuxed Toy Story 4 UHD that has a DTHD Atmos track with slipstream 29 and muxed it back with MKVToolnix to include the UHD video, saind DTHD Atmos track, plus the BD DTS-HD MA track.
Does it mean this THD track on my mux is incorrect somehow? Does it have any sync issues?
Thanks a lot!
Re: DGDemux development
No, it is fine. That fix was for uncommon streams that did not start all major frames with payload_unit_start_indicator=1. TOY_STORY does not do that and it was one of the disks that I had tested with.BDgeek2 wrote: ↑Tue Jul 07, 2020 10:58 pmI've demuxed Toy Story 4 UHD that has a DTHD Atmos track with slipstream 29 and muxed it back with MKVToolnix to include the UHD video, saind DTHD Atmos track, plus the BD DTS-HD MA track.
Does it mean this THD track on my mux is incorrect somehow? Does it have any sync issues?
Re: DGDemux development
There is a marque progress bar and the current m2ts is shown.
Regarding your subs issue, please tell the disk and PID and specify your exact process to show the issue. Also state the error that you encounter.
Re: DGDemux development
I just did some spot checks and cannot find any problems. The files are accepted by SubtitleEdit and MKVToolNix without any issues. Please specify the disk, stream, and your process that creates your issue. You don't want Bullwinkle thinking you are FUD'ing us.
- Bullwinkle
- Posts: 338
- Joined: Thu Sep 05, 2019 6:37 pm