DGDemux development

[Rocky]

I implemented code to avoid parsing the base M2TS streams when only the MVC video is selected for demuxing. Please re-download:

http://rationalqm.us/dgdemux/binaries/D ... c_test.rar

It works fine. In a way it works too fine, because it is blazingly fast compared to demuxing the base video. This just indicates that there is baggage from DGIndexNV that could potentially be excised to speed up demuxing substantially. I'll definitely look into that. But for now I think we are close to a slipstream. Please test it thoroughly. Thank you.

A little more on the potential speedup... When I created DGDemux from DGIndexNV I did some quick-and-dirty things to get free of nVidia APIs. One thing was that, instead of removing all the indexing code, I simply allowed it to run but redirected the DGI file to /nul. Hanging my head in shame! Obviously, getting rid of that indexing code is going to significantly accelerate the application.

[Emulgator]

Test demuxed the smallest 3D cut of MIB_3_3D.
Mediainfo recognises both .avc and .mvc streams properly.

NLE: VegasPro 14..16 on import can not recognise .mvc, I renamed to .avc.
Tries to index, but fails. To be expected, Vegas wasn't made for that.

For fun I let tsMuxeR last build mux a .m2ts from both .avc and .mvc video streams: no complaints.
Playback with MPC-HC (no 3D screen available ATM): Base view looks good.
LAV Splitter sees both video streams.
GraphEdit builds a graph from the remuxed .m2ts
That is all at the moment I can do with it.

Looks good to me, many thanks !

[Guest]

Like I said before, don't do 3D so I don't have the screen for it
Demuxed MVC by itself, that is quick
Demuxed AVC and MVC, fed tsMuxer these two streams and created a 3D ISO
Original playback shows left and right view side by side
Remuxed playback shows the identical image
So, as far as I can test, it looks good to me

[Rocky]

Thank you, gentlemen!

I'll probably wait to slipstream so I can address renols' THD issue. While I'm waiting for the disk I'll look at speeding up the main demuxing. No reason it shouldn't be as fast as the MVC demuxing. Did Sherman do good on MVC demuxing, or what? Where is that boy when we need him?

[Curly]

Where is that boy when we need him?

Waiting for his spurs? I think it is time. I'll keep him on the straight and narrow! Nyuk, nyuk.

[Bullwinkle]

Curly, do you know how to code?

[Emulgator]

Tested tsMuxeR for demuxing comparison:
Demuxed the same MIB_3_3D 00689.mpls using tsMuxeR.
.dts sizes and files come out identical.
.sups do differ.
.avc does differ tiny bits (DGDemux is a tiny bit smaller)
.mvc does differ, DGDemux is bigger.
Says nothing ATM, I guess, Sherman did well

[Curly]

Curly, do you know how to code?

I made broccoli soup last night, so coding is no big deal to me. How about you?

[Curly]

Tested tsMuxeR for demuxing comparison:
Demuxed the same MIB_3_3D 00689.mpls using tsMuxeR.
.dts sizes and files come out identical.
.sups do differ.
.avc does differ tiny bits (DGDemux is a tiny bit smaller)
.mvc does differ, DGDemux is bigger.
Says nothing ATM, I guess, Sherman did well

Winkie wants me to pretend to be intelligent so here goes...
Sups will differ because we use a super-secret algorithm. They are still finely sinked to the pictures, so no sweat. Am I doing good? MVC is bigger cuz we doesn't delete filler shiite (take that profanity filter). Don't know about AVC. I guess smaller is better, unless you're talking about, well, you know.

[Guest]

U pretend real good

[Bullwinkle]

Broccoli, yech! Anyway, Mooses are so big that we can't easily get down to munch that low stuff. Tree bark and leaves, now you're talking my language.

I'm a system architect. I don't dirty my hooves with coding. Students and apprentices, OK?

[Curly]

U pretend real good

Best coder evah!

[Rocky]

Just for fun I implemented the Sherman method for the main demuxing (not just for the MVC stream). So far I have completed AC3 demuxing. So my operation was to demux all 5 AC3 streams from TOY_STORY_4_3D. Here are the timings:

DGDemux old way: 2mins 47 seconds
DGDemux new way: 48 seconds

As you can see, the new way is 3.5 times faster! Let's continue with this effort.

[admin]

By unanimous decision of the forum team, Sherman has been awarded his spurs. Congratulations!

[Levi]

Way to go, Sherman. You are a maestro of coding!

[Rocky]

Here is a test version with all the fixes so far:

MVC support
THD gaps fix
mono LPCM fix

http://rationalqm.us/dgdemux/binaries/DGDemux_test.rar

Feedback will be appreciated as always. DGIndexNV will get the same fix. Hope to slipstream both tomorrow if no issues are found.

[Guest 2]

Is it possible to add to the todo list the support of single m2ts?

[asowaboc]

So, this software doesn't leverage eac3to at all? I'm considering trying it, since eac3to is giving me sync errors and other problems with HDR discs...

[Rocky]

Correct. Totally new application and nothing to do with EAC3TO.

[Rocky]

Is it possible to add to the todo list the support of single m2ts?

It's already on there, that is, to accept a list of m2ts files, said list of course possibly having a single member. However, realistically, it probably won't happen. Too much has been engineered around the MPLS playlist. You can use DGIndexNV for that case.

[Guest]

DGIndexNV allows more than one running instance
Any plans to add that capability to DGDemux?

[Rocky]

Never thought about that. We'll see. Right now everything is focused on the speedup.

[Guest]

Thanks for considering it
It would as a sort of batch/queue system

[Sherman]

Thought I'd give an update on the speedup project.

I have all the video (main, secondary, and MVC), AC3 audio, and LPCM audio converted and working. Should be able to knock out the rest of the formats and PGS subtitles pretty quick. Then some odds and ends like garbage stripping, preallocation, DELAY values, filler NALU stripping, episode demuxing. I think that's it. Still seeing 300-350% speedups.

I found one thing that had me all hot and bothered for a while. I found that everything is being demuxed with _write() instead of fwrite(). That is unbuffered and so is theoretically quite slower. I erroneously thought at first it was just these unbuffered _write()'s responsible for everything but testing refuted that. You see, in practice, for video, it's written by NALU and most of the NALUs are very big, so the penalty is small. For the audio the total demuxed sizes are fairly small compared to video so the penalty again is not great. The case of uncompressed audio is significant though. Some of those can get up to the 10GB level on some disks. So I converted everything to fwrite(). We'll get a speedup from that and, of course, from the new architecture which avoids all the nested buffering and parsing.

[Rocky]

Awesome, Sherman. Really looking forward to the final product.

_write() instead of fwrite()... Wondering how that happened. The original DVD2AVI and DGIndex both have fwrite()'s so what made me change that? Here is what I think happened. When I made DGSplit I found _write() faster (and DGSplit is blazingly fast) but in that use there are very large write sizes, so the lack of buffering was inconsequential. I must have wrongly got into my head that _write() is therefore all-around better. Another hang-my-head-in-shame moment. Cut me some slack. Way back then I was still learning to code competently. The upside of course is that it is easily fixable, as you have discovered. And as you point out, most of the gains are coming from the new architecture.

In case you are wondering, we can use _read() on the input side because reads are buffered internally. And of course DGSource() is not affected because it does not write any files.