[RESOLVED] MKV demux support

[admin]

Thanks for the info, jp. My work is driven by the sample files I receive, so...

[jpsdr]

Maybe you can create you own mkv with pgs this way :
From any m2ts (principal movie file with subtitles) Blu-Ray you have, extract the PGS with eac3to.
Command line would be something like this (if track 15 is a subtitle) :
eac3to 00000.m2ts 15: C:\Movie.sup
Then, create an mkv with mkvtoolnix muxing this sup inside. This way, you are sure to have a proper mkv.
To simply open sup file you have BDSup2sub, or SubtitleEdit.

It's past midnight for me now, so right now, direct to bed, i don't have time to prepare anything.
But, if tomorrow evening (leaving almost 24h), nobody provide you anything, i'll see if i can provide you things helping others.

Must confess, personnaly, i use eac3to to extract audio/PGS, and aegisub to extract ASS if necessary...

[admin]

Ok, thanks, I'll have a go at it. It would still be great to get samples, just to be sure.

[Guest]

Hmm, PGS. That may be a tough one. We'll see. Do you have a sample for me please? I found one on the web but mkvextract just produces garbage where I was expecting a nice image file or something. Anybody know anything about this? A general google search shows people asking how to demux PGS and I did not see a definitive answer. So help me out!

EDIT: Possible solution here:

http://www.makemkv.com/forum2/viewtopic.php?f=1&t=808

The process seems rather unwieldy.

That process appears to involve more than extracting
I was referring to simple extraction
How many test clips, and how long do you need, want?

[admin]

One or a few clips will be fine. Length is not that important but need enough for multiple subtitles. Bandwidth is cheap now.

I would aim to emulate mkvextract and produce just a sup file. From there it's up to you.

[Guest]

I'll start uploading now

[admin]

I've refreshed Guest 2.zip with the "Finished" fix, so please re-download and update.

[admin]

I found one stream with PGS in my streams database (collected over the years). The codec ID reported by gMKVExtractGUI is S_HDMV/PGS and it extracts to a sup file. Still, the more streams the better.

[admin]

I can't figure out from inspection how the MKV's S_HDMV/PGS blocks translate to the SUP file contents. I'll have to look at the source code for mkvextract to see what is going on. Tomorrow, or should I say Tom?

[Guest]

Tom?
How about Dick and Harry

[admin]

I'm close to having PGS working. The first big thing was that the data is zlib compressed in the MKV, so I had to enable zlib support in the matroska code and build zlib for windows, etc. That seems to be working as I can see proper PGS segment data now after decompressing the MKV blocks. Now working on parsing out the blocks to a SUP file.

Before finishing that, though, I am going to slipstream the Guest 2.zip version.

[jpsdr]

Do you mean you've until now handle mkv without zlib support ?
You're lucky, because in the mkv spec, you can choose to compress any steam, should it be audio, video or anything else.
Practical use, compressing audio and video give no benefit, so, in mkvtoolnix, default setting is to not compress audio/video, and compress others things like subtitles. Personnaly, i deactivate all compressions, because even if it's in mkv spec, some hardware players may not like it.
So, if we are talking about the compression that can be enable when merging mkv, it means :
- Audio and video can also be compressed.
- PGS are not always compressed.
Unless... I'm mistaken/mixing things... And the compression of mkv stream is automaticaly handled by your mkv parser, and zlib compressed is something within the specs of PGS ? (Like PNG, you can have uncompressed or zlib compressed data for the picture, but it's within the PNG spec).

Edit :
Looking again at mkvtoolnix gui, in it, you can just choose to enable/disable compression for subtitles. There was once, if my memory is not tricking me, in gui option, the possibility to choose something like : "compress all", "compress just subtitles", "compress nothing". I don't know if the change is just within the gui, but compress all streams is still possible (means still within specs) using advanced command line mode, or if specs have changed and compression of audio and video is not possible anymore.

Edit 2:
And, if again my memory is still not tricking me, there was also the possibility to choose what kind of compression (there was 2 or 3 possibilities, zlib was one of them). So, i don't know if the mkv specs had some "uncertain/unstable" period and have changed to something more restrictive, or if there is still all these possibilties, even if they're almost never used...

[admin]

I wouldn't call it lucky. Previously I supported only video and audio and never encountered an MKV with zlib compression enabled for video or audio, which as you say would make little sense even if it is possible. So there was no need for the zlib support until now.

Yes, I need to honor whether the streams are compressed or not. It's signaled in the MKV obviously.

I have not seen any compression other than zlib, and the MKV library code from Mike supports only zlib, so I will support only zlib for now, keeping open the option to revisit that.

[admin]

I slipstreamed the last Guest 2.zip changes as slipstream 125. Thus Guest 2.zip has been deleted.

[jpsdr]

Yes, I need to honor whether the streams are compressed or not, it's signaled in the MKV obviously.

Do you know by any chance if it's only zlib, or if there is also others compression format that can legaly be used ?
The fact that i remember (properly or not...?) once a gui where you can choose compression format is still bugging me...

[admin]

The MKV spec mentions zlib and header stripping. There are two more listed but they have lines drawn through them as if they have been deleted. Header stripping was already supported by DGDecNV. I looked at the current mkvtoolnix GUI and its compression drop-down offers only zlib (unless I have missed something).

[jpsdr]

Probably not. The two deleted probably existed only for a short time, and i've seen them during this short time. At least my memory is still working properly... for now...
Thanks for all your work.

[admin]

You're welcome, jp, and thank you for your contributions. Keep thinking and coding; it will delay cognitive decline.

I have PGS->SUP demuxing fully working. Demuxed SUPs are binary identical to SUPs from mkvextract. (Keep an eye on signed versus unsigned; it can bite you big time.)
Mosu

I'll slipstream that and then work on ASS and embedded fonts. Guest 2 has sent me a very challenging file.

Another thing I did was make the top-level Demux menu item go straight to the dialog, instead of showing a pulldown menu with only one item. Saving you guys and gals clicks! I also had to configure the demux list as autowrapping multiline to accommodate one of my MKV test files with 34 tracks total, including 25 PGS tracks! Can't remember who sent me that file. The evil Guest 2?

[Guest 2]

The evil Guest 2?

[admin]

You're my hero; that was tongue-in-cheek, or something. Really, I greatly appreciate your contributions.

[Guest 2]

Tried latest build with mkv (h264, 3 srt subs, 2 DTS and 2 AC3 + chapters).

I tried to demux only 3 srt subs and 2 DTS.

The 1st DTS was output with DGI, almost same time. The 1st srt was created as 0 byte file while creating DGI and then completed.

The 2nd DTS was created after the finished message, a lot after.

Then started 2nd srt and finally 3rd srt.

Is it correct with the current status of demux development?

P.S: I really like gMKVExtractGUI naming output. Could you please provide similar? If too much for someone, create a checkbox with the info to be put in filename.

[admin]

Thanks for your test results. Maybe you are seeing delays by Windows explorer in updating the folder contents. Can you please upload that stream to my FTP and I will check. "a lot after" is too vague for me to speculate. Streams are demuxed one-by-one but that runs in parallel with DGI creation. Parsing through an MKV is actually quite fast thanks to the container design. The "Finished" should not be given until all the demuxing completes internally. It would be possible to run the demuxes in one pass (not one-by-one) but it would be a massive rewrite.

I'll implement the naming as you suggest but probably not immediately.

[admin]

Ah, I just tested this again locally and found a bug with the Finished signaling. Also, the files aren't closed until DGIndexNV exits.

You are on your toes, Guest 2. Thank you! I'll slipstream fixes for it and PGS support shortly.

[admin]

OK, slipstreamed it. Things should be working as expected now, Guest 2.

At some point we'll need to test that CLI -a is still working as expected.

[jpsdr]

For zlib, i personnaly use it for writing PNG, but directly put the C files in the project, not a dll.
I'll tell you how i've done it, in case you would be interested.
Go to zlib official site, get the zlib1211.zip source files.
The needed files (to add in your project) are :
adler32.c
compress.c
crc32.c
deflate.c
infback.c
inffast.c
inflate.c
inftrees.c
trees.c
uncompr.c
zutil.c

To use, just "zlib.h" in #include and directly call uncompress.
Issue would be to guess the correct output size...
Probably some iterating process, begin with originalsize*10 for subtitles data, and originalsize*2 for audio/video data. In case of failling, double the buffer size and try again, and continue so.
As i personnaly use only compress2, allocating output same size as input always worked...