[RESOLVED] MKV demux support

Support forum for DGDecNV
User avatar
admin
Posts: 4551
Joined: Thu Sep 09, 2010 3:08 pm

Re: Feature Requests

Post by admin »

Thanks but it's not necessary. I'm linking the static version of the library for the next slipstream as I mentioned in the notifications thread. I already have it working. And the calls are already part of :bow: Mike Matsnev's Matroska code that I use and can't easily be changed. Decompression [using inflate()] is applied to the MKV blocks in memory individually. No worries about size. The inflate() call signals if more memory is needed and you can resize your buffer as needed.
User avatar
admin
Posts: 4551
Joined: Thu Sep 09, 2010 3:08 pm

Re: Feature Requests

Post by admin »

OK, status...

We have renols' issue with Win7. We have to wait for him to report on the fix. I went ahead and added it to slipstream 126 (slipstreamed the slipstream :lol:).

The current version doesn't do uncompressed PGS correctly, and I just found out MakeMKV does not compress the PGS from a blu-ray. I have uncompressed PGS fixed locally. Will slipstream when the Win7 fix is confirmed.
DAE avatar
jpsdr
Posts: 214
Joined: Tue Sep 21, 2010 4:16 am

Re: Feature Requests

Post by jpsdr »

No big deal, it was just in case it could be of use.

I don't realy understand the fact that you don't (for now) work with uncompressed PGS.
Now that you've included zlib, didn't you "just" check if the stream (whatever stream it could be : video, audio, subtitles and any kind/version of each) is compressed or not, and just deflate (or not) if necessary, as you said that in the mkv, there is the information telling if the stream is compressed or not ?
Or is it more complex ?
Because even if for audio and video, the chances to have compressed data is very low, for subtitles, should they be srt, ssa/ass or PGS (or others), they can be compressed or not with probably an equal repartition between compressed and uncompressed.
User avatar
admin
Posts: 4551
Joined: Thu Sep 09, 2010 3:08 pm

Re: Feature Requests

Post by admin »

Thanks for raising that point. At first, after you had written "I don't really understand...", I was reminded of a Dirac anecdote, wherein a student raised his hand and said "I don't understand that last equation." Dirac just went on and finally another student said you've ignored the question. Dirac replied that it was a statement and not a question. But then you went on to ask real questions, so I'll have to answer.

It's all driven by the samples I have in hand. And development is not always straightforward and logical. One learns and realizes things as one goes along. And sometimes things are developed step-by-step with testing at each step (incremental development). And sometimes things go in fits and starts. Finally, I am not a fully understandable person. How boring that would be!

I have the generalized handling working locally. It will be in slipstream 127.
DAE avatar
Guest

Re: Feature Requests

Post by Guest »

I believe that boring you are not!
User avatar
admin
Posts: 4551
Joined: Thu Sep 09, 2010 3:08 pm

Re: Feature Requests

Post by admin »

Thanks! What do you think of my new rank image? --------------------------------------------->

Image
DAE avatar
Guest

Re: Feature Requests

Post by Guest »

Well, he does look boring
I like it
User avatar
admin
Posts: 4551
Joined: Thu Sep 09, 2010 3:08 pm

Re: Feature Requests

Post by admin »

I'll keep it for a while but should find a more appropriate image for myself. Maybe "Resident Genius" or something? No false modesty here.

:hat: myself
User avatar
admin
Posts: 4551
Joined: Thu Sep 09, 2010 3:08 pm

Re: Feature Requests

Post by admin »

We should be at a pretty stable point now, but of course we have to wait for Guest 2 to weigh in. Next will be:

* ASS + attachments.
* Demuxed file naming improvement.
* Chapters.
* Maybe making my track IDs match gMKVExtractGUI/mkvextract. There are at least three Matroska ways to label the track: track ID != track number != track UID. And mine is a fourth: track ID + 1. Relevant discussion here: https://github.com/mbunkus/mkvtoolnix/issues/695

The goal is to handle anything that MakeMKV can throw at us from a Blu-Ray, as well as the typical things authors of MKV files may use.
DAE avatar
Guest

Re: Feature Requests

Post by Guest »

"Resident Genius"
Fits
or
Grand Pubah of things GPU
User avatar
admin
Posts: 4551
Joined: Thu Sep 09, 2010 3:08 pm

Re: Feature Requests

Post by admin »

Isn't it "Poobah"? Still, I like that idea.
DAE avatar
Guest

Re: Feature Requests

Post by Guest »

I stand corrected
Grand Poobah of things GPU
User avatar
admin
Posts: 4551
Joined: Thu Sep 09, 2010 3:08 pm

Re: Feature Requests

Post by admin »

Perfect.
DAE avatar
Guest 2
Posts: 903
Joined: Mon Sep 20, 2010 2:18 pm

Re: Feature Requests

Post by Guest 2 »

admin wrote:
Sat Feb 24, 2018 9:13 am
we have to wait for Guest 2 to weigh in
I am still having stream demuxed one at a time. Is it wanted behaviour until next iteration?

Streams seems perfect :bravo:
User avatar
admin
Posts: 4551
Joined: Thu Sep 09, 2010 3:08 pm

Re: Feature Requests

Post by admin »

Thank you for your testing, Guest 2.

It will remain one-by-one for now. The MKV file parsing is fast enough and the files small enough (compared to video) that everything usually finishes before the DGI file is done or shortly thereafter. For example, with your test file with ASS and fonts being demuxed (see below), it does not feel slow. Sure, it's not completely optimal but to change it would be nightmarish. Be aware that attachments, audio/subs, and DGI all run in their own parallel threads; it is only serialized within attachments and within audio/subs.

I have ASS plus attachment (fonts) demux working. I have to test a bit more and then I'll slipstream it later today or tomorrow morning. I'm going to leave attachments unmarked for demux by default (you have to choose to demux them), while the subs themselves are marked by default. Does this seem reasonable or would you prefer me to mark them by default? My thinking is that the demuxed font files are not changed by changing the project range so demuxing them once should be enough. You don't want it to happen every time you save the project.

I'm really appreciating Mike Matsnev's MKV parsing code base more and more. It handles everything and is thread-safe.
DAE avatar
Guest 2
Posts: 903
Joined: Mon Sep 20, 2010 2:18 pm

Re: Feature Requests

Post by Guest 2 »

admin wrote:
Sun Feb 25, 2018 10:59 am
Does this seem reasonable or would you prefer me to mark them by default? My thinking is that the demuxed font files are not changed by changing the project range so demuxing them once should be enough. You don't want it to happen every time you save the project.
It would be useful to have the correct fonts demuxed according to selected ass sub(s). Sometimes in anime there are subs for songs and banners, other for speech only etc.

P.S: I forgot to tell you that some mkv and expecially mka (audio only) comes with attached jpg for covers and txt with lyrics ;)
User avatar
admin
Posts: 4551
Joined: Thu Sep 09, 2010 3:08 pm

Re: Feature Requests

Post by admin »

How do I tell which subs require which fonts? Do I have to parse the ASS file?

Attached JPGs should work. Have you tested it?
DAE avatar
jpsdr
Posts: 214
Joined: Tue Sep 21, 2010 4:16 am

Re: Feature Requests

Post by jpsdr »

Guest 2 wrote:
Mon Feb 26, 2018 11:16 am
It would be useful to have the correct fonts demuxed according to selected ass sub(s).
Ouch !
Don't you think it's begining to be a little too much difficult ? Because when you mux an mkv, nothing technicaly links the fonts to the ass file (except maybe when the font are included within the ass file, didn't know until very recently, but it's possible, but even in that case, still not sure). Often when i mux mkv, i add the subtitle file, and then after in "attachments", add the font files. The fact is that nothing garanty that all the font requiered are muxed within the mkv. The ass renderer will check in the mkv, and then on the system fonts, and if not found, will use a default font.
Now, if your more and more appreciating Mike Matsnev's MKV parsing code base is able to provide you the list of the fonts the ass file use, good, otherwise, i think it will be diffcult, and need informations provided by an ass/ssa renderer library.
DAE avatar
Guest 2
Posts: 903
Joined: Mon Sep 20, 2010 2:18 pm

Re: Feature Requests

Post by Guest 2 »

admin wrote:
Mon Feb 26, 2018 11:28 am
Attached JPGs should work. Have you tested it?
I have some troubles within these days. Will come back into reality soon ;)
DAE avatar
Guest 2
Posts: 903
Joined: Mon Sep 20, 2010 2:18 pm

Re: Feature Requests

Post by Guest 2 »

admin wrote:
Sun Feb 25, 2018 10:59 am
Thank you for your testing, Guest 2.
You are always welcome.
admin wrote:
Sun Feb 25, 2018 10:59 am
It will remain one-by-one. The MKV file parsing is fast enough and the files small enough (compared to video) that everything usually finishes before the DGI file is done or shortly thereafter.
I demux from a very fast SSD to another physical very fast SSD. Usually I created dgi (with DGNVIndex) + demuxed streams (with GMKV) + encoded some DTSHD to AC3 at the same time. If you rely on HDD demuxing as a speed benchmark, I can agree with you, I have some doubts about demuxing on very fast SSDs and with very large audio tracks, such as ATMOS or DTSMA and very large videos, such as BD.

If it is not too much work for you, I would like to try a parallel demuxing build and make some speed tests.

P.S: A friend of mine just borrowed me Thor Ragnarok BD. Will make some tests.
User avatar
admin
Posts: 4551
Joined: Thu Sep 09, 2010 3:08 pm

Re: Feature Requests

Post by admin »

Guest 2 wrote:
Tue Feb 27, 2018 12:32 pm
If it is not too much work for you, I would like to try a parallel demuxing build and make some speed tests.
As I said, it's a nighmarish redesign. It's not something I can hack out in a few hours for you.
P.S: A friend of mine just borrowed me Thor Ragnarok BD. Will make some tests.
Let us know how it goes. Thanks.

Currently working on SUB/IDX demuxing.
DAE avatar
Guest 2
Posts: 903
Joined: Mon Sep 20, 2010 2:18 pm

Re: Feature Requests

Post by Guest 2 »

admin wrote:
Tue Feb 27, 2018 12:50 pm
As I said, it's a nighmarish redesign. It's not something I can hack out in a few hours for you.
I know and there is no hurry.
admin wrote:
Tue Feb 27, 2018 12:50 pm
Let us know how it goes. Thanks.
I noticed I have a MKV from The dark knight BD on my PC. It's a 30 GB file, with the following streams:
  • 27 GB VC1 video
  • 511 MB 448 kbit AC3 italian audio (I hate movie companies)
  • 1.7 GB 1500 kbit TrueHD english audio
  • 730 MB 640 kbit AC3 english audio
  • italian forced srt
  • italian hearing impaired srt
  • italian regular srt
  • english regular srt
Time to create DGI and to demux every stream (but video) with DGIndex_x64(*) 21:44 with max 6% CPU and source SDD working at 100 MB/s

Time to create DGI with DGIndex_x64 and at the same time demux every stream (but video) with GMKVExtractGUI 3:04 (GMKVExtractGUI finished at 1:10) with max 38% CPU and source SDD working at 515 MB/s

(*) bug: DGIndexNV does not show real elapsed time but only the one to create DGI

The real waste of time is that DGIndexNV reads the same file 7 times (1 for dgi+1st audio stream and 1 for every other stream, even tiny srt) for a total of 210 GB of disk traffic. I thought your choice was made from a HDD perspective but it will be even worse with a slow HDD.

You did a really great job and I deeply admire your programming skills but my common sense tells me that with these numbers nobody will use DGIndex to demux streams too.

EDIT: just to make you have sleepless nights. AviSynth can process audio too. DGDecNV (if and when parallelized demux will work) could provide audio streams... :twisted:
DAE avatar
Guest 2
Posts: 903
Joined: Mon Sep 20, 2010 2:18 pm

Re: Feature Requests

Post by Guest 2 »

Let's work hard again.

Such a good product deserves all our efforts.

admin :hat:
User avatar
admin
Posts: 4551
Joined: Thu Sep 09, 2010 3:08 pm

Re: Feature Requests

Post by admin »

Thanks for that and for your contributions, Guest 2.

I have demuxing of all MKV streams in one pass working in a proof-of-concept console app. While it is similar to mkvextract.exe I can't simply run mkvextract.exe from DGIndexNV because it doesn't have an option to specify a range of timecodes (to implement the project range). Mosu declined to consider adding a parameter to mkvextract to specify the range. That's strange to me because it seems like a useful feature more generally and wouldn't he want to encourage usage in third-party apps? And building mkvextract on Windows to add the parameter myself led me into a quagmire which finally resulted in discovering Visual Studio C++ does not support the language features required. So I had to bite the bullet and just write my own version. When it's polished I can incorporate it into DGDecNV with the generalized demux dialog.
User avatar
admin
Posts: 4551
Joined: Thu Sep 09, 2010 3:08 pm

Re: Feature Requests

Post by admin »

OK, boys and girls, I have a test version of parallel demuxing for you [I'm a fast coder ;) ]: http://rationalqm.us/misc/dgmkvextract.exe

The syntax is: dgmkvextract file.mkv
(or full path, and sorry no progress indication or other output other than errors)

This is a 64-bit console app. Don't know if it will run on Win7. If not, let me know. The app will extract all audio, subs, and attachments that exist in the file. Of course it is done in one pass through the MKV file. Elapsed time for the full Wonder Woman blu-ray is 15 seconds. gMKVExtractGUI took 20 seconds.

Support is there for SRT, SSA, ASS, PGS, and VOBSUB. For VOBSUB there are small time differences with the real mkvextract but they are insignificant and SubtitleEdit converts the produced SUB+IDX to SRT very well. I haven't tracked down the reason for the discrepancies but the following all differ slightly: VobSub extracted from the original VOBs, mkvextract'ed, and dgmkvextract'ed. If I have time I will try to get to the bottom of it.

After this is tested clean I'll incorporate it into DGIndexNV (as integrated code, not forking this EXE) with the following additional things:

* better output filenames (yes, I know the names from this test app suck)
* limit to the project range
* limit to selected tracks/attachments
* chapter support

Your test results will be appreciated. There are bound to be some glitches I imagine. Have fun!

:hat: Guest 2 for pressing for parallel operation
Post Reply