[RESOLVED] HEVC/H.265 10/12-bit Support

[admin]

Hey MeteorRain, can you give me a sample like you posted details for so I can test it? Thanks.

[MeteorRain]

[link deleted]

It's always my pleasure to contribute to the development. You have done a great job and I highly appreciate it.

[admin]

Got your stream. Thank you!

Will test and report after my coffee, eggs, and bacon.

[admin]

Works fine. Will slipstream later today.

[MeteorRain]

Works fine. Will slipstream later today.

[admin]

I have slipstreamed it. Please re-download and update DGIndexNV, DGDecodeNV, and the DGIndexNV manual. Note that the DGI file version is bumped to 16 and backwards compatibility is lost, so you will have to re-make your DGI files (you can be smart and edit the file instead: change NV15 to NV16 and add the DEPTH line). It is too messy and error-prone (as well as really increasing my testing burden) to try to keep backward compatibility.

Please test it out as thoroughly as possible with all file types and when you give an OK I will announce it in the binaries notification thread. Thank you.

I saw your change to the title and your nice picture above. You are a funny bunny.

[MeteorRain]

The reason I was asking for backward compatibility was because, usually when fansubbing it takes a long time from top to bottom, and by the time we finish the subs and are about to encode, the dgi files are unusable. Worse in my case -- I use rake (a build system like make) to "build" videos so if I make changes to dgi files, it will cause all dependencies to be rebuilt.

I wonder if you can always assume DEPTH 8 if it's missing? Since you can't index Main10 in previous versions anyway.

Still I got a workaround though, as I can use the avs function naming prefix to specify a version every time I migrate to the new version ( dg4k_dgsource for 2052 and dgsource for 2051, etc. ) but it's always good to have the backward compatibility.

I'll do a test run on a 81G ts file and see how it works.

[admin]

Sorry, I'm just not going to do anything for backward DGI compatibility. You can write a small program that edits your existing DGI files if it is that important to you. All you need to do is change NV15 to NV16 in the first line and adds "DEPTH 8" after the AUDIO line.

That sample you gave has a lot of PPS's at the start of the DGI file. I hope too many PPS's won't be an issue for your 81GB file.

[MeteorRain]

That's fine, no worries. Lots of workaround available for me.

Just notice that my far out-dated nvidia driver doesn't like it (expected as it was a year back). Will give it an upgrade when I reach home. Probably gonna report back tomorrow.

I mean, good job sir. That's so sweet

[admin]

OK thanks, waiting for your results...

[MeteorRain]

Tears come out.

Code: Select all

avsmeter -timelimit=10 test.avs

AVSMeter 2.2.8 (x86)
AviSynth+ 0.1 (r1576, x86) (2.6.0.5)

Number of frames:                34800
Length (hh:mm:ss.ms):     00:09:40.720
Frame width:                      3840
Frame height:                     2160
Framerate:                      59.926 (210938/3520)
Colorspace:                       YV12

Frames processed:               886 (0 - 885)
FPS (min | max | average):      8.366 | 117.0 | 88.55
Memory usage (phys | virt):     1210 | 1337 MiB
Thread count:                   7
CPU usage (average):            12%

Time (elapsed):                 00:00:10.006

[MeteorRain]

Yea we do have a problem here. If you checked the clip I sent to you, it should say 53 seconds. Indexed and load it to DGNV, it says 21 seconds.

Also random access result is different than sequential access, so I'm guessing the index was not correctly done?

(To reproduce, load the one line AVS into VDM, sequentially moving forward for about 500 frames, press F5 to reload, should demonstrate the issue.)

[admin]

Yes, the index file is not correct. Don't panic, I will sort it out. This is actually the first non-trivial HEVC file I've encountered, and HEVC testing until now has been minimal.

[admin]

OK, it turned out to be not so hard. I had never seen TSA_N/TSA_R trailing pictures before, so adding them fixed things up nicely. I now get all the frames and reload in Vdub works as expected.

I need to fix something else first, though, before I give it to you. I've been assuming streams begin properly with SPS, PPS, and IDR/CRA/BLA but this transport stream is cut randomly so it is not the case here. I have to add code to ignore leading NALUs until a proper starting sequence is encountered. It might be a day or two as I am binge watching Breaking Bad and want to finish the last season today.

Thank you for your stream and testing.

[admin]

Please re-download 2052, update DGIndexNV, and re-index. All the frames should now be obtained. Test first with the sample and then the 81G file.

Please note the following:

1. The DGI file claims 3181 coded frames but you get only 3177 in VirtualDub. The reason is that you have 3 non-decodable frames at the start and 1 at the end due to uncontrolled cutting of the transport stream.

2. You may have to manually fix audio sync because I have not yet done the code for that for HEVC in the case of leading non-decodable frames.

[MeteorRain]

Tried through remote desktop, and all looks good.

Although seems to be a bit slower than expected. Will report when I get home and test again.

[admin]

Sweet. I'd like to know the result for your big 81GB file too when you have it done. I'm worried about all those PPS's.

I have to copy back frames to the CPU to deliver to Avisynth so it will be slower than a DirectShow player app that doesn't have to do that. Copying UHD frames over the bus is a pretty heavy burden. If you are resizing down in your script doing it on the GPU will save much of that overhead. Also, are you sure your card has full HW decoding and not HW/SW? I also convert NV12->YV12 on the CPU. Perhaps I can do that on the GPU with some performance gain.

I'll revisit my copy and NV12->YV12 code to make sure it is as optimal as possible. That wasn't so important until UHD came along.

[MeteorRain]

has full HW decoding

-- I think so? It's a 950, the slowest full hw decoding card.

[MeteorRain]

Indexing took me 11m30s.

1SPS, 757923 PPSes.

AVSMeter measured decoding speed at 26fps.

FPS 210938 / 3520
This is not that right though. Should say 60000/1001.

I was told that the duration is 05:13:09.209, but MediaInfo said it's 05:16:06.864.

I'll generate a L-SMASH works index and compare it to the dgi index to see what's happening. (Or maybe you are interested in a copy of both too?)

[MeteorRain]

Code: Select all

1042531:CRA 79708227282 2 1042482
1042532:TRAIL 0 1042483
1042533:TRAIL 0 1042484
1042534:TRAIL 1 1042485
1042535:TRAIL 0 1042486
1042536:TRAIL 0 1042487
1042537:TRAIL 1 1042488
1042538:TRAIL 0 1042489
1042539:TRAIL 0 1042490
1042540:TRAIL 1 1042491
1042541:TRAIL 0 1042492
1042542:TRAIL 0 1042493
1042543:TRAIL 1 1042494
1042544:TRAIL 0 1042495
1042545:TRAIL 0 1042496
1042546:TRAIL 1 1042497
1042547:TRAIL 0 1042498
1042548:TRAIL 0 1042499
1042549:TRAIL 1 1042500
1042550:TRAIL 0 1042501
1042551:TRAIL 0 1042502
1042552:TRAIL 1 1042503
1042553:TRAIL 0 1042504
1042554:TRAIL 0 1042505
1042555:TRAIL 1 1042506
1042556:TRAIL 0 1042507
1042557:TRAIL 0 1042508
1042558:TRAIL 1 1042509
1042559:RASL 0 1042510
1042560:RASL 0 1042511
1042561:CRA 79710566190 2 1042512
1042562:TRAIL 0 1042513
1042563:TRAIL 0 1042514
1042564:TRAIL 1 1042515
1042565:TRAIL 0 1053439
1042566:TRAIL 0 1053440
1042567:TRAIL 1 1053441
1042568:TRAIL 0 1053442
1042569:TRAIL 0 1053443
1042570:TRAIL 1 1053444
1042571:TRAIL 0 1053445
1042572:TRAIL 0 1053446
1042573:TRAIL 1 1053447
1042574:TRAIL 0 1053448
1042575:TRAIL 0 1053449
1042576:TRAIL 1 1053450
1042577:TRAIL 0 1053451
1042578:TRAIL 0 1053452
1042579:TRAIL 1 1053453
1042580:TRAIL 0 1053454
1042581:TRAIL 0 1053455
1042582:TRAIL 1 1053456
1042583:TRAIL 0 1053457
1042584:TRAIL 0 1053458
1042585:TRAIL 1 1053459
1042586:RASL 0 1053460
1042587:RASL 0 1053461
1042588:CRA 80557428546 2 1053462
1042589:TRAIL 0 1053463
1042590:TRAIL 0 1053464
1042591:TRAIL 1 1053465
1042592:TRAIL 0 1053466
1042593:TRAIL 0 1053467
1042594:TRAIL 1 1053468
1042595:TRAIL 0 1053469
1042596:TRAIL 0 1053470
1042597:TRAIL 1 1053471
1042598:TRAIL 0 1053472
1042599:TRAIL 0 1053473
1042600:TRAIL 1 1053474
1042601:TRAIL 0 1053475
1042602:TRAIL 0 1053476
1042603:TRAIL 1 1053477
1042604:TRAIL 0 1053478
1042605:TRAIL 0 1053479
1042606:TRAIL 1 1053480
1042607:TRAIL 0 1053481
1042608:TRAIL 0 1053482
1042609:TRAIL 1 1053483
1042610:TRAIL 0 1053484
1042611:TRAIL 0 1053485
1042612:TRAIL 1 1053486
1042613:TRAIL 0 1053487
1042614:TRAIL 0 1053488
1042615:TRAIL 1 1053489
1042616:RASL 0 1053490
1042617:RASL 0 1053491
1042618:CRA 80559780802 2 1053492
1042619:TRAIL 0 1053493
1042620:TRAIL 0 1053494
1042621:TRAIL 1 1053495
1042622:TRAIL 0 1053496
1042623:TRAIL 0 1053497
1042624:TRAIL 1 1053498
1042625:TRAIL 0 1053499
1042626:TRAIL 0 1053500
1042627:TRAIL 1 1053501
1042628:TRAIL 0 1053502
1042629:TRAIL 0 1053503
1042630:TRAIL 1 1053504
1042631:TRAIL 0 1053505
1042632:TRAIL 0 1053506
1042633:TRAIL 1 1053507
1042634:TRAIL 0 1053508
1042635:TRAIL 0 1053509
1042636:TRAIL 1 1053510
1042637:TRAIL 0 1053511
1042638:TRAIL 0 1053512
1042639:TRAIL 1 1053513
1042640:TRAIL 0 1053514
1042641:TRAIL 0 1053515
1042642:TRAIL 1 1053516
1042643:TRAIL 0 1053517
1042644:TRAIL 0 1053518
1042645:TRAIL 1 1053519
1042646:RASL 0 1053520
1042647:RASL 0 1053521

Skipped about 800M of data here. However if I cut that missing part (with margin, of course) into another file, then dgnv correctly recognize it as a 4:26.297 clip.

https://drive.google.com/file/d/0B9rw02 ... sp=sharing
https://drive.google.com/file/d/0B9rw02 ... sp=sharing

[admin]

FPS just reports what is in the stream. If the ratio is 59.9x, that is all that matters.

MediaInfo relies upon TS timestamps. DGDecNV counts actual frames. If there is a discontinuity they may differ.

What evidence do you have that DGDecNV is losing 800MB? Are there some frames missing during the playback? Can you cut a portion and prove it to me?

I don't know why you gave me that last portion of a DGI file or what I am supposed to do with it.

General principle: If you want me to address an issue you have to give me enough to reproduce it.

I don't know anything about LWI indexes and I don't care to learn.

[MeteorRain]

Let me explain this.

Mediainfo shows 5:16:06.9 and LWlibavvideosource() gives me 5:16:07, so let's assume they are correct.

So I went back to the stream, side by side put dgnv and lw in 2 windows and compare the difference.

I noticed that everything is good until frame 1042586. So if you seek to 1042585 and play sequentially, everything is good. If you seek to 1042586, then that's a completely different scene -- which is wrong.

So I took a look at the dgi file, and found the part I post above. No it's not the last, it's around 1042585.
You should be able to clearly see it skipping about 800MB.

1042531:CRA 79708227282 2 1042482
1042561:CRA 79710566190 2 1042512
1042588:CRA 80557428546 2 1053462
1042618:CRA 80559780802 2 1053492

So I cut out the portion that caused the problem, re-indexed and, guess what -- all is good.

Now, please suggest what I can do.

Just FYI, I post the dgi and lwi index file in my last reply.

[MeteorRain]

Specifically, I cut from 79,336,000,000 to 80,571,539,384, and dgnv measures 4:26.297.

My best hope is to re-index it and see if it magically fixes.

Please advise me any other options.

[admin]

Thank you for your testing.

Please use DGSplit to extract from the original TS the offset range 76450458002 to 76466745570 and upload that for me. There is something fishy in the DGI in that range.

[MeteorRain]

What I did: Played the video
What I saw: Duration 5:16:07, segment in question plays without problem.

What I did: Indexed ts and load it in VDM
What I saw: Duration 5:13:09

What I did: Jump to #1042586
What I saw: Some girls talking

What I did: Jump to #1042585
What I saw: A man singing

What I did: Right arrow to #1042586
What I saw: The man still singing

What I did: F5 to reload the avs and current frame
What I saw: Some girls talking

What I did: Cut from 79,336,000,000 to 80,571,539,384, at given position before the man singing and after the girls talking
What I saw: A file of 1.15GB, 4:26 by mediainfo

What I did: Indexed ts and load it in VDM
What I saw: Duration 4:26.291

What I did: Re-indexing the whole ts
What I saw: No difference in dgi files

https://drive.google.com/file/d/0B9rw02 ... sp=sharing