[RESOLVED] 7 extra frames on 1 source

Support forum for DGDecNV
User avatar
MeteorRain
Posts: 130
Joined: Fri Jul 25, 2014 4:04 pm
Location: New Jersey

[RESOLVED] 7 extra frames on 1 source

Post by MeteorRain »

Source is 3840x2160 HEVC 10bit 60000/1001, muxed in MKV and later remuxed in MP4.

Version is 200 and 202.

Total frames about 215k.

When file is cut short, index file is correct. When file is close to original (215k frames), index file is incorrect.

Code: Select all

# Short file index
259_0_100.m4v 307188553

...
5767:CRA 5765760 2 -32
5768:TRAIL 0 -31
5769:TRAIL 0 -30
5770:TRAIL 0 -29
5771:TRAIL 0 -28
5772:TRAIL 0 -27
5773:TRAIL 0 -26
5774:TRAIL 0 -25
5775:TRAIL 0 -24
5776:TRAIL 0 -23
5777:TRAIL 0 -22
5778:TRAIL 0 -21
5779:TRAIL 0 -20
5780:TRAIL 0 -19
5781:TRAIL 0 -18
5782:TRAIL 0 -17
5783:TRAIL 0 -16
5784:TRAIL 0 -15
5785:TRAIL 0 -14
5786:TRAIL 0 -13
5787:TRAIL 0 -12
5788:TRAIL 0 -11
5789:TRAIL 0 -10
5790:TRAIL 0 -9
5791:TRAIL 0 -8
5792:RASL 0 -7
5793:RASL 0 -6
5794:RASL 0 -5
5795:RASL 0 -4
5796:RASL 0 -3
5797:RASL 0 -2
5798:RASL 0 -1
5799:CRA 5797792 2 0
5800:TRAIL 0 1
...

Code: Select all

# Longer file index
259_0_3580.m4v 10949928342

...
5767:CRA 5765760 2 -32
5768:TRAIL 0 -31
5769:TRAIL 0 -30
5770:TRAIL 0 -29
5771:TRAIL 0 -28
5772:TRAIL 0 -27
5773:TRAIL 0 -26
5774:TRAIL 0 -25
5775:TRAIL 0 -24
5776:TRAIL 0 -23
5777:TRAIL 0 -22
5778:TRAIL 0 -21
5779:TRAIL 0 -20
5780:TRAIL 0 -19
5781:TRAIL 0 -18
5782:TRAIL 0 -17
5783:TRAIL 0 -16
5784:TRAIL 0 -15
5785:TRAIL 0 -14
5786:TRAIL 0 -13
5787:TRAIL 0 -12
5788:TRAIL 0 -11
5789:TRAIL 0 -10
5790:TRAIL 0 -9
5791:TRAIL 0 -8
5792:RASL 0 -7
5793:RADL 0 -7
5794:RASL 0 -6
5795:RADL 0 -6
5796:RASL 0 -5
5797:RADL 0 -5
5798:RASL 0 -4
5799:RADL 0 -4
5800:RASL 0 -3
5801:RADL 0 -3
5802:RASL 0 -2
5803:RADL 0 -2
5804:RASL 0 -1
5805:RADL 0 -1
5806:CRA 5797792 2 0
Note the RADL between -1 and -7. 7 extra frames were inserted at that point for random access. Sequential access with zero random access is correct all the way to the end.
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: 7 extra frames on 1 source

Post by Rocky »

We have to find a way to get me the stream. Do you have a host somewhere I can download it?

Also, does it fail if you extract the ES and index that?
User avatar
MeteorRain
Posts: 130
Joined: Fri Jul 25, 2014 4:04 pm
Location: New Jersey

Re: 7 extra frames on 1 source

Post by MeteorRain »

You won't believe this.

Code: Select all

259.265 10978655729
...
5767:CRA 294906551 2 -32
5768:TRAIL 0 -31
5769:TRAIL 0 -30
5770:TRAIL 0 -29
5771:TRAIL 0 -28
5772:TRAIL 0 -27
5773:TRAIL 0 -26
5774:TRAIL 0 -25
5775:TRAIL 0 -24
5776:TRAIL 0 -23
5777:TRAIL 0 -22
5778:TRAIL 0 -21
5779:TRAIL 0 -20
5780:TRAIL 0 -19
5781:TRAIL 0 -18
5782:TRAIL 0 -17
5783:TRAIL 0 -16
5784:TRAIL 0 -15
5785:TRAIL 0 -14
5786:TRAIL 0 -13
5787:TRAIL 0 -12
5788:TRAIL 0 -11
5789:TRAIL 0 -10
5790:TRAIL 0 -9
5791:TRAIL 0 -8
5792:RASL 0 -7
5793:RADL 0 -7
5794:RADL 0 -7
5795:RASL 0 -6
5796:RADL 0 -6
5797:RADL 0 -6
5798:RASL 0 -5
5799:RADL 0 -5
5800:RADL 0 -5
5801:RASL 0 -4
5802:RADL 0 -4
5803:RADL 0 -4
5804:RASL 0 -3
5805:RADL 0 -3
5806:RADL 0 -3
5807:RASL 0 -2
5808:RADL 0 -2
5809:RADL 0 -2
5810:RASL 0 -1
5811:RADL 0 -1
5812:RADL 0 -1
5813:CRA 296559906 2 0
...

Code: Select all

259_cut.265 307214654
...
5767:CRA 294906551 2 -32
5768:TRAIL 0 -31
5769:TRAIL 0 -30
5770:TRAIL 0 -29
5771:TRAIL 0 -28
5772:TRAIL 0 -27
5773:TRAIL 0 -26
5774:TRAIL 0 -25
5775:TRAIL 0 -24
5776:TRAIL 0 -23
5777:TRAIL 0 -22
5778:TRAIL 0 -21
5779:TRAIL 0 -20
5780:TRAIL 0 -19
5781:TRAIL 0 -18
5782:TRAIL 0 -17
5783:TRAIL 0 -16
5784:TRAIL 0 -15
5785:TRAIL 0 -14
5786:TRAIL 0 -13
5787:TRAIL 0 -12
5788:TRAIL 0 -11
5789:TRAIL 0 -10
5790:TRAIL 0 -9
5791:TRAIL 0 -8
5792:RASL 0 -7
5793:RASL 0 -6
5794:RASL 0 -5
5795:RASL 0 -4
5796:RASL 0 -3
5797:RASL 0 -2
5798:RASL 0 -1
5799:CRA 296559906 2 0
...

Code: Select all

dd if=259.265 of=259_cut.265 bs=307214654 count=1
Should I shoot the original file to your FTP?
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: 7 extra frames on 1 source

Post by Rocky »

Yes please upload it.

I don't know what you are saying that I won't believe. Sorry a bit slow this early in the morning.
User avatar
MeteorRain
Posts: 130
Joined: Fri Jul 25, 2014 4:04 pm
Location: New Jersey

Re: 7 extra frames on 1 source

Post by MeteorRain »

Uploaded. Encryption password is the same.

I extracted ES using

Code: Select all

mkvextract tracks 20191108.mkv 0:259.265
Rocky wrote:
Sat Feb 01, 2020 4:18 am
I don't know what you are saying that I won't believe.
It's the phenomenon that indexing a lengthy file with bit identical content (till the cut part) results a different index.
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: 7 extra frames on 1 source

Post by Rocky »

Yes, it is hard to believe. Shocking!

You said the password is the same...same as what?
User avatar
MeteorRain
Posts: 130
Joined: Fri Jul 25, 2014 4:04 pm
Location: New Jersey

Re: 7 extra frames on 1 source

Post by MeteorRain »

Same as my last sample, which is

Code: Select all

msg7086
User avatar
MeteorRain
Posts: 130
Joined: Fri Jul 25, 2014 4:04 pm
Location: New Jersey

Re: 7 extra frames on 1 source

Post by MeteorRain »

A very quick guess from me.

The source is cut from a TV stream, so the very first IDR is actually close to the end of the program. If I truncate the file to remove IDR and following stream, the file is indexed correctly. So it's a few RADLs that were supposed to be positioned around the IDR that were moved to the incorrect places.

I'll do a quick POC re-organizing the RADLs to their correct positions to see if that solves the problem.
User avatar
MeteorRain
Posts: 130
Joined: Fri Jul 25, 2014 4:04 pm
Location: New Jersey

Re: 7 extra frames on 1 source

Post by MeteorRain »

Removing leading RASLs and re-organizing RADLs to their corresponding IDRs partially solves the problem.

Still seeing seeking inaccuracy around the IDR. Might be a different issue.
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: 7 extra frames on 1 source

Post by Rocky »

Thank you for the stream and your analysis. I am currently getting only 100 KB/s download speed from my host. So it will unfortunately be a few days before I can even look at it. Why do I pay for 100 MB/s internet connection? :evil:
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: 7 extra frames on 1 source

Post by Rocky »

Got the stream, thank you. I know what's going on but I'm in the middle of a kitchen tile project (hard to stop in midstream). Ever start a project and wish you hadn't? :?

The index entries are sorted by POC per GOP (to get display order in the DGI file), but it seems that it is not implementing the GOP boundaries correctly in this case, so the negative POC values in later GOPs migrate to the first GOP.

Notice anything different about me? 8-)
User avatar
MeteorRain
Posts: 130
Joined: Fri Jul 25, 2014 4:04 pm
Location: New Jersey

Re: 7 extra frames on 1 source

Post by MeteorRain »

:scratch: You got a new house?
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: 7 extra frames on 1 source

Post by Rocky »

Actually, have an older one and the kitchen was from the early 70s. You can imagine. Homely country look with goofy masonite wainscoting and rose-adorned wallpaper -- projectile vomit -- people used to like this stuff?! So I gutted it to the bare walls and plywood subfloor. Thomasville cabinets (functional not foo-foo) (Made in USA) and quartz countertop are in. Backsplash subway tiling in progress. Saving thousands doing it myself but didn't plan on massive hand cramps. Final result is gonna be sweet!

I'll try to find time to look at your issue tomorrow.

SPOILER: My avatar now faces the other way (toward the post instead of away). It feels more welcoming and inclusive. :P
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: 7 extra frames on 1 source

Post by Rocky »

Found and fixed the bug! Slipstream coming later today. Gotta lay some tile first. :(
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: 7 extra frames on 1 source

Post by Rocky »

Should be working in slipstream 203. Please advise if OK and I can mark this resolved.

Real good find there, MeteorRain. Thank you. :salute:
User avatar
MeteorRain
Posts: 130
Joined: Fri Jul 25, 2014 4:04 pm
Location: New Jersey

Re: 7 extra frames on 1 source

Post by MeteorRain »

Wow that's a lot of work. Didn't notice your avatar change though (looks pretty much the same so...)

Good to know the bug is fixed. I'm gonna test it against my other samples to make sure it's working perfect. (Hint: there was some seeking accuracy problem I noticed, so I'll double check.) Will let you know!

Thanks!
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: 7 extra frames on 1 source

Post by Rocky »

The bug would definitely cause seeking issues, so hopefully those should be fixed too.
User avatar
MeteorRain
Posts: 130
Joined: Fri Jul 25, 2014 4:04 pm
Location: New Jersey

Re: 7 extra frames on 1 source

Post by MeteorRain »

Definitely negative POC shifting problem is solved.

Seeking inaccuracy still presents. I have uploaded a minimized sample to your FTP "259test4.hevc", please check frame 24, sequential and random access.
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: 7 extra frames on 1 source

Post by Rocky »

OK, will do. I saw that file up there and wondered if it was you.
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: 7 extra frames on 1 source

Post by Rocky »

It's working fine for me. What are you seeing?
User avatar
MeteorRain
Posts: 130
Joined: Fri Jul 25, 2014 4:04 pm
Location: New Jersey

Re: 7 extra frames on 1 source

Post by MeteorRain »

Random access to 23 and move forward to 24 I'm seeing the end of the program.

Random access to 24 and I'm seeing the next program.

(I recall there's a tool that can play avs forward and backward and check seeking accuracy but I don't remember where it is.)

Using VirtualDub2 64-bit, AviSynth+, DG 2053SS203.
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: 7 extra frames on 1 source

Post by Rocky »

OK, I see it now. Investigating...
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: 7 extra frames on 1 source

Post by Rocky »

Hmm, it's those pesky negative POCs again. I have a fix working locally but have to do a lot of functional and regression testing.
User avatar
MeteorRain
Posts: 130
Joined: Fri Jul 25, 2014 4:04 pm
Location: New Jersey

Re: 7 extra frames on 1 source

Post by MeteorRain »

Any reason keeping them as-is? Like, what if we just shift it to above zero? Just my 0.02 guess though.
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Re: 7 extra frames on 1 source

Post by Rocky »

It's not the negativity; it's the fact that the leading RADLs are B frames, and leading B frame code hasn't been implemented for HEVC hithertoo. This is actually the first HEVC stream seen here that has leading B frames. I knew the day must come, so now it's time to take care of business.
Post Reply