Page 1 of 2
[RESOLVED] 7 extra frames on 1 source
Posted: Fri Jan 31, 2020 7:24 pm
by MeteorRain
Source is 3840x2160 HEVC 10bit 60000/1001, muxed in MKV and later remuxed in MP4.
Version is 200 and 202.
Total frames about 215k.
When file is cut short, index file is correct. When file is close to original (215k frames), index file is incorrect.
Code: Select all
# Short file index
259_0_100.m4v 307188553
...
5767:CRA 5765760 2 -32
5768:TRAIL 0 -31
5769:TRAIL 0 -30
5770:TRAIL 0 -29
5771:TRAIL 0 -28
5772:TRAIL 0 -27
5773:TRAIL 0 -26
5774:TRAIL 0 -25
5775:TRAIL 0 -24
5776:TRAIL 0 -23
5777:TRAIL 0 -22
5778:TRAIL 0 -21
5779:TRAIL 0 -20
5780:TRAIL 0 -19
5781:TRAIL 0 -18
5782:TRAIL 0 -17
5783:TRAIL 0 -16
5784:TRAIL 0 -15
5785:TRAIL 0 -14
5786:TRAIL 0 -13
5787:TRAIL 0 -12
5788:TRAIL 0 -11
5789:TRAIL 0 -10
5790:TRAIL 0 -9
5791:TRAIL 0 -8
5792:RASL 0 -7
5793:RASL 0 -6
5794:RASL 0 -5
5795:RASL 0 -4
5796:RASL 0 -3
5797:RASL 0 -2
5798:RASL 0 -1
5799:CRA 5797792 2 0
5800:TRAIL 0 1
...
Code: Select all
# Longer file index
259_0_3580.m4v 10949928342
...
5767:CRA 5765760 2 -32
5768:TRAIL 0 -31
5769:TRAIL 0 -30
5770:TRAIL 0 -29
5771:TRAIL 0 -28
5772:TRAIL 0 -27
5773:TRAIL 0 -26
5774:TRAIL 0 -25
5775:TRAIL 0 -24
5776:TRAIL 0 -23
5777:TRAIL 0 -22
5778:TRAIL 0 -21
5779:TRAIL 0 -20
5780:TRAIL 0 -19
5781:TRAIL 0 -18
5782:TRAIL 0 -17
5783:TRAIL 0 -16
5784:TRAIL 0 -15
5785:TRAIL 0 -14
5786:TRAIL 0 -13
5787:TRAIL 0 -12
5788:TRAIL 0 -11
5789:TRAIL 0 -10
5790:TRAIL 0 -9
5791:TRAIL 0 -8
5792:RASL 0 -7
5793:RADL 0 -7
5794:RASL 0 -6
5795:RADL 0 -6
5796:RASL 0 -5
5797:RADL 0 -5
5798:RASL 0 -4
5799:RADL 0 -4
5800:RASL 0 -3
5801:RADL 0 -3
5802:RASL 0 -2
5803:RADL 0 -2
5804:RASL 0 -1
5805:RADL 0 -1
5806:CRA 5797792 2 0
Note the RADL between -1 and -7. 7 extra frames were inserted at that point for random access. Sequential access with zero random access is correct all the way to the end.
Re: 7 extra frames on 1 source
Posted: Fri Jan 31, 2020 7:53 pm
by Rocky
We have to find a way to get me the stream. Do you have a host somewhere I can download it?
Also, does it fail if you extract the ES and index that?
Re: 7 extra frames on 1 source
Posted: Fri Jan 31, 2020 8:18 pm
by MeteorRain
You won't believe this.
Code: Select all
259.265 10978655729
...
5767:CRA 294906551 2 -32
5768:TRAIL 0 -31
5769:TRAIL 0 -30
5770:TRAIL 0 -29
5771:TRAIL 0 -28
5772:TRAIL 0 -27
5773:TRAIL 0 -26
5774:TRAIL 0 -25
5775:TRAIL 0 -24
5776:TRAIL 0 -23
5777:TRAIL 0 -22
5778:TRAIL 0 -21
5779:TRAIL 0 -20
5780:TRAIL 0 -19
5781:TRAIL 0 -18
5782:TRAIL 0 -17
5783:TRAIL 0 -16
5784:TRAIL 0 -15
5785:TRAIL 0 -14
5786:TRAIL 0 -13
5787:TRAIL 0 -12
5788:TRAIL 0 -11
5789:TRAIL 0 -10
5790:TRAIL 0 -9
5791:TRAIL 0 -8
5792:RASL 0 -7
5793:RADL 0 -7
5794:RADL 0 -7
5795:RASL 0 -6
5796:RADL 0 -6
5797:RADL 0 -6
5798:RASL 0 -5
5799:RADL 0 -5
5800:RADL 0 -5
5801:RASL 0 -4
5802:RADL 0 -4
5803:RADL 0 -4
5804:RASL 0 -3
5805:RADL 0 -3
5806:RADL 0 -3
5807:RASL 0 -2
5808:RADL 0 -2
5809:RADL 0 -2
5810:RASL 0 -1
5811:RADL 0 -1
5812:RADL 0 -1
5813:CRA 296559906 2 0
...
Code: Select all
259_cut.265 307214654
...
5767:CRA 294906551 2 -32
5768:TRAIL 0 -31
5769:TRAIL 0 -30
5770:TRAIL 0 -29
5771:TRAIL 0 -28
5772:TRAIL 0 -27
5773:TRAIL 0 -26
5774:TRAIL 0 -25
5775:TRAIL 0 -24
5776:TRAIL 0 -23
5777:TRAIL 0 -22
5778:TRAIL 0 -21
5779:TRAIL 0 -20
5780:TRAIL 0 -19
5781:TRAIL 0 -18
5782:TRAIL 0 -17
5783:TRAIL 0 -16
5784:TRAIL 0 -15
5785:TRAIL 0 -14
5786:TRAIL 0 -13
5787:TRAIL 0 -12
5788:TRAIL 0 -11
5789:TRAIL 0 -10
5790:TRAIL 0 -9
5791:TRAIL 0 -8
5792:RASL 0 -7
5793:RASL 0 -6
5794:RASL 0 -5
5795:RASL 0 -4
5796:RASL 0 -3
5797:RASL 0 -2
5798:RASL 0 -1
5799:CRA 296559906 2 0
...
Code: Select all
dd if=259.265 of=259_cut.265 bs=307214654 count=1
Should I shoot the original file to your FTP?
Re: 7 extra frames on 1 source
Posted: Sat Feb 01, 2020 4:18 am
by Rocky
Yes please upload it.
I don't know what you are saying that I won't believe. Sorry a bit slow this early in the morning.
Re: 7 extra frames on 1 source
Posted: Sat Feb 01, 2020 10:10 am
by MeteorRain
Uploaded. Encryption password is the same.
I extracted ES using
Code: Select all
mkvextract tracks 20191108.mkv 0:259.265
Rocky wrote: ↑Sat Feb 01, 2020 4:18 am
I don't know what you are saying that I won't believe.
It's the phenomenon that indexing a lengthy file with bit identical content (till the cut part) results a different index.
Re: 7 extra frames on 1 source
Posted: Sat Feb 01, 2020 12:41 pm
by Rocky
Yes, it is hard to believe. Shocking!
You said the password is the same...same as what?
Re: 7 extra frames on 1 source
Posted: Sat Feb 01, 2020 1:30 pm
by MeteorRain
Same as my last sample, which is
Re: 7 extra frames on 1 source
Posted: Sat Feb 01, 2020 7:46 pm
by MeteorRain
A very quick guess from me.
The source is cut from a TV stream, so the very first IDR is actually close to the end of the program. If I truncate the file to remove IDR and following stream, the file is indexed correctly. So it's a few RADLs that were supposed to be positioned around the IDR that were moved to the incorrect places.
I'll do a quick POC re-organizing the RADLs to their correct positions to see if that solves the problem.
Re: 7 extra frames on 1 source
Posted: Sat Feb 01, 2020 9:25 pm
by MeteorRain
Removing leading RASLs and re-organizing RADLs to their corresponding IDRs partially solves the problem.
Still seeing seeking inaccuracy around the IDR. Might be a different issue.
Re: 7 extra frames on 1 source
Posted: Sun Feb 02, 2020 8:04 am
by Rocky
Thank you for the stream and your analysis. I am currently getting only 100 KB/s download speed from my host. So it will unfortunately be a few days before I can even look at it. Why do I pay for 100 MB/s internet connection?
Re: 7 extra frames on 1 source
Posted: Mon Feb 03, 2020 12:14 pm
by Rocky
Got the stream, thank you. I know what's going on but I'm in the middle of a kitchen tile project (hard to stop in midstream). Ever start a project and wish you hadn't?
The index entries are sorted by POC per GOP (to get display order in the DGI file), but it seems that it is not implementing the GOP boundaries correctly in this case, so the negative POC values in later GOPs migrate to the first GOP.
Notice anything different about me?
Re: 7 extra frames on 1 source
Posted: Mon Feb 03, 2020 6:26 pm
by MeteorRain
You got a new house?
Re: 7 extra frames on 1 source
Posted: Mon Feb 03, 2020 7:57 pm
by Rocky
Actually, have an older one and the kitchen was from the early 70s. You can imagine. Homely country look with goofy masonite wainscoting and rose-adorned wallpaper -- projectile vomit -- people used to like this stuff?! So I gutted it to the bare walls and plywood subfloor. Thomasville cabinets (functional not foo-foo) (Made in USA) and quartz countertop are in. Backsplash subway tiling in progress. Saving thousands doing it myself but didn't plan on massive hand cramps. Final result is gonna be sweet!
I'll try to find time to look at your issue tomorrow.
SPOILER: My avatar now faces the other way (toward the post instead of away). It feels more welcoming and inclusive.
Re: 7 extra frames on 1 source
Posted: Tue Feb 04, 2020 9:13 am
by Rocky
Found and fixed the bug! Slipstream coming later today. Gotta lay some tile first.
Re: 7 extra frames on 1 source
Posted: Tue Feb 04, 2020 10:38 am
by Rocky
Should be working in slipstream 203. Please advise if OK and I can mark this resolved.
Real good find there,
MeteorRain. Thank you.
Re: 7 extra frames on 1 source
Posted: Tue Feb 04, 2020 10:44 am
by MeteorRain
Wow that's a lot of work. Didn't notice your avatar change though (looks pretty much the same so...)
Good to know the bug is fixed. I'm gonna test it against my other samples to make sure it's working perfect. (Hint: there was some seeking accuracy problem I noticed, so I'll double check.) Will let you know!
Thanks!
Re: 7 extra frames on 1 source
Posted: Tue Feb 04, 2020 10:54 am
by Rocky
The bug would definitely cause seeking issues, so hopefully those should be fixed too.
Re: 7 extra frames on 1 source
Posted: Tue Feb 04, 2020 11:08 am
by MeteorRain
Definitely negative POC shifting problem is solved.
Seeking inaccuracy still presents. I have uploaded a minimized sample to your FTP "259test4.hevc", please check frame 24, sequential and random access.
Re: 7 extra frames on 1 source
Posted: Tue Feb 04, 2020 11:09 am
by Rocky
OK, will do. I saw that file up there and wondered if it was you.
Re: 7 extra frames on 1 source
Posted: Tue Feb 04, 2020 11:14 am
by Rocky
It's working fine for me. What are you seeing?
Re: 7 extra frames on 1 source
Posted: Tue Feb 04, 2020 11:22 am
by MeteorRain
Random access to 23 and move forward to 24 I'm seeing the end of the program.
Random access to 24 and I'm seeing the next program.
(I recall there's a tool that can play avs forward and backward and check seeking accuracy but I don't remember where it is.)
Using VirtualDub2 64-bit, AviSynth+, DG 2053SS203.
Re: 7 extra frames on 1 source
Posted: Tue Feb 04, 2020 11:42 am
by Rocky
OK, I see it now. Investigating...
Re: 7 extra frames on 1 source
Posted: Wed Feb 05, 2020 5:09 pm
by Rocky
Hmm, it's those pesky negative POCs again. I have a fix working locally but have to do a lot of functional and regression testing.
Re: 7 extra frames on 1 source
Posted: Wed Feb 05, 2020 11:45 pm
by MeteorRain
Any reason keeping them as-is? Like, what if we just shift it to above zero? Just my 0.02 guess though.
Re: 7 extra frames on 1 source
Posted: Thu Feb 06, 2020 2:44 pm
by Rocky
It's not the negativity; it's the fact that the leading RADLs are B frames, and leading B frame code hasn't been implemented for HEVC hithertoo. This is actually the first HEVC stream seen here that has leading B frames. I knew the day must come, so now it's time to take care of business.