[RESOLVED] successive IDRs with same idr_pic_id

[laserfan]

...detected in file:

http://www.mediafire.com/?cnqek6h8dvp1t2c

This is a 30Mb DGSplit-ted clip from the front of a commerical Blu-ray movie, which with DGIndexNV v2039 yields the subject error when doing a Save As project. I wonder if this is indeed a defective stream, or if it might be some new, legal construct that you haven't seen before? Reason I ask, if I make an avs:

DGSource("summit.dgi", debug=true)

and step-thru the file, I never find two IDRs in a row (or even an IDR and I together, at least afaict). I found this oddity after re-encoding the movie using Avisynth/DGDecodeNV/x264 , only to discover my manually-placed I-frames (using qpfile) were wrong; they were off by a frame or two after conversion. In this short clip try the following:

1. Open it with DGIndexNV and step-forward to the first (dim but unmistakeable) appearance of "SUMMIT entertainment" in the logobox; Info window shows this as frame 347
2. Save as project/dgi and get one error
3. Open the .dgi with Vdub using the above .avs line and see SUMMIT first appear at frame 345

As an aside, I should mention DG that while the GUI DGIndexNV produced the error pop-up, using the command-line version did not throw an error anywhere that I could find in either the .log or the .dgi. I only discovered this when using DGIndexNV to prepare this post-with-sample for you to look at.

EDIT: It occurred to me (only After posting of course ) that I ought to check using DGIndexNV to see just how many of these stream errors are in the movie, and sure enough, there is apparently only the one, right up-front somewhere! So I'd guess that the one up-front problem shifted all my I-points cuz the index was wrong, and that if there were further errors, then subsequent I-frames would be off even more. Not sure how to fix this without getting audio & subtitles out-of-sync, so maybe I will just not bother to re-encode at all.

If you have no ideas for effecting a repair DG, I'll at least make a "request for alerting about this error with the CLI version" e.g. put it in the log perhaps, for a future update. Dunno how many other BDs may exist with this problem--I don't do this very much.

[admin]

The issue is discussed here:

http://neuron2.net/board/viewtopic.php?f=8&t=101

I do not plan to do anything in support of illegal streams.

[laserfan]

Yes I saw that; but given this was from a commercial BD I thought it warranted a look.

Or perhaps I should say "I thought you would want to see it" since it's probably not the only BD of its kind out there.

[laserfan]

Do I understand correctly then: you have not looked at the clip I uploaded, and aren't going to? It might be nice if you at least confirmed: yes there are x IDRs in a row with the same id. I don't have the tools/knowledge to examine these.

And you don't care that the CLI doesn't show the error in any way? Some of us only use the CLI version and never (well, rarely) the GUI.

[admin]

laserfan, I'd never leave you in the lurch if I could avoid it, thanks to all your valuable feedback and testing you've done for me over the years.

I looked at it and it does indeed have an instance of the problem.

Long ago users asked me to disable all popups when invoking via CLI, because popups interrupt batch operations. Do you have an alternative idea in mind for alerting you? Maybe popups with a timeout?

I will ask Nvidia if they can mitigate this. It has to be done in CUVID because it currently silently throws away frames.

Mail just sent to Nvidia:

"We are starting to see streams that have successive IDR pictures with the
same idr_pic_id (so far always = 0). This has been seen on commercial blurays.
It can also be created by accident when concatenating streams, if the first
ends with an IDR. As far as I can tell, CUVID silently discards the "extra"
IDRs. I realize it's an illegal condition but I am wondering if it can be
mitigated in CUVID, i.e., just accept the extra ones and bump the idr_pic_id
internally to pretend the stream did it right. Your thoughts on this would be
appreciated.

On another matter, we noticed that the new VP5 engine kicks some major butt!
Everybody wants one now but it's available only on the low end 520. Do you know
when higher end cards will be available with VP5?"

[laserfan]

I appreciate your reply--I came here to edit my earlier post, because it had occurred to me, since I will see if there's something hinky about the placement of I-frames when I look at chapter marks, that I can simply run DGIndexNV GUI and it will tell me if there's this particular problem. I wouldn't like a popup in the CLI any more than the next guy, so my solution (run the GUI if any question) is preferred by me, and I can see now from your msg to nvidia that you appear to be handcuffed if "CUVID silently discards the extra IDRs"--I was really looking for something in the log, or at the end of the process perhaps.

I don't make alot of BD backups. This one has a subtitle track outside of the 'scope picture area, which I need for my wife. Who knows, maybe the problem is a one-off, though I did check the original BD and it has the issue as well, so I don't think it's a ripping error.

In any case for this particular project I scrapped the homemade qpfile, and despite the IDR error up-front was able to reencode and remux the thing w/o any further issue. So don't expend alot of energy on this on my behalf, but thanks for looking!

[admin]

Reply from Nvidia (do they have good support or what?):

"It should be easy to detect, so I'll try to fix it asap (though not sure which driver it will make it into). As a potential workaround, if you already know the picture boundaries, insert a dummy access unit delimiter start code (00.00.01.09) between frames should force the parser to do the right thing.

All new GPUs coming out should include the new VP5 engine (iirc, at some point this fall)."

I will experiment with the suggested workaround, but worst case we just wait for the driver update.

[admin]

OK, here's the scoop.

For your stream, which is a bluray, the AUDs (m2ts requires AUDs) are already present so the stream is handled correctly despite the warning popup. For streams without AUDs, the workaround fixes them.

I will add this workaround to the next release (which I am working on now) and remove the warning.

I'll mark this resolved for now. Thank you for bringing this to my attention!

And finally, I have not forgotten your issue with stream type reporting for VC1 streams.

[laserfan]

I will add this workaround to the next release (which I am working on now) and remove the warning.

I'll mark this resolved for now. Thank you for bringing this to my attention!

And finally, I have not forgotten your issue with stream type reporting for VC1 streams.

I'm not sure what it means that "the stream is handled correctly despite the warning" but will look forward to the fix, cuz my issue with the thing was that I used your index to ID frame numbers at scene changes, but then after it was recoded it turned-out they were misplaced i.e. "off" from where I'd expected them to be (maybe I said this already but I keep thinking about how to explain what it was I was seeing and have different takes on it every time I think about it!).

I gotta tell ya, your having a response (and possible fix) for this in a matter of hours is just plain amazing. So I must congratulate you for having secured the confidence of no-doubt "exactly the right guy" within nvidia to communicate with. It might be that he's that good with all his "customers", but I will give you the credit for being able to comm with him in such a way that he understands instantly! Not an easy thing I believe!

It doesn't hurt also that you remember such obscure stuff as the VC1 issue I reported!

[laserfan]

Oops I guess you deleted yr post when I tried to reply. Sorry but here's what I'd typed

If you are experiencing an issue, please tell me how do duplicate it.

Hmmm I thought I did in Post 1--it shifts the index by 2 frames (seen with 347 becoming 345).

So when I use the frame numbers from the .dgi/avs/vdub examination and recode, these numbers now appear with different/wrong frames.

I did use, as I said, the .dgi to re-encode, but this time I did not try to assert I-frames using --qpfile. I just let x264 recode the whole thing, then afterward I looked for the I-frames it made (by itself) and used those for chapter marks. It worked. I guess though I need to look again to compare the original to the re-code and see if the framenos line-up perfectly...

EDIT: OK using vdub and debug=true on the original h264 program, the SUMMIT appearance is made on frame 345, as stated above. After feeding x264 with this .avs, then indexing again the output of the x264-conversion, the SUMMIT appearance is shown as being frame 346 in Vdub. Opening this file withDGIndexNV GUI and stepping-forward it shows the SUMMIT as 347, just as the original!

EDIT: Another odd thing, and I dunno if this is of interest or not, but this program appears to be encoded differently from other BDs I have seen: most have strictly 24 frame GOPs, and this one appears to have 20 frame GOPs. Odd, maybe it uses a different process e.g. Red Digital Cinema or some such...

[admin]

I wish you wouldn't respond to deleted posts! There's a reason I deleted it.

I wasn't clear when I said the stream should be OK. I meant that CUVID would handle it correctly. But DGDecNV indexing does not (without the workaround).

So, everything is correct when using the workaround that will be in version 2040. But one thing you may be overlooking: The frame number in the GUI is one-based, because it is a count; the frame number when serving is zero-based. So when you test version 2040, please keep that in mind.

If you don't mind, let's forget trying to analyze the behavior of 2039; we know it's broken for this stream. Please wait for 2040 and test that.

[laserfan]

I wish you wouldn't respond to deleted posts! It just muddies our waters.

Nah, I couldn't possibly be confusing you, could I?

Please just edit-out anything you want--I only posted for your eyes anyway.

But one thing you may be overlooking: The frame number in the GUI is one-based, because it is a count; the frame number when serving is zero-based. So when you test version 2040, please keep that in mind.

Although that thought crossed my mind, I didn't think there would be a difference between the two, so thanks for pointing this out! I almost never use the GUI!

I'll test this program again whenever 2040 appears, thanks for yr efforts.

[admin]

Version 2040 is released.

[laserfan]

Version 2040 is released.

And Yes this problem appears fixed! I re-indexed the entire program and using debug=true with avs/Vdub looked at all the chapter frames, and they are now identified perfectly, where before they showed as off-by-one. I also noticed:

1. Where before the original 264 appeared (using avs/Vdub) to start with IDR then P-frame, now is shown to start with two IDR frames

2. Where the framecount before was 159911 frames, now it is 159912 which is what eac3to had reported when I extracted it in the first place

Great work Donald, thanks a lot for the (incredibly speedy) fix!