Seamless branching and additional TrueHD frames

[stevew]

Unfortunately not, I have an AMD card.

[stevew]

I have found where ffprobe is missing the frames (no clue why it's missing them, but I know where).

So, if I run ffprobe.exe -show_frames -select_streams 2 d:\BDMV\STREAM\00505.mpls I get the following for the first 2 frames (unimportant fields removed):

Code: Select all

[FRAME]
key_frame=1
pts_time=2625.059000
pkt_dts_time=2625.059000
duration_time=0.000833
[/FRAME]
[FRAME]
key_frame=0
pts_time=2625.091500
pkt_dts_time=2625.091500
duration_time=0.000833
[/FRAME]

The first major frame is reported as starting at 2625.059000 and being 0.000833 in duration, so this frame finishes at 2625.059000 + 0.000833 = 2625.059833s. The next frame is reported as starting at 2625.091500. 2625.091500 - 2625.059833 = 0.031667 which indicates there is a gap of 0.031667s (or 38 frames) between the first 2 frames that ffprobe returns. This aligns with the number of frames I thought were missing but they appear to be missing from a different segment than I thought.

The is true of some of the other segments too:

Code: Select all

         frame 1 start + dur    frame 2 start    gap
00506    11.651500              11.651500        0
00502    2534.724000            2534.744833	 0.020833 (25 frames)
00505    2625.059833            2625.091500      0.031667 (38 frames)
00504    7235.011500            7235.011500      0
00503    7295.588167            7295.598167      0.1 (12 frames)

So you were correct that the missing frames appear to be from the start of the subsequent segment rather from the end of the previous segment. And maybe it's possible that ffprobe is mis-reporting the PTS start time for the first major frame of a segment, because if it was and the 'missing' minor frames actually come before the major frame's PTS then your theory from earlier would indeed be correct and would explain everything. And given we think that DGDemux is correct and it is showing the 'missing' minor frames come before the major frame (the screenshot from my 3rd post i think confirms this) I think there's a very high likelihood that your theory is correct.

[Rocky]

Too bad about your AMD. You're missing a lot.

Your analysis sounds plausible. When the disk arrives I'll be able to know for sure and will post an analysis.

[stevew]

I was also thinking, it would explain what I'm seeing with eac3to when it's provided with a single segment as input. A valid THD file must start with a major frame, so if eac3to sees some minor frames first in a segment it would have no choice but to throw them away until it received the first major frame, at which point it would be able to start the creation of the THD file. As a result, the collection of THD files would have less frames than DGDemux reports which is what I'm seeing.

And I guess when eac3to is provided with multiple segments as input maybe it processes each of them in turn and appends to its output (as opposed to binary appending them all and processing the appended stream). And maybe this explains the PES errors?

I don't know anything about the valid structure of a Blu-ray but it almost feels as though this disc is badly (or at least oddly) mastered - I don't think I've encountered another disc that's structured like this. Fingers crossed you are able to confirm when the disc arrives, and thanks you all your help

[Rocky]

Fingers crossed you are able to confirm when the disc arrives, and thank you all your help.

You're welcome. The disk arrives today.

[Rocky]

Ripping the disk...

[Rocky]

I studied the first two segments using separate DGIndexNV demuxing, which uses demux method 2 and does not discard anything.

For 00506.m2ts, there is a video duration of 2523.6878333 seconds, and an audio duration of 2523.0725 seconds. So, as postulated, this segment has more video than audio.

For 00502.m2ts, there is a video duration of 90.34025 seconds, and an audio duration of 90.315 seconds. Importantly, the first THD frame of this segment is a major.

These results confirm my theory that the cause of the discrepancy with separate demuxing is the audio being ahead of video in the second segment. Your idea that leading minors are discarded is eliminated. That leaves only the PTS values to account for the type 1 demuxers discarding audio frames, as I theorized. Therefore, I consider your original question to be answered.

I do want to look at eac3to behavior for two reasons: to confirm that the code uses the PTS difference to discard the leading audio frames before the first video frame, and 2) to explain the PES errors when using the + input syntax, which I have not yet attempted to duplicate.

This disk is unusual in that normally there is more audio than video in a segment. I don't recall ever seeing this before. We can check the bluray spec to see if it is legal, but I'd have to guess that it must be.

[stevew]

Thanks for taking a look Rocky, it's really appreciated.

It sounds like you've figured it out but I don't think I'm quite following

When you say the segments have a video an audio duration, is this what is reported by the timestamps or have you calculated these durations from the number of actual frames in the stream and the framerate? Are you able to share the number of audio and video frames that DGIndexNV reports is in each segment at all please?

I'm obviously misinterpreting what you are saying but if 00506.m2ts has a video duration of 2523.6878333 seconds and an audio duration of 2523.0725 seconds then there's 0.6153333 seconds of 'missing' audio in this segment. But when demuxing the playlist via. DGDemux there is no missing audio overall so I'm still confused as to where the frames that make up 0.6153333 seconds of audio are? Are you saying they come before the video start time in 00502.m2ts?

[Rocky]

When you say the segments have a video and audio duration, is this what is reported by the timestamps or have you calculated these durations from the number of actual frames in the stream and the framerate?

As previously explained the DG tools use method 2, i.e., they do not use the timestamps and do not discard anything. They iterate through the transport packets on the relevant PID and detect and count the frames.

Are you able to share the number of audio and video frames that DGIndexNV reports is in each segment at all please?

Sure, no problem:

00506:
60508 video frames
23659 major, 3004028 minor, 3027687 total, 2523.072500 duration

00502:
2166 video frames
847 major, 107531 minor, 108378 total, 90.315000 duration

I'm obviously misinterpreting what you are saying but if 00506.m2ts has a video duration of 2523.6878333 seconds and an audio duration of 2523.0725 seconds then there's 0.6153333 seconds of 'missing' audio in this segment.

It's in the following segment, as I have explained previously.

But when demuxing the playlist via DGDemux there is no missing audio overall so I'm still confused as to where the frames that make up 0.6153333 seconds of audio are? Are you saying they come before the video start time in 00502.m2ts?

Of course I am saying that, and have clearly explained it in previous posts. If they are not in segment 1 and they are not missing when joining segments 1 and 2, then they must be in segment 2. You can believe me when I tell you that I am not hiding them in my nest and throwing them out just to confuse you.

[stevew]

Sorry, I'm clearly being a bit slow today and apologies if I am frustrating you by not 'getting it'.

If it's not too much trouble, how many thd frames do the other 3 segments have? If you post those I will hopefully either get it, or be able to explain why I don't understand.

[Rocky]

Ha ha, I know what you're thinking. Standby...

[Rocky]

OK. I've got it. The counts you asked for are the same as those in your first post, so your observed discrepancy is not yet explained. So I dug deeper.

Tracing the transport packets in 00502.m2ts, here is the start of the first packet received on the THD PID 0x1100 (verified through DGIndexNV, M2TSParse, and a binary editor):

2e ad bc 6c 47 31 00 15 ec aa 7b 02 03 fa fe ...

Without here going into the spec for how to interpret these bytes (see link below), it is clear that the packet_unit_start_indicator (pusi) flag is 0. That means that a new THD frame is not starting here. The pusi flag does not get set to 1 until some ways into the file. The DG tools (and of course the others as well) will not start accepting data until the first pusi = 1 packet is seen. So all this data is lost when processing the segments separately.

A single THD frame is made up of multiple transport packets. So when this disk was authored a single THD frame's packets was split between segments. I'll have to check but that could very well be illegal. It won't matter for concatenation situations but it will for separate demuxing of the segments. It is also the likely reason for the dirty PES byte errors with eac3to as well.

This doesn't mean the method 1 tools can't also throw away data for timestamp reasons. Both loss mechanisms could be in play. I haven't looked into what the other demuxers are doing with the timestamps.

If you'd like to learn more about how to parse the data, see this thread:

viewtopic.php?f=5&t=1351

[Curly]

c'mon guys its lunch time

[Sherman]

Hey Rocky, why not add a warning print for this case to the various tools: DGDemux, DGIndexNV, and M2TSParse?

[Rocky]

Go for it, kid, I have to finish my nut caching.

[stevew]

Ah ha! Thank you so much for your very detailed investigation Rocky.

That all makes perfect sense and indeed now answers my question about why there is a difference in the frame numbers.

Many thanks for taking a look, excellent detective work

Oh, and enjoy the movie

[Rocky]

I'm half-way through it right now.

[Rocky]

It was a good flick. Entertaining. Great cast and acting. Some good plot twists. And some Stones music at the end. What's not to like?

[stevew]

Glad you enjoyed it

[Sherman]

Here's a toy for you. It will let you see these bad packets.

https://rationalqm.us/misc/M2TSParse.exe

[m2tsparse file pid_in_hex]
m2tsparse g:\rips\knives_out\bdmv\stream\00502.m2ts 1100 >out.txt

This outputs:

Code: Select all

Warning: non-pusi packet preceding first pusi packet
Warning: non-pusi packet preceding first pusi packet
Warning: non-pusi packet preceding first pusi packet
Warning: non-pusi packet preceding first pusi packet
Warning: non-pusi packet preceding first pusi packet
Warning: non-pusi packet preceding first pusi packet
Warning: non-pusi packet preceding first pusi packet
Warning: non-pusi packet preceding first pusi packet
...........
1100[0]: TrueHD 228125085 [2534723 ms]
1100[1]: TrueHD 228125160 [2534724 ms]
1100[2]: TrueHD 228125235 [2534724 ms]
1100[3]: TrueHD 228125310 [2534725 ms]
1100[4]: TrueHD 228125385 [2534726 ms]
1100[5]: TrueHD 228125460 [2534727 ms]
1100[6]: TrueHD 228125535 [2534728 ms]
...

Using this tool you can show that 00502, 00505, 00504, and 00503 all have this bad start. 00506 is clean at the start.

[stevew]

That's absolutely great, thank you Sherman.

And thanks again to Rocky for the investigation and assistance.

[Rocky]

You are most welcome, stevew. Do come again!

[hubblec4]

Hi everyone

Today I had some time for cE and wanted to post something and then I see this new thread, and once again Detective Sherlock-Rocky-Holmes is great, as is Doctor-Sherman-Watson.

Great job!

[Sherman]

Thank you, Mr hubblec4. Love your Holmes/Watson analogy.