[RESOLVED] Repeat frames at end of encode

[BarryW]

Re: Repeat frames encoded at end of encode

I've recently moved from DGAVCIndexNV 1.0.13 to DGIndexNV 2038 due to an update with MeGUI causing it to lock up on the preview screen with an AVS created with the older tool. The reason I haven't moved till now is that every version of DGIndexNV tools I've tried (2019 and 2024) have had repeat frames at the end of an encode.

Here's an example (the last minute of a recent BBC-HD programme):
sample.ts (65.36MB)
http://www.mediafire.com/?pp75hxgp0qp8w5x
It's in the usual 1440x1088 cropped to 1440x1080 then expanded to 1920x1080 format that the BBC HD uses.

Here's the AVS file I used to encode it with x264 r1913 2-pass "x264: DXVA-High Def" profile 2900Kbps max GOP 125:
sample.avs (160B)
http://www.mediafire.com/?eeiojmxyymvqd4y

Here's the DGI file created by DGIndexNV 2038 from the TS file:
sample.dgi (26.37KB)
http://www.mediafire.com/?g2mssxddo6n8eyn

Here's the encoded MKV using MeGUI 1989 (svn) with x264 r1913 2-pass "x264: DXVA-High Def" profile @2900Kbps.
sample.mkv
http://www.mediafire.com/?wsr53608uq8f3ea
You'll see 1 repeat frame at the end.

Here's the last 8 seconds of the 50 minute encode that used DGIndexNV 2038 and CUVID Server 2019.
sample2.mkv (3.37MB)
http://www.mediafire.com/?etij9fgfn1y4oeu
You'll see 2 repeat frames at the end if you use the 'Step' function of MPC-HC.

When previewing the AVS file using AvsP you won't see a problem at the end of the file.
When using DGAVCIndexNV 1.0.13 and CUVID Server 1.0.4 you won't get the repeat frames - everything works ok.

Tools used:
NVidia 8600 GT with driver 258.96
DGIndexNV: 2038
CUVID Server: 2019
AvsP: 2.0.2
MeGUI: 1989 (svn)
x264: r1913
MPC-HC: 1.5.1.2903 - all internal filters disabled
ffdshow: rev3760
Haali: 1.11.96.14
OS: Windows XP SP3
DirectX: 9.0

The BBC made a change to their HD encoder recently (22nd Feb 2011).
You need the latest version of Haali to view the TS file with MPC-HC else use VLC 1.1.7

[admin]

Here's the last 8 seconds of the 50 minute encode that used DGIndexNV 2038 and CUVID Server 2019.

The CUVID Server is obsolete and MUST NOT be used with DGDecNV 2038.

I do not see any problem with your stream serving the AVS script I made from it into VirtualDub.

[BarryW]

Thanks -that sorted it out.
I'll try reading the changes log in future.

[BarryW]

I'm getting a repeat frame again on another encode that uses DGIndexNV 2038. There was no CUVID Server running. The problem is that it only occurs at the end of a 1 hour 4.5Gbyte TS HD file that takes 5 hours to encode. AvsP shows no problem. I split off the last 30 seconds of the TS file to try and replicate the problem with a smaller TS file but this doesn't show the problem. I'll try again tonight with a larger sample. In the meantime I've gone back to the old DGAVCIndexNV 1.0.13 tool.

[BarryW]

I can't replicate the problem with a small TS file. I've tried encoding the last 1.5 minutes but no problem appears at the end. The problem only appears in the 1 hour encode.

[admin]

The only way I can help is to have a stream that produces the issue. If you are motivated you will find a way to give it to me, even if you have to mail me a physical disk. Of course, you can also trim the last few frames of the stream in your script as a workaround.

[BarryW]

I'll post you a couple of DVD's with the original TS file and my MKV encode.

[BarryW]

While your waiting for the file here's the last 7 seconds of a 1 hour encode.

sample-with-frame-repeat-at-end.mkv (268KB)
http://www.mediafire.com/?a92kb5ja9kca24k

The x264 maximum GOP was set at 100 (or 4 seconds). The repeat frame appears to be from the last frame of the previous GOP at 4 seconds back from the end of the file.

[admin]

I'll post you a couple of DVD's with the original TS file and my MKV encode.

I don't really need to see any encodes. I need the source file.

When I have seen this symptom before, it has always been due to cutting that left just the start of a frame at the end of the file, i.e., an incomplete coded frame. On the other hand, if it does not occur when serving into VirtualDub, we have to consider what you are doing with your encoding and muxing, as that can be at fault. So I need the source file and a description of your process that produces the MKV.

[BarryW]

TS Packet Editor leaves the last 'I' frame in the edit - this is the normal behaviour for TS Packet Editor but it can be disabled.
I thought the final 'I' frame would be needed by any 'B' frames in the final GOP.
I'll try encoding it without the last 'I' frame and see what happens.

The file is cut in a single piece. I encode the complete thing including the adverts.
After encoding I chop the MKV up using Global Split in mkvmerge GUI and then Append the non-advert bits back together again.
The x264 will normally create an 'I' frame at a scene change so editing using this method normally works.
Just in case it doesn't I lower the maximum GOP to 100 (4 sec at 25fps) so that I'll find an 'I' frame somewhere within 4 seconds worst case of the edit point.
This method also guarantees the audio stays in sync across the edit points. The audio delay in this case is about +50ms. For BBC HD it's normally about -1050ms.

The TS is captured with TSReader from ITV1 HD.
It's 1440x1080 (displayed as 1920x1080) 9.5Mbps with 256Kbps 2 channel AC3.
The file size is just under 4.7Gbyte so it'll just fit onto a standard DVD.
Where do I post the disk ?

[admin]

The file is cut in a single piece. I encode the complete thing including the adverts.

Does the repeated frame occur at this point (i.e., in the complete encode) or only later in the process?

[BarryW]

I'll setup a torrent so you can download the file.
It shouldn't take long to download once it's on the seedbox assuming you have a fast connection.
The upload time to the seedbox will be about 9 or 10 hours.

[admin]

I don't do torrent.

I want you to answer my question in my previous post. If I decide that the problem is potentially in DGDecNV then I will give you an address to mail the disk.

[BarryW]

Yes, the repeat frame(s) occur at the end of the encode of the complete file including the adverts - this is the first step after editing the file with TS Packet Editor.
Sometimes when running the TS Packet Editor Scan function on a TS file it'll detect a time code wrap-around at the end of a TS file - it switches on a time-code fix to sort the problem out but I don't remember this happening with this particular file. I've tried encoding the last 30 seconds and 1.5 minutes but can't replicate the problem with a small TS file.
I also don't see a problem at the end of the file when using the AvsP preview tool to preview AVS file before encoding.

I compared the DGA (from DGAVCIndexNV 1.0.13) against the DGI (from DGIndexNV 2038).
The frame counts tally in both files but the frame layout is sometimes different between the two decoders.

[admin]

I don't see your process clearly. Is it like this:

1. Capture TS.
2. Edit/cut TS with TS Packet editor.
3. Encode.
4. Put in MKV.
5. Edit MKV with some cutting tool.

If not please clarify. What is the purpose of the TS Packet editor operation?

Is the repeat frame present in the encoded AVC raw file before muxing into MKV?

[BarryW]

The captured TS will have at least 5 minutes of video before and after the programme so TS Packet Editor just edits out the programme complete with the adverts included. Obviously BBC is easier as there's no adverts.

The complete TS file is then encoded (with the adverts) into a MKV file.

The problem occurs at this point when viewing the end of the MKV file.
I've tried chopping out the last second but there's no 'I' frame to cut on - the repeat frame is either a 'P' or 'B' frame.
Previewing the AVS file (using AvsP) shows no problem at the end.

Assuming the MKV is ok I usually start chopping up the MKV file at this point (to remove the adverts) and create the final MKV file or if the file has a repeat frame I get the old DGAVCIndexNV 1.0.13 tool going and start again. I crop and resize in the AVS file - I don't use the DGIndexNV crop and resize options. I only use deinterlace=1 operation.

[BarryW]

I could setup an FTP server or just have a direct download from my server if that helps although some browsers have a file size download limit. The built-in error checking in torrent clients gives the most reliable solution for shifting large files. If the torrent isn't published then it performs a similar function to FTP.

[admin]

Please setup an FTP. Again, I need the TS going into the encode step.

[BarryW]

The file should be available Saturday morning (26th March 2011).

[BarryW]

I just re-encoded the short sample.ts file using DGIndexNV 2038 and the repeat frame (or frames) are no longer present in the MKV file. All I've done is re-boot the PC. If you remember I was originally starting the CUVID Server 2019. After you said it wasn't needed I closed it down and re-encoded without it - I didn't realise it wasn't necessary. By re-booting the PC the problem with the sample.ts file has mysteriously disappeared. I haven't re-encoded the big file yet. It'll be ready for download about 2am GMT 26th March 2011.

[admin]

OK, thanks, I'll be away all weekend at a water polo tournament, so I won't be able to do anything until Monday.

[BarryW]

The TS file is in this folder for direct download (time to download at least 2hours):
https://violet.uk.to/uploads/739jDH3EFNe/

You need to temporarily accept the invalid security certificate to get to the folder.
Here's the details:
Md5sum: 40927468B3B2CFB855F1D8B3B3F604CE

TS Packet Editor Scan log:

PID Tree:
PAT PID: 0x0000 Ver: 31 TS ID: 0x0801
0x0120: PMT Ver: 4 Program : 1
0x093A: H.264/AVC Video (as defined in ITU-T Rec. H.264 | ISO/IEC 14496-10 Video)
0x093B: AC-3 (ATSC A/53B audio)

PID List:
PID: 0000: PAT PID: 0x0000 Ver: 31 TS ID: 0x0801
PID: 0120: 0x0120: PMT Ver: 4 Program : 1
PID: 093A: H.264/AVC Video (as defined in ITU-T Rec. H.264 | ISO/IEC 14496-10 Video)
H.264 1440x1088 (1920x1080) @ 25.000 fps
PID: 093B: AC-3 (ATSC A/53B audio)
AC3: 2/0: L,R 48KHz 256Kbit/s

Filter Info:
Video Resolution from Codec: 1440 x 1080 @ 25.00 fps
Approximate System Bitrate: 10.12 Mbits/s
Approximate Duration: 00:58:51.97

Video Duration by PTS: 00:58:52.020
Filesize: 04.36GB Average bitrate 10.12 Mbps

PID: 0x093A: H.264 1440x1088 (1920x1080) @ 25.000 fps
PID: 0x093B: AC3: 2/0: L,R 48KHz 256Kbit/s
176600 Total Video Frames found
19248 Total Audio Frames found

Continuity Errors: 0, Transport Error Indicator Errors: 0
Timestamp gaps: 0, PCR gaps: 0
Total number of separate discontinuous regions: 0

End Of Log.
------
DGIndexNV 2038 log:

Stream Type: Transport [188]
Video Type: AVC
Profile: Main
Level: 4
Coded Size: 1440x1088
SAR: 4:3
Display Size: 1920x1080
Frame Rate: 25.000000 fps
Colorimetry: BT.709 [1]
Frame Structure: Fields (TFF)
Frame Type: I
Coded Number: 88300
Playback Number: 88300
Frame Repeats: 0
Field Repeats: 0
Bitrate:
Bitrate (Avg):
Bitrate (Max):
Audio Stream: 93b: AC3 2/0 48 256
Elapsed: 0:01:12
Remain: 0:00:00
FPS:
Info: Finished!
------
After re-booting the PC I'm re-encoding it again to see what happens.
Let me know when you've downloaded it so I can delete it.

[BarryW]

I've just finished re-encoding the TS file again with the same settings as before and there are no repeat frames at the end. I can only assume that after closing down CUVID Server 2019 it left something behind in memory causing future encodings to have the odd frame or two added at the end. Since re-booting the PC the problem has mysteriously disappeared.

You can still download the TS and have a look at it but I don't think you'll find anything.
In the TS Packet Editor log there are 2 durations given which might cause confusion:

Approximate Duration: 00:58:51.97
Video Duration by PTS: 00:58:52.020

[BarryW]

I tested the download with Chrome 10 and the md5 was the same.
Download time was about 2 hours.

[BarryW]

Just encoded another 1 hour HD file using DGIndexNV 2038 and there was no repeat frame(s) at the end.
It looks like the problem is solved.