Generate QP file to force frame types

Support forum for DGDecNV
DAE avatar
Boulder
Posts: 113
Joined: Fri Jul 29, 2011 7:22 am

Generate QP file to force frame types

Post by Boulder »

One more request: functionality to create a qpfile based on chapter timestamps during indexing. I've sometimes noticed that x264 and x265 miss some obvious scene changes and may not place a keyframe where the chapter begins (or if the chapter begins without a scene change, the issue is even more probable).

https://x265.readthedocs.io/en/master/c ... ion-qpfile for specification, most likely frametype K would do here.
DAE avatar
Guest

Feature Requests

Post by Guest »

One more request: functionality to create a qpfile based on chapter timestamps during indexing
:agree:
+1
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Feature Requests

Post by Rocky »

"Specify a text file which contains frametypes and QPs for some or all frames."

What is a QP and how is it obtained?
DAE avatar
Boulder
Posts: 113
Joined: Fri Jul 29, 2011 7:22 am

Feature Requests

Post by Boulder »

Rocky wrote:
Sun Aug 01, 2021 8:20 am
"Specify a text file which contains frametypes and QPs for some or all frames."

What is a QP and how is it obtained?
QP is not needed in this case, it can be left out. QP is basically the frame quantizer, it's used if you want to force a fixed quantizer on some frame instead of the one the encoder decides to use.
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Feature Requests

Post by Rocky »

Well then, what is it that you are asking for exactly? You want a file with the frame number and frame type for all frames that start chapters?
User avatar
DJATOM
Posts: 176
Joined: Fri Oct 16, 2015 6:14 pm

Feature Requests

Post by DJATOM »

QP file usually used to override encoder's frame type decisions. It's fine to declare IDR on chapter start, so player will navigate faster. But honestly I believe that's not that urgent and can be achieved by some python script:

Code: Select all

import re
with open(r'chapter_file.txt') as f:
    lines = f.read()
    bmDict = []
    tList = re.findall(r'(\d+)=(\d+):(\d+):(\d+\.\d+)', lines)
    fps = 24000 / 1001
    for index, hr, min, sec in tList:
        sec = int(hr)*3600 + int(min)*60 + float(sec)
        bmDict.append(str(round(sec*fps)))
    with open(f'qp_file.qp', 'w') as q:
        for n in bmDict:
            q.write(f'{n} I -1\n')
DAE avatar
Boulder
Posts: 113
Joined: Fri Jul 29, 2011 7:22 am

Feature Requests

Post by Boulder »

Rocky wrote:
Sun Aug 01, 2021 10:01 am
Well then, what is it that you are asking for exactly? You want a file with the frame number and frame type for all frames that start chapters?
Yes, exactly that - to ensure the playback starts from the correct place when jumping to a chapter. DJATOM's Python script is definitely nice as well
:bravo:
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Feature Requests

Post by Rocky »

OK, thank you. It'll be a while as the MP4 upgrade is quite involved and we want to get that done first.
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Feature Requests

Post by Rocky »

DJATOM wrote:
Sun Aug 01, 2021 10:51 am
QP file usually used to override encoder's frame type decisions. It's fine to declare IDR on chapter start, so player will navigate faster. But honestly I believe that's not that urgent and can be achieved by some python script:
Very interesting, thank you DJ. I think we can just convert that to C++ and generate the qp file when the chapters file is generated. I'll try it out.
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Feature Requests

Post by Rocky »

I have the qpfile stuff working and will give slipstreams for DGIndexNV and DGDemux later today.
DAE avatar
Boulder
Posts: 113
Joined: Fri Jul 29, 2011 7:22 am

Feature Requests

Post by Boulder »

Rocky wrote:
Mon Aug 09, 2021 6:59 am
I have the qpfile stuff working but I have two questions for you:

1. I create *.chapters.txt and *.qp.txt. Should I delete the .txt and just create *.chapters and *.qp?

2. Should I do (for example)

456 I -1

or, because the x265 doc says the QP field is optional, just

456 I

The x265 docs suggest the latter. Does it matter? Examples I have seen online use the -1.
1. I'd keep chapters as .txt but for QP, .qp would be a good extension IMO.
2. The latter at least works properly so it's the safest bet :D
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Feature Requests

Post by Rocky »

Thank you. I had already concluded to use -1 before your post, hence the edit. Regarding the extensions, I'll use .txt for both because double click opens my text editor. ;)
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Feature Requests

Post by Rocky »

I released the QP stuff as DGDecNV slipstream 231. DGDemux to follow.
DAE avatar
Boulder
Posts: 113
Joined: Fri Jul 29, 2011 7:22 am

Feature Requests

Post by Boulder »

Rocky wrote:
Mon Aug 09, 2021 11:24 am
Thank you. I had already concluded to use -1 before your post, hence the edit. Regarding the extensions, I'll use .txt for both because double click opens my text editor. ;)
Hey, no worries regarding the extension and x265 will override that quantizer value anyway since it's out of the default range for minimum and maximum.

Thanks a lot! :bravo:
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Feature Requests

Post by Rocky »

You are most welcome, and thank you for the suggestion.
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Feature Requests

Post by Rocky »

QP file generation added to DGDemux slipstream 52.

Recent DGDecNV (230 and 231) and DGDemux (52) changes will now be ported to the linux versions.
User avatar
DJATOM
Posts: 176
Joined: Fri Oct 16, 2015 6:14 pm

Generate QP file to force frame types

Post by DJATOM »

Nice!
DAE avatar
Boulder
Posts: 113
Joined: Fri Jul 29, 2011 7:22 am

Generate QP file to force frame types

Post by Boulder »

I tested this on Total Recall (should be this one: https://www.amazon.com/Total-Recall-4K- ... 08YNQ5CVD/) and there are a few chapter points where the frame number calculation is off by one, so maybe it's due to rounding?

Code: Select all

CHAPTER01=00:00:00.000
CHAPTER01NAME=
CHAPTER02=00:05:50.583
CHAPTER02NAME=
CHAPTER03=00:11:27.375
CHAPTER03NAME=
CHAPTER04=00:18:14.000
CHAPTER04NAME=
CHAPTER05=00:24:50.917
CHAPTER05NAME=
CHAPTER06=00:29:37.625
CHAPTER06NAME=
CHAPTER07=00:37:05.875
CHAPTER07NAME=
CHAPTER08=00:43:11.833
CHAPTER08NAME=
CHAPTER09=00:48:35.667
CHAPTER09NAME=
CHAPTER10=00:55:02.042
CHAPTER10NAME=
CHAPTER11=01:00:03.917
CHAPTER11NAME=
CHAPTER12=01:15:21.167
CHAPTER12NAME=
CHAPTER13=01:20:16.167
CHAPTER13NAME=
CHAPTER14=01:27:07.083
CHAPTER14NAME=
CHAPTER15=01:32:18.458
CHAPTER15NAME=
CHAPTER16=01:39:53.042
CHAPTER16NAME=
CHAPTER17=01:53:15.958
CHAPTER17NAME=

Code: Select all

0 I -1
8415 I -1  --- correct one is 8414
16497 I -1
26256 I -1
35782 I -1
42663 I -1
53421 I -1
62205 I -1  --- correct one is 62204, which is a B-frame in the original video  :o 
69976 I -1
79249 I -1
86494 I -1
108508 I -1
115588 I -1  --- it's the correct one but once again a B-frame in the original
125451 I -1  --- correct one is 125450
132924 I -1  --- correct one is 132923
143833 I -1
163104 I -1
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Generate QP file to force frame types

Post by Rocky »

What happens with DJATOM's script?

EDIT: OK, I think I know what is going on. But please do report results with DJATOM's script if possible.

EDIT2: I used your posted chapters file with DJATOM's program and fps 24.00. It matches the correct output you posted. So I will fix that and sneak in an update. Standby...

The fact that some are B frames is inconsequential.
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Generate QP file to force frame types

Post by Rocky »

Please re-download DGDecNV 231 and/or DGDemux 52 and test again. Should be working as expected. If not, please report.
DAE avatar
Boulder
Posts: 113
Joined: Fri Jul 29, 2011 7:22 am

Generate QP file to force frame types

Post by Boulder »

Yes, seems to be correct now. I tested a 25fps source as well and looks good. The wondering about B-frames was just general, I thought all chapters must start at an I-frame but I guess I was very much wrong there.

I also tested a Matroska file but the QP file is not produced, got the chapter file though.
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Generate QP file to force frame types

Post by Rocky »

Boulder wrote:
Wed Aug 11, 2021 11:58 pm
Yes, seems to be correct now. I tested a 25fps source as well and looks good. The wondering about B-frames was just general, I thought all chapters must start at an I-frame but I guess I was very much wrong there.
You can have an open GOP with B frames that do not point backwards.
I also tested a Matroska file but the QP file is not produced, got the chapter file though.
I forgot about MKV. I'm using mkvextract now to get chapters from MKV, so I'll make a standalone chapters2qp executable and invoke it automatically.

Thank you for your testing.
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Generate QP file to force frame types

Post by Rocky »

I added a binary chapters2qp.exe to the 231 distribution. You can use it to generate the QP file manually from a chapters file. Today I will revise DGIndexNV to invoke it automatically for MKV (use normal chapters file format, not XML; advise me if XML is important to you).

D:\Don\Programming\C++\chapters2qp\x64\Release>chapters2qp
chapters2qp 1.0 Copyright (c) 2021 Donald A. Graft, All rights reserved
Usage: chapters2qp frame_rate chapters_file
frame_rate: video frame rate (float)
chapters_file: chapters file to be used (character string holding path)
DAE avatar
Boulder
Posts: 113
Joined: Fri Jul 29, 2011 7:22 am

Generate QP file to force frame types

Post by Boulder »

Thanks, just tested it on a set of existing chapter files for my next encoding batch, works perfectly :) I can use the regular format, no need for XML support.
User avatar
Rocky
Posts: 3557
Joined: Fri Sep 06, 2019 12:57 pm

Generate QP file to force frame types

Post by Rocky »

Great. Thank you for your testing. I have the automatic invocation working and will sneak it in to 231 later today.
Post Reply