[RESOLVED] Best way to decimate...

Support forum for DGDecNV
DAE avatar
magic144
Posts: 25
Joined: Sat Feb 19, 2011 2:53 pm

[RESOLVED] Best way to decimate...

Post by magic144 »

so I've bought a new Nvidia card for the sole purpose of using DGDecNV

it works great!

now there's an outstanding issue...

I'm trying to reencode a BBC Disc (Doctor Who, Season 5 Blu-Ray). This is VC-1, Advanced Profile, Level 3 material.
The file indexes OK, and the hardware deinterlacing looks great.

Only problem is, the output requires some sort of decimation, since it has repeating frames.
The output rate is 29.97fps, but the source is almost certainly 25fps (PAL from the UK).

Decimate(cycle=6) on the deinterlace=1 output from DGSource() produces 24.975fps output and can be seen (on close inspection) to have removed some unique frames on occasion.
This is because, whilst the cadence is normally 5 unique + 1 repeat, every now and again (with unknown frequency) there are 6 unique frames in row!

I have tried srestore(frate=25, speed=-1) using yadif as a bobber and not deinterlacing from DGSource(), and it is successful, but some motion shows evidence of chroma blurring or lagging that isn't evident in the straight deinterlace=1 output of DGSource().
(I don't know why, but using yadif as a bobber seems to be faster than using the bobbed output - deinterlace=2 - option from DGSource!!)

Is there a better way to do this?
Is there a better way to decimate... I want to delete 1 in *almost* every 6, but just not every time.
And can I use a method that preserves the deinterlaced output of the card (which is about as clean as the picture will get I guess).

I wonder if there's anything in the source that can be used as guidance??

Thanks in advance for any and all help!
User avatar
admin
Posts: 4551
Joined: Thu Sep 09, 2010 3:08 pm

Re: Best way to decimate...

Post by admin »

Post a link to an unprocessed source stream sample showing the effects you mentioned.
DAE avatar
magic144
Posts: 25
Joined: Sat Feb 19, 2011 2:53 pm

Re: Best way to decimate...

Post by magic144 »

I've uploaded a clip here:-
http://www.mediafire.com/?7lqca1zcddc3q

This is a shared folder that includes the .dgi and .mkv files for the source clip,
as well as a .log file created by DGIndexNV. I have also uploaded the .avs script used to examine the behaviour of Decimate() - decimate.avs.

---decimate.avs---
LoadPlugin("C:\Users\User\Desktop\dgdecnv2038\DGDecodeNV.dll")
DGSource("video (1)-034.dgi", deinterlace=1)
Decimate(cycle=6,show=true)
---

In general, you can look at (step through) the clip itself to observe the cadence.

Specifically, using the AviSynth script decimate.avs, observe what happens between output frame 509 and output frame 510.
Decimate is correctly choosing 1 frame in every 6 to discard.

Input frame set 606-611 are those under consideration at the time of choosing output frame 509, and output frame 509 corresponds with input frame 610. The decision has already been made by Decimate() to discard frame 611.

Now for the next output frame, output frame 510, the decision block moves forward to frames 612-617.
In this case, ALL of these frames are Unique, so Decimate() chooses to drop frame 612, meaning that output frame 510 is chosen from input frame 613, and we've lost a valid source frame (612).

You can see all of the source frames in question just using this source script (source.avs)

---source.avs---
LoadPlugin("C:\Users\User\Desktop\dgdecnv2038\DGDecodeNV.dll")
DGSource("video (1)-034.dgi", deinterlace=1)
info()
---

and indeed, you can see that frame 611 is a dupe, whereas frame 612 is not.
You can also see that 612-617 are all unique, whereas the more usual cadence is 5 unique + 1 dupe, 5+1, 5+1, 5+1,... etc.

So this has occurred because we have a break in the usual sequence of 5 unique + 1 dupe frames, and this has happened to have occurred at the boundary of a block (each block for consideration is 6 frames in this case).

As far as I know, all of this is the correct behaviour of Decimate().
But it is these occasional and untimely occurrences of 6 sequential, unique source frames that can lead to this problem, and hence why I need to find a better technique for discarding the genuine dupe frames and restoring 25fps.

By the way, I am using Decimate 5.2.3 in this example.
I've only concentrated on describing Decimate() here, but as I've said, I can use srestore() to obtain a valid 25fps output, it's just that a) I lose the hardware deinterlacing, and b) there is some chroma blurring that you don't see in the hardware-deinterlaced output.
I am hoping there is a smarter/better way to do the decimation??!

Thanks again for your help,
m
DAE avatar
magic144
Posts: 25
Joined: Sat Feb 19, 2011 2:53 pm

Re: Best way to decimate...

Post by magic144 »

Hmm, there's also some strange stuttering at the very start of the encoded stream, when trying to use srestore() with DGSource() here too...
Possibly due to an interaction between srestore() and DGSource()... I will have to explain this in more detail with supporting data later - it's very late on Sunday night!

Suffice it to say, if I encode a small clip, at the very start there is some temporal stuttering.
If I turn the same small source clip into an intermediate (large) Huffyuv .avi and re-encode (still using the same avs, just a different source line), the stuttering is absent.
User avatar
admin
Posts: 4551
Joined: Thu Sep 09, 2010 3:08 pm

Re: Best way to decimate...

Post by admin »

Using MultiDecimate I find the base frame rate of that clip to be around 22.5 fps. So I think you won't find a cycle=6 solution for this. What are you going to do with 22 fps video?
DAE avatar
magic144
Posts: 25
Joined: Sat Feb 19, 2011 2:53 pm

Re: Best way to decimate...

Post by magic144 »

That doesn't make sense... Most of the content is sequences of 5+1 (5 unique + 1 dupe) - by inspection
however very occasionally there is a 6+0 - by inspection

for a 29.97 source, that would lead to something slightly greater than 24.975... (29.97*5/6 + delta)
and since it is known to be from the BBC, I can only assume the source to be 25fps

if I look at the frames, I don't see any evidence of 22??
User avatar
admin
Posts: 4551
Joined: Thu Sep 09, 2010 3:08 pm

Re: Best way to decimate...

Post by admin »

Nobody has yet written a globally intelligent decimator for abitrary decimation. You are welcome to suggest solutions.

The problem is that given a video of M frames that you want to transform to N frames (N < M), how do you pick the "best" frames to remove, while retaining acceptable audio sync at all points of the video.
DAE avatar
magic144
Posts: 25
Joined: Sat Feb 19, 2011 2:53 pm

Re: Best way to decimate...

Post by magic144 »

Yeah, I dunno either - I think you would have to know/understand the process that did the forward transform (N->M) in the first place.
The sooner interlacing is quashed, the better! In a flat-panel, modern display world, I don't understand why there's even any need for geographical (framerate) discrepencies or interlacing!

srestore() seems to do a decent job, however I'm having this issue with stuttering - I'll supply some more info by next weekend...

thanks for now, appreciate the feedback!
m
DAE avatar
magic144
Posts: 25
Joined: Sat Feb 19, 2011 2:53 pm

Re: Best way to decimate...

Post by magic144 »

Hi Donald,
here's the files and discussion regarding the stuttering issue...

http://www.mediafire.com/?e8s6eqse62n8u

----

Source video clip is opening 30s of episode, video30s.mkv.
Index clip using DGIndexNV => video30s.dgi

2 approaches to encoding same clip:
METHOD 1. Convert via Huffyuv intermediate.

Convert the clip into a Huffyuv .avi using this script,
---huffy.avs---
LoadPlugin("C:\Users\User\Desktop\dgdecnv2038\DGDecodeNV.dll")
DGSource("video30s.dgi")
TRIM(0,150)
---
This produces (via VirtualDub using ffdshow/HuffYUV) video30s-huffy150fr.avi.

Now use a second script to encode to h.264.
NB the video30s-huffy150fr.grf file is produced using MONOGRAM GraphStudio (v0.3.2.0 Beta), just rendering video30s-huffy150fr.avi via FileSource (Async.), AVI Splitter and ffdshow (rev3721).
---testhuffy.avs---
Import("Srestore.avsi")
LoadPlugin("mt_masktools-26.dll")
LoadCplugin("C:\Program Files\MeGUI\tools\yadif\yadif.dll")
DirectShowSource("video30s-huffy150fr.grf",audio=false)
AssumeFPS(30000,1001) # as per source
yadif(mode=1,order=1)
srestore(frate=25,speed=-1)
---
This produces testhuffy-muxed.mkv, which plays back smoothly.


METHOD 2. Use this more straightforward script.
---test.avs---
LoadPlugin("C:\Users\User\Desktop\dgdecnv2038\DGDecodeNV.dll")
Import("Srestore.avsi")
LoadPlugin("mt_masktools-26.dll")
LoadCplugin("C:\Program Files\MeGUI\tools\yadif\yadif.dll")
DGSource("video30s.dgi")
yadif(mode=1,order=1)
srestore(frate=25,speed=-1)
TRIM(0,125) #125 is 150*5/6
---
This produces (via MeGUI) test-muxed.mkv. As you can see, there is a noticeable temporal stutter at the start of playback.


The question is, what induces the initial stuttering in approach 2. This is the preferred approach since it doesn't involve a very large intermediate file! Something, however, is not "playing nice".


FYI...

Command lines MeGUI generates when encoding the clips:-
----
"C:\Program Files\MeGUI\tools\x264\x264.exe" --level 4.1 --preset slow --crf 20 --no-dct-decimate --sar 1:1 --output "D:\Temp\_org\Stutter\testhuffy.264" "D:\Temp\_org\Stutter\testhuffy.avs"
"C:\Program Files\MeGUI\tools\x264\x264.exe" --level 4.1 --preset slow --crf 20 --no-dct-decimate --sar 1:1 --output "D:\Temp\_org\Stutter\test.264" "D:\Temp\_org\Stutter\test.avs"

Srestore() is found here:-
http://avisynth.org/mediawiki/Srestore
also requires Masktools2, found here:-
http://avisynth.org/mediawiki/MaskTools2
User avatar
admin
Posts: 4551
Joined: Thu Sep 09, 2010 3:08 pm

Re: Best way to decimate...

Post by admin »

Just out of curiosity, where are you getting these MKVs?

I'm not going to debug/analyse Srestore. Can you demonstrate a problem with DGSource() alone?
DAE avatar
magic144
Posts: 25
Joined: Sat Feb 19, 2011 2:53 pm

Re: Best way to decimate...

Post by magic144 »

I'm using eac3to to extract the episode from my BD directly to an mkv - I certainly would not trouble you with any pirated material or anything like that!
something like:-
eac3to L: 2) 2: video.mkv

Hmm, I don't know how to generate the issue with DGSource alone...
I was perhaps thinking it was something to do with the way srestore() was requesting frames from DGSource() - I don't know anything about how the filters interact in AviSynth, but maybe it is requesting frames from DGSource() in a funky way that is causing the problem??

Sorry I can't be of more help, but it has taken a very long time to get all the files together to even demonstrate the issue!!
Is there any way you can debug DGSource()'s behaviour in this scenario to see if anything is amiss?
DAE avatar
magic144
Posts: 25
Joined: Sat Feb 19, 2011 2:53 pm

Re: Best way to decimate...

Post by magic144 »

Hmm, the discrepency appears to NOT be between DGSource and srestore, BUT something at the interface with x264 itself...

If I load up the (METHOD 2) test.avs file into virtualdub and encode to an AVI (HuffYUV) I get a smooth output.

I just found and tried x264vfw so that I could attempt encoding to h.264 directly from virtualdub.
I CAN get a smooth encode, BUT I have to do one extra thing...

With normal settings, I get this message and frames are lost in the output:
x264vfw [warning]: Few frames probably would be lost. Ways to fix this:
x264vfw [warning]: - if you use VirtualDub or its fork than you can enable 'VirtualDub Hack' option
x264vfw [warning]: - you can enable 'File' output mode
x264vfw [warning]: - you can enable 'Zero Latency' option

So if I enable this "VirtualDub Hack" option, an .avi containing h.264 is correctly obtained (with smooth playback).
Looking on the x264vfw support forum, I see this description of the "hack"...

http://sourceforge.net/projects/x264vfw ... ic/3503993
The "VirtualDub Hack" option is highly recommended when using x264vfw with VirualDub (or VirtualDubMod). And it does NOT recommended with other encoding programs. It helps to overcome the limitation of VFW interface with its ideology of "one frame in, one frame out" when using B-frames or frame-base multithreading (or other techniques where frame buffering is needed).

Can this explain why there is an issue when using 'regular' x264.exe with this script?
User avatar
admin
Posts: 4551
Joined: Thu Sep 09, 2010 3:08 pm

Re: Best way to decimate...

Post by admin »

Sorry but I can't help you with x264vfw. Have you tried the normal x264 CLI?
DAE avatar
magic144
Posts: 25
Joined: Sat Feb 19, 2011 2:53 pm

Re: Best way to decimate...

Post by magic144 »

Yes, that's what MeGUI uses, and that's what produces the stuttering...
I get the same end-product if I run the x264.exe command myself from the command-line.
User avatar
admin
Posts: 4551
Joined: Thu Sep 09, 2010 3:08 pm

Re: Best way to decimate...

Post by admin »

You've told me that the script played in VirtualDub is fine. That appears to exonerate DGDecNV. Therefore, If you want help from me on this you'll have to make it real easy for me. Describe fully the *minimal case* that gives the problem and put ALL the needed pieces I need in one place (so far yadif is missing; if your minimal case includes x264.exe, give me that too).

Also, try generating just a raw h264 from MEGUI, rather than creating an MKV.
DAE avatar
magic144
Posts: 25
Joined: Sat Feb 19, 2011 2:53 pm

Re: Best way to decimate...

Post by magic144 »

Hi Donald,

I will update x264.exe and the yadif .dll in that shared folder.

FYI, I have posted this issue on the x264 forum in case some extra eyes can see what the issue is.
http://forum.doom9.org/showpost.php?p=1 ... tcount=180

Thanks!!

*EDIT*
files are uploaded.

so to reproduce the issue, just perform the METHOD2 encode, i.e.
x264.exe --level 4.1 --preset slow --crf 20 --no-dct-decimate --sar 1:1 --output test.264 test.avs
(or output directly to test.mkv if preferred)
NB - MeGUI is just used by me as a front-end, it's not necessary to reproduce the issue

for which you will need these files
x264.exe
test.avs
Srestore.avsi
mt_masktools-26.dll
yadif.dll
video30s.dgi
video30s.mkv
DGDecodeNV.dll (2038)

you may have to regenerate video30s.dgi yourself or adjust the path to video30s.mkv in video30s.dgi
you may also have to adjust the loadplugin paths in test.avs depending on where you put the dlls, etc

your output video should compare with test-muxed.mkv from my shared folder - you can notice stuttering at the beginning of playback
User avatar
admin
Posts: 4551
Joined: Thu Sep 09, 2010 3:08 pm

Re: Best way to decimate...

Post by admin »

I followed your process, creating test.264. I opened test.264 and did single step playback from the beginning. I do *not* see any stuttering.
DAE avatar
magic144
Posts: 25
Joined: Sat Feb 19, 2011 2:53 pm

Re: Best way to decimate...

Post by magic144 »

how did you perform this "single-step playback"?
User avatar
admin
Posts: 4551
Joined: Thu Sep 09, 2010 3:08 pm

Re: Best way to decimate...

Post by admin »

Open test.264 in DGIndexNV. Hit the >| button. Then use the right arrow key to step by frames. Maybe the MKV muxer or MKV player is at fault.
DAE avatar
magic144
Posts: 25
Joined: Sat Feb 19, 2011 2:53 pm

Re: Best way to decimate...

Post by magic144 »

you really have to watch it in real-time to observe the temporal stuttering

if you compare my test-muxed.mkv with testhuffy-muxed.mkv, you should see a difference

are you saying your test.264 does NOT playback (in realtime) like my test-muxed.mkv??

It looks like there are dupes in test-muxed.mkv that don't occur in testhuffy-muxed.mkv (at the start, before the panning speeds up from moon to earth)
User avatar
admin
Posts: 4551
Joined: Thu Sep 09, 2010 3:08 pm

Re: Best way to decimate...

Post by admin »

magic144 wrote:you really have to watch it in real-time to observe the temporal stuttering
If I can't see it when stepping through with a reliable viewer (DGIndexNV) then the source is clean. If a player has a problem with it, what's that to do with me?
if you compare my test-muxed.mkv with testhuffy-muxed.mkv, you should see a difference
I'm interested only in the minimal case. Is your minimal case now not minimal? :)
are you saying your test.264 does NOT playback (in realtime) like my test-muxed.mkv??
I don't want anything to do with your created MKVs. Show that the test.264 has an issue.

I assume you are aware that the yadif/srestore will reduce the apparent smoothness. Is this possibly what you are seeing? To me, stuttering means forward-backward motion as with the wrong field order, or jerking due to missing/duplicated frames, neither of which I see. If you are observing just that the smoothness is reduced, how can that be a surprise, given your processing?

I'm dubious that going through an intermediate HUFFYUV file can make any difference. But I'll try that. Still I will simply compare the two resulting .264 files.

Finally, what's the point of the GRF/DirectShow stuff in your METHOD 1? Can't you just use AVISource() in a script?
User avatar
admin
Posts: 4551
Joined: Thu Sep 09, 2010 3:08 pm

Re: Best way to decimate...

Post by admin »

I used AVISource() for your method 1. The .264 files from method 1 and method 2 are indistinguishable to me.
DAE avatar
magic144
Posts: 25
Joined: Sat Feb 19, 2011 2:53 pm

Re: Best way to decimate...

Post by magic144 »

The problem I'm seeing looks like dupe frames (not forward/backward) - it appears more obvious to me in real-time, but maybe that's just me.

Can you post the .264 file(s) you got so I can compare mine vs yours then??
I'll try and describe the anomolies in reference to specific frame numbers after that.

And no, I'm not changing my minimal case - I was just offering my two .mkv's for comparison so you could appreciate the difference I am seeing. For me, the .264 vs the .mkv makes no difference. The frame stepping is the same in both for me.

I didn't know you had such disdain for mkv :-)
(I somewhat understand, .264 is the pure stream).

cheers,
m

oh yeah, I just use the GRF so I can control (or at least be sure/aware of) which filters are used (ffdshow in this case).

also - what version of AviSynth are you using?
I'm using 2.5.8.5 I think
(avisynth.dll in C:\Windows\System32, 351,744 bytes)
plus I'm using Win7 32-bit Home Premium, fwiw
User avatar
admin
Posts: 4551
Joined: Thu Sep 09, 2010 3:08 pm

Re: Best way to decimate...

Post by admin »

I don't disdain MKV. I just want to limit the number of variables to a minimum. That's also why I want to eliminate DirectShow, which is notoriously flaky.

I will upload my encode when I get home and then post the link here.

My Avisynth is 2.58.
User avatar
admin
Posts: 4551
Joined: Thu Sep 09, 2010 3:08 pm

Re: Best way to decimate...

Post by admin »

Post Reply