I've created a thread over on the Nvidia forums about this:

https://forums.geforce.com/default/topi ... in-378-66/

Hopefully someone picks it up.

Oh dear I guess I'm at the mercy of Nvidia then

Hi Selur, yes I saw that workaround but my PC is mainly for gaming so I'm not going to be languishing on old drivers for very long. What are the downsides of avisynth 64-bit? I just want to run some encodes in Hybrid without having to stick to outdated GPU drivers.

I'm getting exactly the same behaviour as Selur, his first post describes exactly the same steps I took trying to get around this including the 64-bit exe working but when combined with the 32-bit DLL, the "CUDA error!" green screen. I'm hitting all the same error messages in all the same scenarios....