Issue with path

Support forum for DGDecNV
User avatar
tormento
DG Approved/Moose Approved
Posts: 542
Joined: Mon Sep 20, 2010 2:18 pm

Issue with path

Post by tormento »

I think there has been a regression within the various releases, after my computer broke.

As, after recovering it, I jumped to latest I don't really know which one.

I used to utilize a "—" (ALT+0151) symbol on some paths, both BDRIP input and destination, such as:

G:\Raw\padrino III coda, Il — La morte di Michael Corleone

containing the ripped BD structure and

F:\In\2_37 padrino III coda, Il — La morte di Michael Corleone

as destination for DGI, demuxed tracks and AVS.

It does not work anymore. Various encoding GUIs tell me a generic hint to use Fullpath (which is enabled) or simply tell me they can't find the DGI file.

I don't think it's a long path problem as if I replace "—" with "-" (standard minus), everything works ok.
User avatar
Rocky
Moose Approved
Posts: 2172
Joined: Fri Sep 06, 2019 12:57 pm

Issue with path

Post by Rocky »

Try it directly with just your script and VirtualDub2 and report results. I am not responsible for what encoding apps do.

Probably it's a unicode thing. Upgrade to the latest Win10 and then set the beta option to always use UTF-8 codepage in the regional settings.

We just went through all this with staxrip.

viewtopic.php?p=12931#p12931
User avatar
tormento
DG Approved/Moose Approved
Posts: 542
Joined: Mon Sep 20, 2010 2:18 pm

Issue with path

Post by tormento »

Rocky wrote:
Sat Jan 23, 2021 7:31 pm
Try it directly with just your script and VirtualDub2 and report results.
Yep, I tried it too and it tells me it can't find the file on line xx of the script.

I am reporting this because never happened before. Do you have some older build to give me to try?
User avatar
tormento
DG Approved/Moose Approved
Posts: 542
Joined: Mon Sep 20, 2010 2:18 pm

Issue with path

Post by tormento »

I have found DGDecNV 2053.0.0.216 in a backup from July.

Results with VirtualDub2_x64:

DGI and AVS generated by DGIndexNV 216 works with DGDecodeNV 216
DGI and AVS generated by DGIndexNV 223 have path issue with DGDecodeNV 216
DGI and AVS generated by DGIndexNV 216 works with DGDecodeNV 223
DGI and AVS generated by DGIndexNV 223 have path issue with DGDecodeNV 223

It seems it's DGIndexNV the one to investigate.
User avatar
Rocky
Moose Approved
Posts: 2172
Joined: Fri Sep 06, 2019 12:57 pm

Issue with path

Post by Rocky »

Compare the DGI files generated with the two DGIndexNV versions. How do they differ?

I'll test this after breakfast.
User avatar
tormento
DG Approved/Moose Approved
Posts: 542
Joined: Mon Sep 20, 2010 2:18 pm

Issue with path

Post by tormento »

Rocky wrote:
Sun Jan 24, 2021 4:35 am
Compare the DGI files generated with the two DGIndexNV versions. How do they differ?
There is definitely a difference in the two DGI, as you can see below:

Image
User avatar
Rocky
Moose Approved
Posts: 2172
Joined: Fri Sep 06, 2019 12:57 pm

Issue with path

Post by Rocky »

Still lying groggy in my nest right now.

Do you have a similar difference in the AVS scripts that were generated?
User avatar
tormento
DG Approved/Moose Approved
Posts: 542
Joined: Mon Sep 20, 2010 2:18 pm

Issue with path

Post by tormento »

Rocky wrote:
Sun Jan 24, 2021 4:52 am
Do you have a similar difference in the AVS scripts that were generated?
Yep, I was about posting.

Image

I replaced DGI comparison too with a cleaner one.
User avatar
Rocky
Moose Approved
Posts: 2172
Joined: Fri Sep 06, 2019 12:57 pm

Issue with path

Post by Rocky »

Did you try setting the Win10 UTF-8 option that I mentioned?
User avatar
tormento
DG Approved/Moose Approved
Posts: 542
Joined: Mon Sep 20, 2010 2:18 pm

Issue with path

Post by tormento »

Rocky wrote:
Sun Jan 24, 2021 5:23 am
Did you try setting the Win10 UTF-8 option that I mentioned?
I know it and I tried it multiple times on the past years and months but it never went out of beta and it's inconvenient to keep it enabled as I have other, major, problems with file management.

Anyway, as there is a clear change between DG builds on the very same operative system, perhaps something changed in the compile environment if not in the source itself.
User avatar
Rocky
Moose Approved
Posts: 2172
Joined: Fri Sep 06, 2019 12:57 pm

Issue with path

Post by Rocky »

Please cooperate with my troubleshooting procedure and answer my questions. Does it work with the option enabled? If so, then I know it is a unicode issue.

Going back to 216-era code willl break our unicode support, which currently depends on the option.

If you want to be really cooperative tell us what problems you encounter with the option set.

EDIT: Don't I sound like a dumbass?
User avatar
tormento
DG Approved/Moose Approved
Posts: 542
Joined: Mon Sep 20, 2010 2:18 pm

Issue with path

Post by tormento »

Rocky wrote:
Sun Jan 24, 2021 9:11 am
Please cooperate with my troubleshooting procedure
Ok, just let me finish the encode as I have to reboot to enable it.
User avatar
Rocky
Moose Approved
Posts: 2172
Joined: Fri Sep 06, 2019 12:57 pm

Issue with path

Post by Rocky »

Cool, thank you. Here's the deal. Everything works fine with the beta option set for me using your unicode filename. I wouldn't be put off by the "beta". Lots of people use it with no issues, including me. If you have issues we'll need to know what they are!

DGIndexNV now also sets a UTF-8 manifest, so it generates correct UTF-8 output files even without the beta option. But third-party applications may fail to read it properly (VirtualDub, encoding apps, etc.). We cannot control them. You could ask authors of non-complying programs to get onboard with the 'UTF-8 everywhere' initiative. It's the only way DG tools will support unicode. No problems communicating with linux systems, etc. Face it, UTF-16 was a big mistake.

http://utf8everywhere.org/

So at this time, until UTF-8 support becomes ubiquitous, it is safest to set the beta option. If you cannot use the option, until your desired applications support UTF-8, do not try to use unicode names.

Finally, if you really want 216-like behavior, you can turn off the beta option and remove the UTF-8 manifest using ResourceHacker. Previously people were hacking to include the manifest! If you feel forced to do this, it sure would be great to know why.

Your move.
User avatar
tormento
DG Approved/Moose Approved
Posts: 542
Joined: Mon Sep 20, 2010 2:18 pm

Issue with path

Post by tormento »

Rocky wrote:
Sun Jan 24, 2021 10:32 am
Cool, thank you. Here's the deal. Everything works fine with the beta option set for me using your unicode filename. I wouldn't be put off by the "beta". Lots of people use it with no issues, including me. If you have issues we'll need to know what they are!
You and perhaps the majority of people is using english based language.

For me and all the other people, the downside is that ANSI-Only programs (most older programs) will show garbage instead of accented characters.

I have more than 50k files stored and a good percentage of those has some kind of accent or "strange" character.

I simply can't find and correct all of them and some other can't be corrected at all, such as my University cloud on Onedrive

Image (it should be Università di Pavia)

that will cause massive mayhem on files and programs referring to its content.

The good news is that, with Unicode support turned on, latest DG works.

I anyway doubt that I will keep that support on for the reason I explained you, i.e. too many programs that I use are ANSI-only and they will cause a mess. I prefer to stop using "strange" characters on material path when working with DG.

Thanks for all your support, comprehension and clarification. :salute:
User avatar
Rocky
Moose Approved
Posts: 2172
Joined: Fri Sep 06, 2019 12:57 pm

Issue with path

Post by Rocky »

Delete the manifest and turn off the beta option. If that is good for you I can make it a setting.
User avatar
tormento
DG Approved/Moose Approved
Posts: 542
Joined: Mon Sep 20, 2010 2:18 pm

Issue with path

Post by tormento »

Rocky wrote:
Sun Jan 24, 2021 2:15 pm
Delete the manifest and turn off the beta option. If that is good for you I can make it a setting.
That would be a true act of kindness. Thank you. :salute:
User avatar
Rocky
Moose Approved
Posts: 2172
Joined: Fri Sep 06, 2019 12:57 pm

Issue with path

Post by Rocky »

Oh Jeez, it's not programmable. We may have to have two versions: DGIndexNV and DGIndexNV_UTF8. Please, no.
User avatar
Rocky
Moose Approved
Posts: 2172
Joined: Fri Sep 06, 2019 12:57 pm

Issue with path

Post by Rocky »

OK, we can ship with an external manifest file DGIndexNV.exe.manifest instead of embedding it. Then if you don't want UTF-8 support just rename the manifest file so it isn't found. I'll test that and release tomorrow if OK.
User avatar
Rocky
Moose Approved
Posts: 2172
Joined: Fri Sep 06, 2019 12:57 pm

Issue with path

Post by Rocky »

@tormento and renols

Please re-download and test 2053.0.0.224. The binaries now do not contain any embedded manifests, and instead an external manifest file is included. Here is how it works:

For UTF-8 operation (unicode support), do one or both of these two things:
--------------------------------------------------------------------------------------------

1. Enable the "beta" option in Win10's regional settings to force the OS to use UTF-8 globally.
If you don't do this but only 2 below, then 3rd-party apps may not be able to read the
AVS and DGI files properly, depending on whether they ask the OS to use the UTF-8
codepage (i.e., they have a manifest asking for it). If this option is set, all applications
automatically use the UTF-8 codepage. We should all be migrating towards this.
Tell your application authors that they need to support UTF-8!

2. Make sure the file DGIndexNV.exe.manifest is co-located with DGIndexNV.exe.
This forces DGIndexNV.exe to ask for the UTF-8 codepage. Watch out, though,
you will be unable to open files whose names include legacy code values that
are not in UTF-8, and your AVS/DGI files will be written in UTF-8.

For legacy codepage operation (system locale):
----------------------------------------------------------

Turn off the win10 "beta" option and delete DGIndexNV.exe.manifest from the directory that
holds DGIndexNV.exe.

Given that Win10 doesn't allow dynamic control of the manifest, this seems to be the best (only?)
compromise to cover all our bases.

Please let us know how it goes. If it's OK, I'll announce it in the Binaries Notification thread.
User avatar
Rocky
Moose Approved
Posts: 2172
Joined: Fri Sep 06, 2019 12:57 pm

Issue with path

Post by Rocky »

Y'all want to hear something really funny (tragic)?

Turns out that you cannot build CUDA files with Visual Studio when the beta UTF-8 option is on! And apparently it's not just CUDA files.

https://stackoverflow.com/questions/578 ... -dd-vctool

Wow, Microsoft makes the compiler and the OS, and they foist this caca on us? Gonna do some more research on this error, because there has to be more to this story. Microsoft cannot seem to figure it out:

https://developercommunity.visualstudio ... -i-46.html

So, now I have the beta option off. I find that if I do not include the manifest, then what tormento wants is working, i.e., you can open the disk without problems and write the AVS/DGI files in the legacy codepage, so it acts like 216 did. But now if you include the manifest, things don't work if the filename includes codes not present in UTF-8. Microsoft informs us that we need to perform "legacy code page detection and conversion" in such a case, although things are fine for filenames without non-UTF-8 codes. What a cluster-you-know-what.

I'm going to leave things as is, i.e., no embedded manifest, for now, while I research the options. The presence of the external manifest means you will get UTF-8 AVS/DGI files, and some apps may not be able to read them (with the beta option off). To avoid that, rename/delete the manifest file. Legacy code page detection, my ass!

So, tormento and renols, for 216-like behavior, turn off the beta option and rename the manifest file.

And now, because I don't want to spend my life navigating Microsoft screw-ups, I'm going to implement MPLS explosion for tormento. Wheee!
User avatar
Rocky
Moose Approved
Posts: 2172
Joined: Fri Sep 06, 2019 12:57 pm

Issue with path

Post by Rocky »

Should probably ship with manifest file name 'DGIndexNV.exe.manifest~'. Then for the few that want UTF-8, they would delete the ~. Also, for normal system locale setting (not the beta option) you will want to run without the manifest.

I'll update 224 that way right now.

EDIT: Done. It should behave like 216 out-of-the-box. You'll have to take action to get UTF-8 support.
User avatar
tormento
DG Approved/Moose Approved
Posts: 542
Joined: Mon Sep 20, 2010 2:18 pm

Issue with path

Post by tormento »

Rocky wrote:
Mon Jan 25, 2021 9:44 am
Turns out that you cannot build CUDA files with Visual Studio when the beta UTF-8 option is on! And apparently it's not just CUDA files.
Yesterday I read https://en.wikipedia.org/wiki/Unicode_i ... _platforms and some doubts arose in me. Anyway I did not say anything because I know you are a skilled coder.
Rocky wrote:
Mon Jan 25, 2021 9:44 am
So, tormento and renols, for 216-like behavior, turn off the beta option and rename the manifest file. And now, because I don't want to spend my life navigating Microsoft screw-ups, I'm going to implement MPLS explosion for tormento. Wheee!
Well, thanks! :belly-laugh:
User avatar
tormento
DG Approved/Moose Approved
Posts: 542
Joined: Mon Sep 20, 2010 2:18 pm

Issue with path

Post by tormento »

P.S: Just tested and works fine.
User avatar
Rocky
Moose Approved
Posts: 2172
Joined: Fri Sep 06, 2019 12:57 pm

Issue with path

Post by Rocky »

Thanks big t. I'll mark it closed and make a Binaries Notification announcement.
User avatar
Sherman
Moose Approved
Posts: 329
Joined: Mon Jan 06, 2020 10:19 pm

Issue with path

Post by Sherman »

Rocky wrote:
Mon Jan 25, 2021 9:44 am
I'm going to implement MPLS explosion for tormento.
Rocky, to make up for my earlier disrespect, can I do that for you? Your wish is my command.
Post Reply