[RESOLVED] DGIndexNV 2048 doesn't handle Shift-JIS names

[admin]

DGDecodeNV 2048 could not open the script. Does that work for you?

I traced and although the DGI filename string is correct in the script, my filter does not receive it correctly from Avisynth. It's wrong already in args[0].AsString(dgi). My locale is set to Japanese of course.

I'm getting ready to throw in the towel. There appears to be no solution short of modifying Avisynth.

On the other hand, didn't FFMS2 claim to have solved this? Anybody know what they did?

[admin]

Read my follow up!

[admin]

Yes, I said that in my post.

What version of Avisynth do you use? Can you attach the script please?

Something is strange because I do not receive the string correctly from Avisynth.

[admin]

I open it in Notepad (not Notepad+). I see the correct Japanese text there.

Looking at your package now. Thank you.

[admin]

Aha. Try it turning off full paths in DGIndexNV options.

OK, so we have a few things to fix.

[admin]

I open it in Notepad (not Notepad+). I see the correct Japanese text there.

I never use notepad but I seem to recall that it saves to UTF-8 by default.

Nope, it doesn't. But Notepad+ converts it to UTF-8 if there are non-ANSI characters.

The problem occurs when I save without full path information. That I can probably fix.

So maybe I don't have to throw in the towel.

[admin]

I'll try to fix those few things, but right now... I'm off to the pool!

Thanks for your help.

[Aleron Ives]

I never use notepad but I seem to recall that it saves to UTF-8 by default.

Notepad will save to ISO 8859-1 by default, unless you include characters unsupported by that codepage. If you do, you'll get an error saying that you must either a) go back and select a different encoding that supports the content you wrote or b) save anyway and have your unsupported characters turn into gibberish. If you save as UTF-8 in Notepad, it will always write a BOM. There is no way to get rid of it. If you want to switch between character sets quickly and view which character set a specific text file uses, I've found MadEdit to be pretty simple and effective, even if it's orphaned.

Also, I am scared to change my locale because I may not be able to switch back because I cannot read Japanese.

I think it's worth noting that depending upon your Windows version, locale changes are not even possible. Windows 7 Home Premium and Professional do not allow you to properly change your locale, for instance. Only Windows 7 Ultimate will let you add the MUI language pack for proper language support, which means any solution which requires locale switching is off limits to many users. It is perhaps reasonable to assume people who need support for multiple languages would have purchased Ultimate anyway, but I think it's also reasonable to assume people who have second languages other than English may want support for those languages in their programs, even if they don't want to change the language of Windows itself. Windows XP does not have this problem, nor does Windows 8 (AFAIK), since Microsoft didn't release 8 with so many different versions.

OK, remember the script is created by my AVS template system. That's another good reason not to use UTF-8 for the script, because a user making a script by hand will not be using UTF-8.

Why not? Even Notepad lets you save text files in UTF-8, which is what you'd have to do to reference files named in other languages (unless you wanted UTF-16, for some reason).

[admin]

I think it's worth noting that depending upon your Windows version, locale changes are not even possible. ...

It's useful to know that. I would think, though, that a Japanese system would already have the right locale set. Anyway, this is the only way we can get things working, absent changes to Avisynth.

Why not? Even Notepad lets you save text files in UTF-8, which is what you'd have to do to reference files named in other languages (unless you wanted UTF-16, for some reason).

Perhaps, but it is moot, because Avisynth will reject the files.

I've already fixed the window title and relative paths issues. I'll just check for any further such issues, fix any I find, and then make a test version available. I haven't decided whether to slipstream it or make a 2049 version.

[admin]

It did change some font rendering on Windows GUI elements on my system (not the language, just how they appear), so it is not totally benign as you suggest.

[admin]

E.g., the text in the Audio Demux menu got all skinny looking. Also, look at the screenshot below:

[admin]

I have uploaded to the download area a build called dgdecnv2048_test.zip. This should fix all the issues we have noticed (and others). I would appreciate your test results. Both DGIndexNV and DGDecodeNV need to be updated. And, of course, your system locale needs to be set appropriately.

I'd especially like to hear from jmt247 as to whether his issue is resolved.

[admin]

Thanks!

[admin]

That's good to hear.

Here are the things I fixed but didn't test thoroughly:

AVS template handling (with non-ANSI file names)
Load project (ditto)
MP4/MKV, especially audio demuxing of them

Thanks again for your expert advice and testing.

[admin]

Sweet

Then I will slipstream it into the release 2048 and post here when that is done.

Now I am going to dip my feet into 64-bit ASM stuff. I wanted to implement some anticracking stuff but there is no inline ASM for 64-bit. VS2013 has the 64-bit MASM, though, I just have to figure out how to build and invoke from C++ etc. And it will come in useful if I decide to low-level optimize any normal functionality. E.g., I may ditch the CUDA NV12->RGB24 and replace it with ASM. That will avoid issues when nVidia obsoletes older sm's. BTW, they never responded to my post about it at their forum.

[admin]

Too late. Slipstream completed. It was really hard to rename the zip.

Our testing and my robust coding will save the bacon.

[jmt247]

OP, here. the filename handling really is now perfect for my use. Thank you very much!

[admin]

Great. Thanks for the feedback and for pointing out the issue.

[AJR]

Hello,

I am not sure if I understand this thread correctly. Does it mean that now are supported file names with special national characters, like in Czech language are? If so then I have to report it doesnt work properly

If I try create index file via cli of this file:

Code: Select all

Mezi životem a smrtí (5. díl) - Pád jedné legendy.h264

then in the indexed file *.dgi is this:

Code: Select all

Mezi �ivotem a smrt� (5. d�l) - P�d jedn� legendy.h264

After that I cant open *.avs file (which include path to *.dgi) with VD because of "missing" file If I remove special czech characters before dgi creation everything is ok.

Is it bug or just my ignorance?

[admin]

Make sure that you:

1. Have set your system locale to the correct language.

2. Have the latest 2048.

[AJR]

All these setting are fine.

I have just found the problem is not related to DGDecNV. I use private GUI for encoding and if I build dgi via this GUI (of course it still use your DGDecNV) it produce wrong dgi with bad characters.

Sorry for my fault. Your tool works great Tested via GUI and CLI and both create fine dgi with correct characters.

[admin]

Good to hear and thanks for the update.

[AJR]

Hello,

can you please integrate this also in the DgDecIM ? I would very appreciate it.

Thank you for interest

[admin]

Sorry but DGDecIM is no longer developed.

[AJR]

I know it, but i hoped you could integrate it with minimal effort if you already have code for dgdecnv.

Anyway thanks for reply. I appreciate your work.