flac dev - Mar 2013 - Patch to add Unicode filename support for win32 flac

On 19.3.2013 15:49, JonY wrote:
> On 3/19/2013 19:59, Janne Hyv?rinen wrote:
>> On 18.3.2013 12:25, Erik de Castro Lopo wrote:
>>> JonY wrote:
>>>
>>>> Before anyone does anything, see __wgetmainargs
>>>> <http://msdn.microsoft.com/en-us/library/ff770599.aspx>.
>>>>
>>>> It can expand wildcards. Since it already provides
argc/argv/env, it is
>>>> more a less a drop-in replacement for the main() arguments.
>>> +1
>>>
>>> Erik
>> Alright, here's a patch utilizing this function. There's a lot
of
>> changes here.
>> Project files have a new configuration called "Release
(UTF8)", intended
>> to be used when building the command line tools. This project has the
>> required UTF-8 define in place so all libraries expect things in
encoded
>> format.
>> Regular Debug and Release configurations do not use any new tricks so
>> existing projects won't break when compiled with those settings.
>>
>> I'm at work and couldn't do extensive testing, but command line
FLAC.exe
>> seems to perform everything right with this.
>>
>> Metaflac probably requires some minor tweaks but I wanted to show some
>> progress so that 1.3 doesn't slip out the door while I'm busy.
>>
> Should the following
> #if defined _MSC_VER || defined __MINGW32__
>
> be simplified to
> #ifdef WIN32
>
> ? Alternatively _WIN32. Since it's win32 and not something compiler
> specific.
Indeed. Not sure what I was thinking.
>
> As for the macros:
> +#define flac_vfprintf vfprintf_utf8
> +#define flac_fopen fopen_utf8
> +#define flac_stat _stat64_utf8
> ...
>
> I leave this up for Erik to decide, though I would have preferred not
> using rename macros at all.
I think this is the sanest way. Only few #ifdefs instead of wrapper 
functions filled with them that do essentially nothing on *nix.
>
> Not sure if these were intended:
> +#include <sys/stat.h> /* for flac_stat() */
> +#include <sys/utime.h> /* for flac_utime() */
> +#include <io.h> /* for flac_chmod() */
Nope. Bug from mass replace.
>
> As for calling __wgetmainargs, I have some concerns about the security
> implications:
> LoadLibrary("msvcrt.dll") <- Which msvcrt? Theoretical
security exploit.
There is msvcrt.dll in the System32 dir in all supported Windows 
systems. That is what the function targets, but of course LoadLibrary 
searches from exe's dir first. I think security exploit concerns are 
warrantless, if you can place malicious replacement c-runtime dll in the 
exe's path you have already won.
>
> I think it is best to link it directly, please use the following
> prototype and call it directly:
>
> ============================================> #ifdef _DLL
> #define CALL_DLLIMPORT __declspec(dllimport)
> #else
> #define CALL_DLLIMPORT
> #endif
> int __cdecl CALL_DLLIMPORT __wgetmainargs(int*, wchar_t***, wchar_t***,
> int, int*);
> ============================================>
> This should simplify the error handling logic and help against
> LoadLibrary handle leaks, though the leak should not be an issue in
> practice since it is only called once. The symbol should also be present
> in MSVCR* DLLs.
This alone does nothing. It requires linking with an object file that 
then deals with the function. If we link against msvcrt.lib the flac.exe 
binary will no longer be static and it won't work without external 
runtimes (which would also be loaded from the exe's dir if they exist 
there). Linking with msvcmrt.lib won't find the function and unicode 
version msvcurt.lib causes this error:
Error    1    error LNK2005: ___iob_func already defined in 
msvcurt.lib(MSVCR110.dll)    G:\test\LIBCMT.lib(_file.obj)    test
Error    2    error LNK1169: one or more multiply defined symbols 
found    G:\test\Release\test.exe    test
>
> In stat_utf8, dealing with 32bit/64bit time_t? The following check
> should really compile time rather than runtime:
> sizeof(*buffer) == sizeof(st)
Entire stat_utf8 function can be removed. The code was changed later to 
always use stat64 one. Though I'd assume compiler to optimize sizeof 
checks appropriately.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 19.03.2013 20:35, Janne Hyv?rinen wrote:
> 
> On 19.3.2013 15:49, JonY wrote:
>> On 3/19/2013 19:59, Janne Hyv?rinen wrote:
>>> On 18.3.2013 12:25, Erik de Castro Lopo wrote:
>>>> JonY wrote:
>>>> 
>>>>> Before anyone does anything, see __wgetmainargs 
>>>>>
<http://msdn.microsoft.com/en-us/library/ff770599.aspx>.
>>>>> 
>>>>> It can expand wildcards. Since it already provides
>>>>> argc/argv/env, it is more a less a drop-in replacement for
>>>>> the main() arguments.
>>>> +1
>>>> 
>>>> Erik
>>> Alright, here's a patch utilizing this function. There's a
lot
>>> of changes here. Project files have a new configuration called
>>> "Release (UTF8)", intended to be used when building the
command
>>> line tools. This project has the required UTF-8 define in place
>>> so all libraries expect things in encoded format. Regular Debug
>>> and Release configurations do not use any new tricks so 
>>> existing projects won't break when compiled with those
>>> settings.
>>> 
>>> I'm at work and couldn't do extensive testing, but command
line
>>> FLAC.exe seems to perform everything right with this.
>>> 
>>> Metaflac probably requires some minor tweaks but I wanted to
>>> show some progress so that 1.3 doesn't slip out the door while
>>> I'm busy.
>>> 
>> 
>> As for calling __wgetmainargs, I have some concerns about the
>> security implications: LoadLibrary("msvcrt.dll") <- Which
msvcrt?
>> Theoretical security exploit.
> 
> There is msvcrt.dll in the System32 dir in all supported Windows 
> systems. That is what the function targets, but of course
> LoadLibrary searches from exe's dir first. I think security exploit
> concerns are warrantless, if you can place malicious replacement
> c-runtime dll in the exe's path you have already won.
See [1] for the info. According to that article,
LoadLibraryA("msvcrt.dll") should be perfectly safe.


[1]
http://msdn.microsoft.com/en-us/library/windows/desktop/ms682586%28v=vs.85%29.aspx

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJRSJzFAAoJEOs4Jb6SI2CwmdMIAJZyqiZ+PjwivvZ0QM5l/DdD
aoS6zf5/crr6Dm6IMm58Sf/4RMIRUM4d/4wXZ4xKyQvP7wmxzzAN5QDTwHRXsDY7
CyjpPKv1NGiwOYux9fNUEQyo7eNvtl8BEg2pq7fXLH2h/kBnCtB7/V+X8OZqiSFD
na2YnUOpNxBq0LnmkM3gQqlXE9ajsBDUNYfFDHBVtu3ZXDUVwKhdH8kX2pTJcjeS
QSCwdxGS2uGVUj+E/+hUny3wcBEVN2z8gGxWzrGAMTcV4dYHlcD7cKXfG59eAw93
HomBZvrRLVss4aXrV2fGZ5VQm2AjlNRFKTNhtxkZ7npB4ManN0C0vuPxYMe3Uuk=6WoQ
-----END PGP SIGNATURE-----

On 19.3.2013 19:13, LRN wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 19.03.2013 20:35, Janne Hyv?rinen wrote:
>> On 19.3.2013 15:49, JonY wrote:
>>> On 3/19/2013 19:59, Janne Hyv?rinen wrote:
>>>> On 18.3.2013 12:25, Erik de Castro Lopo wrote:
>>>>> JonY wrote:
>>>>>
>>>>>> Before anyone does anything, see __wgetmainargs
>>>>>>
<http://msdn.microsoft.com/en-us/library/ff770599.aspx>.
>>>>>>
>>>>>> It can expand wildcards. Since it already provides
>>>>>> argc/argv/env, it is more a less a drop-in replacement
for
>>>>>> the main() arguments.
>>>>> +1
>>>>>
>>>>> Erik
>>>> Alright, here's a patch utilizing this function.
There's a lot
>>>> of changes here. Project files have a new configuration called
>>>> "Release (UTF8)", intended to be used when building
the command
>>>> line tools. This project has the required UTF-8 define in place
>>>> so all libraries expect things in encoded format. Regular Debug
>>>> and Release configurations do not use any new tricks so
>>>> existing projects won't break when compiled with those
>>>> settings.
>>>>
>>>> I'm at work and couldn't do extensive testing, but
command line
>>>> FLAC.exe seems to perform everything right with this.
>>>>
>>>> Metaflac probably requires some minor tweaks but I wanted to
>>>> show some progress so that 1.3 doesn't slip out the door
while
>>>> I'm busy.
>>>>
>>> As for calling __wgetmainargs, I have some concerns about the
>>> security implications: LoadLibrary("msvcrt.dll") <-
Which msvcrt?
>>> Theoretical security exploit.
>> There is msvcrt.dll in the System32 dir in all supported Windows
>> systems. That is what the function targets, but of course
>> LoadLibrary searches from exe's dir first. I think security exploit
>> concerns are warrantless, if you can place malicious replacement
>> c-runtime dll in the exe's path you have already won.
> See [1] for the info. According to that article,
> LoadLibraryA("msvcrt.dll") should be perfectly safe.
>
>
> [1]
>
http://msdn.microsoft.com/en-us/library/windows/desktop/ms682586%28v=vs.85%29.aspx
>
>
Seems safe indeed.
Attached an updated patch where metaflac works too.
Test MSVC 2012 compiles of flac.exe and metaflac.exe uploaded to 
http://www.saunalahti.fi/~cse/temp/flac-1.3pre2-mod.zip

-------------- next part --------------
A non-text attachment was scrubbed...
Name: utf8_io_v2.zip
Type: application/x-zip-compressed
Size: 24509 bytes
Desc: not available
Url :
http://lists.xiph.org/pipermail/flac-dev/attachments/20130319/b5811db9/attachment-0001.bin

On 3/20/2013 00:35, Janne Hyv?rinen wrote:
>>
>> As for calling __wgetmainargs, I have some concerns about the security
>> implications:
>> LoadLibrary("msvcrt.dll") <- Which msvcrt? Theoretical
security exploit.
> 
> There is msvcrt.dll in the System32 dir in all supported Windows 
> systems. That is what the function targets, but of course LoadLibrary 
> searches from exe's dir first. I think security exploit concerns are 
> warrantless, if you can place malicious replacement c-runtime dll in the 
> exe's path you have already won.
> 
Yeah, which is why I said it was theoretical.

I've seen code that use __ImageBase to over the import tables to find
out which MSVCR* DLL is used and use GetModuleHandleA to avoid LoadLibrary.
>>
>> I think it is best to link it directly, please use the following
>> prototype and call it directly:
>>
>> ============================================>> #ifdef _DLL
>> #define CALL_DLLIMPORT __declspec(dllimport)
>> #else
>> #define CALL_DLLIMPORT
>> #endif
>> int __cdecl CALL_DLLIMPORT __wgetmainargs(int*, wchar_t***, wchar_t***,
>> int, int*);
>> ============================================>>
>> This should simplify the error handling logic and help against
>> LoadLibrary handle leaks, though the leak should not be an issue in
>> practice since it is only called once. The symbol should also be
present
>> in MSVCR* DLLs.
> 
> This alone does nothing. It requires linking with an object file that 
> then deals with the function. If we link against msvcrt.lib the flac.exe 
> binary will no longer be static and it won't work without external 
> runtimes (which would also be loaded from the exe's dir if they exist 
> there). Linking with msvcmrt.lib won't find the function and unicode 
> version msvcurt.lib causes this error:
> Error    1    error LNK2005: ___iob_func already defined in 
> msvcurt.lib(MSVCR110.dll)    G:\test\LIBCMT.lib(_file.obj)    test
> Error    2    error LNK1169: one or more multiply defined symbols 
> found    G:\test\Release\test.exe    test
> 
There is no __wgetmainargs in the static libcmt? Interesting.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 834 bytes
Desc: OpenPGP digital signature
Url :
http://lists.xiph.org/pipermail/flac-dev/attachments/20130320/a54e6be6/attachment.pgp