Vorbis dev - Sep 2001 - vorbis-tools reorganziation and UTF-8 stuff

I just moved a bunch of shared code (like 4 copies of getopt) into a share
directory within vorbis-tools.  I also moved the UTF-8 code from oggenc into
the vorbis-tools/share directory as well so that it could be used by all of
the tools, since they all need to handle UTF-8 comments correctly.  Header
files for the shared code are stored in vorbis-tools/include.

I have already fixed vorbiscomment to encode comments in UTF-8 (like oggenc
already did), but I have not fixed ogg123 or oggenc to display UTF-8 comments
in the native charset yet.  This is because utf8.c does not have any decode
functions, since they weren't needed.  Since I would rather debug the
kcarnold_branch ogg123 than figure out UTF-8, I'm throwing out this list of
requests to the more motivated and/or knowledgable people out there:

1. Check vorbiscomment to make sure I really did fix UTF-8 encoding on it.  (I
don't know how to test it.)

2. Write utf8_decode() routines for vorbis-tools/share/utf8.c.  We actually
need three implementations: Windows, UNIX with iconv and UNIX without
iconv.  Look at the utf8_encode() routines for examples of what I'm talking
about.  There are already utf8_decode() stubs in both the Win32 and UNIX
#ifdef sections.

3. Update ogginfo to use the newly written utf8_decode() routines.  Whoever
does #2 should do #3 so they have a testbed for their work.

4. Fix oggenc building on Win32.  Since I moved the getopt and utf8 code
around, I'm sure I broke Win32 builds.  :(

Thanks to anyone who tackles these problems!


---
Stan Seibert

--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to
'vorbis-dev-request@xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is
needed.
Unsubscribe messages sent to the list will be ignored/filtered.

Stan Seibert:
> I have already fixed vorbiscomment to encode comments in UTF-8 (like oggenc
> already did), but I have not fixed ogg123 or oggenc to display UTF-8
comments
> in the native charset yet. This is because utf8.c does not have any decode
> functions, since they weren't needed. Since I would rather debug the
> kcarnold_branch ogg123 than figure out UTF-8, I'm throwing out this
list of
> requests to the more motivated and/or knowledgable people out there:
 
Bruno Haible wrote a portable libiconv library that probably runs on
anything vorbis-tools runs on. You could just make people install
libiconv if their system doesn't provide an adequate version of iconv.

I know it's tempting to think "let's just include some simple
charset
conversion functions so we have fewer dependencies", but you also have
to think about the duplication of effort, bloat and maintenance burden
that occurs when each program on a system includes its own set of
simple charset conversion functions.

So, I think you should seriously consider throwing out all the charset
conversion stuff from vorbis-tools and simply requiring iconv.

By the way, it's a good idea to feed data through iconv even when the
charset is the same: a decent version of iconv will validate UTF-8
data if told to convert from UTF-8 to UTF-8.

Also, do you want transliteration? You can get iconv to do approximate
conversion, e.g. convert 'ö' to '"o'. This is good for
displaying
data, but there is a danger of the user not realising that the data
has been transliterated and converting it back to UTF-8, for example
when editing a comment in a non-UTF-8 locale.

You might want versions of

int utf8_decode(const char *from, char **to, const char *encoding)
int utf8_encode(const char *from, char **to, const char *encoding)

that return:

0 - data was exactly converted
1 - data was approximately converted
2 - input was invalid
3 - unknown encoding

Both functions could make an attempt at converting the data even when
there is an error, though you probably only want to use that facility
when converting the data for display, if at all.

Edmund

--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to
'vorbis-dev-request@xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is
needed.
Unsubscribe messages sent to the list will be ignored/filtered.

I updated the oggenc.dsp project file for win32 building with MSVC.
In addtion, I added two new configurations to the project for dynamic 
building and linking for debug/release, so in addition to "Win32
Debug"
and "Win32 Release", there is "Win32 Debug DLL" and
"Win32 Release DLL".

I also would like to propose a small amount of reorganization for win32
building.
instead of having the project file in the same directory as oggenc, how about 
adding a subdirectory called "win32" under "vorbis-tools"
and locate the win32
project there.  In addtion I can create projects for ogginfo and ogg123.  Then
I would create an workspace to encompass these projects as well as the 
library projects, thus providing single step building.  Comments?

On 9/22/2001 at 5:14 PM volsung@asu.edu wrote:
>I just moved a bunch of shared code (like 4 copies of getopt) into a share
>directory within vorbis-tools.  I also moved the UTF-8 code from oggenc
>into
>the vorbis-tools/share directory as well so that it could be used by all of
>the tools, since they all need to .
>
>4. Fix oggenc building on Win32.  Since I moved the getopt and utf8 code
>around, I'm sure I broke Win32 builds.  :(
>
>Thanks to anyone who tackles these problems!
>
>---
>Stan Seibert
--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to
'vorbis-dev-request@xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is
needed.
Unsubscribe messages sent to the list will be ignored/filtered.