Erik Kruus
2000-Dec-19 11:34 UTC
[vorbis-dev] what script or command makes the codebooks? (fwd)
---------- Forwarded message ---------- Date: Mon, 18 Dec 2000 15:50:05 -0500 (EST) From: Erik Kruus <kruus@on2.com> Reply-To: vorbis-dev@xiph.org To: vorbis-dev@xiph.org Subject: Re: [vorbis-dev] what script or command makes the codebooks? I've been through the entire vq table process during the last week. Some folks have already suffered through listening to .ogg files encoded with a 75 kbps [stereo, 44kHz input] mode. I ran another training run that took 30 hours or so on a PIII-600 on a small, but reasonably-sized training set, and Monty gave me a file outlining what he had done to start me off, and at the end, I attach a section. There are some things you need to do because the current vq/ is not quite compatible with the current lib/. I do my retraining with the Ugliest-Bash-Script-Ever-Written, and have it mostly automatic. On Mon, 18 Dec 2000, Gregory Maxwell wrote:> On Sun, Dec 17, 2000 at 11:07:28PM -0600, Mike Coleman wrote: > > Gregory Maxwell <greg@linuxpower.cx> writes: > > > No. It's easy to generate training data, it's just a define in the source. > > > > Yeah, but I'm trying to see exactly how these particular codebooks were > > produced. (Actually, it would be kind of nice if each .vqh file included a > > "pedigree" inside a comment that described exactly how it was produced (args, > > etc), and maybe even gave a list of training data sources together with > > md5sums. > > They almost do, nothing that nice though.. Their params (in the vqhs) and > the training set is all you need to reproduce them. > > The process runs the data through several commands, so its hard to track the > data and it's assume that you know what you are doing if you are using those > tools.:) > > Steps, in rough, > > Change #if 0 in lib source to make it generate training data,These are compile-time switches. I added targets to my Makefile.am to build with -DTRAIN_LSP and/or -DTRAIN_RES (and/or -DTRAIN_RESENT, if that's more what you need).> > Train the LSP data (no info here because I havn't done it since Vorbis > switched to lattice for LSP).You'll probably need to change some code in latticetune.c for this to work. (I had to: the q_sequencep treatment was a bit off, and you don't interleave that data).> > Regenerate training data with LSP books because residue is affected by LSP > books.Here I mucked with vorbisenc.c to add a new mode with a certain bit-rate window.> > Run residue training data into residuesplit (parms in mode_*.h but not > stated in an obvious way), which splits the residue into partitions.Do this on a LARGE partition with many gigs of free space, to get a feel for how big those intermediate files will be.> > Lattice build each partition (parms can be extracted from the vqhes).latticetune.c is the source. Use the version that does interleaving.> > Latticepare, to reduce for targeted bitrate.'entries' or something is initialized badly if your input tables are already sparse. There's a trivial fix, but I pared my tables differently for low-threshold ones, and kept the full table for the high-residual cases. I'll probably use latticepare myself for future experimentation, it's just that I haven't yet done enough myself to be convinvced I changed it correctly enough. (I recommend you tune them without cutting them down in size, if you want to use latticepare. I use rather large training sets, so I asked my scripts to zero all the never-used entries for me.)> > Huffbuild to produce, aux book.Works fine.> > Latticehint the books.Works fine. (I think I rewired some of the vq code to use the lib/ search versions, which actually do fall back to a brute-force search. This is nicer than the vq/ version, which will fail, at least for some programs, because brute force is not included). Oh yah, latticehint'ing the trivial lattice (a single quant-level) is not handled correctly. Don't try it. Hard-wire those guys directly to the first item (the '0') in secondstages[] (?name?)>At this point, I did a bunch of Perl or sed commands to produce the final mode_XX.h file, and use yet more Perl/'C' hackery to read the "final" mode_XX.h file in at run time into the encoder to actually use the new trial mode. All [my code] totally unclean, and no way it's good enough to be included in the vorbis trunk, unfortunately. There's been so much to do, that I just stop when it works. Right now I'm way behind in updating my docs from beta2-floundering-newbie to something slightly less confused-sounding and appropriate to beta3. If you want to see what I've been using, and promise not to hang yourself when you see the sheer ugliness of how it "works", send me an e-mail. I'ld be more than glad for someone to streamline the whole process in a way that had a nonzero chance of being accepted into the cvs tree!> It's been some time, so sorry I havn't been much help.. :( But with Monty > very busy, I figured something is better then nothing..So you now have two somethings. I'll let you judge whether it's better or worse than nothing. Erik.> > --- >8 ---- > List archives: http://www.xiph.org/archives/ > Ogg project homepage: http://www.xiph.org/ogg/ > To unsubscribe from this list, send a message to 'vorbis-dev-request@xiph.org' > containing only the word 'unsubscribe' in the body. No subject is needed. > Unsubscribe messages sent to the list will be ignored/filtered. >************ 96 **************** residuesplit residue_0.vqd 0,128,32 res0_96_128 0,,32 ,1.5,32 ,2.5,32 ,6.5,32 ,1 residuesplit residue_1.vqd 0,1024,32 res0_96_1024 0,,32 9,2,32 ,2,32 21,4,32 ,4, 0 0 1 0 1 2 0 1 2 3 5 0 1 2 3 5 8 11 0 1 2 3 5 8 11 14 17 20 24 28 31 35 39 43 47 51 56 61 66 71 72 78 84 90 96 102 0 0 1 0 1 0 1 3 0 1 3 0 1 3 5 8 0 1 3 5 8 11 14 0 1 3 5 8 11 14 17 20 24 28 31 35 39 43 47 51 56 61 66 71 72 78 84 90 96 102 huffbuild res0_96_128aux.vqd 0,4,2 huffbuild res0_96_1024aux.vqd 0,32,2 latticebuild res0_128_128_1.vql >! res0_96_128_1.vqh latticehint res0_96_128_1.vqh >! temp.vqh restune temp.vqh res0_96_128_1.vqd >! res0_96_128_1.vqh latticehint res0_96_128_1.vqh >! temp.vqh mv temp.vqh res0_96_128_1.vqh latticebuild res0_128_128_2.vql >! res0_96_128_2.vqh latticehint res0_96_128_2.vqh >! temp.vqh restune temp.vqh res0_96_128_2.vqd >! res0_96_128_2.vqh latticehint res0_96_128_2.vqh >! temp.vqh mv temp.vqh res0_96_128_2.vqh latticebuild res0_128_128_3.vql >! res0_96_128_3.vqh latticehint res0_96_128_3.vqh >! temp.vqh restune temp.vqh res0_96_128_3.vqd >! res0_96_128_3.vqh latticehint res0_96_128_3.vqh >! temp.vqh mv temp.vqh res0_96_128_3.vqh latticebuild res0_128_128_4.vql >! res0_96_128_4.vqh latticehint res0_96_128_4.vqh >! temp.vqh restune temp.vqh res0_96_128_4.vqd >! res0_96_128_4.vqh latticehint res0_96_128_4.vqh >! temp.vqh mv temp.vqh res0_96_128_4.vqh latticebuild res0_128_128_5.vql >! res0_96_128_5.vqh latticehint res0_96_128_5.vqh >! temp.vqh restune temp.vqh res0_96_128_5.vqd >! res0_96_128_5.vqh latticehint res0_96_128_5.vqh >! temp.vqh mv temp.vqh res0_96_128_5.vqh latticebuild res0_96a_1024_1.vql >! res0_96_1024_1.vqh latticehint res0_96_1024_1.vqh >! temp.vqh restune temp.vqh res0_96_1024_1.vqd >! res0_96_1024_1.vqh latticehint res0_96_1024_1.vqh >! temp.vqh mv temp.vqh res0_96_1024_1.vqh latticebuild res0_96a_1024_2.vql >! res0_96_1024_2.vqh latticehint res0_96_1024_2.vqh >! temp.vqh restune temp.vqh res0_96_1024_2.vqd >! res0_96_1024_2.vqh latticehint res0_96_1024_2.vqh >! temp.vqh mv temp.vqh res0_96_1024_2.vqh latticebuild res0_96a_1024_3.vql >! res0_96_1024_3.vqh latticehint res0_96_1024_3.vqh >! temp.vqh restune temp.vqh res0_96_1024_3.vqd >! res0_96_1024_3.vqh latticehint res0_96_1024_3.vqh >! temp.vqh mv temp.vqh res0_96_1024_3.vqh latticebuild res0_96a_1024_4.vql >! res0_96_1024_4.vqh latticehint res0_96_1024_4.vqh >! temp.vqh restune temp.vqh res0_96_1024_4.vqd >! res0_96_1024_4.vqh latticehint res0_96_1024_4.vqh >! temp.vqh mv temp.vqh res0_96_1024_4.vqh latticebuild res0_96a_1024_5.vql >! res0_96_1024_5.vqh latticehint res0_96_1024_5.vqh >! temp.vqh restune temp.vqh res0_96_1024_5.vqd >! res0_96_1024_5.vqh latticehint res0_96_1024_5.vqh >! temp.vqh mv temp.vqh res0_96_1024_5.vqh latticebuild res0_96a_1024_6.vql >! res0_96_1024_6.vqh latticehint res0_96_1024_6.vqh >! temp.vqh restune temp.vqh res0_96_1024_6.vqd >! res0_96_1024_6.vqh latticehint res0_96_1024_6.vqh >! temp.vqh mv temp.vqh res0_96_1024_6.vqh latticebuild res0_96a_1024_7.vql >! res0_96_1024_7.vqh latticehint res0_96_1024_7.vqh >! temp.vqh restune temp.vqh res0_96_1024_7.vqd >! res0_96_1024_7.vqh latticehint res0_96_1024_7.vqh >! temp.vqh mv temp.vqh res0_96_1024_7.vqh --- >8 ---- List archives: http://www.xiph.org/archives/ Ogg project homepage: http://www.xiph.org/ogg/ To unsubscribe from this list, send a message to 'vorbis-dev-request@xiph.org' containing only the word 'unsubscribe' in the body. No subject is needed. Unsubscribe messages sent to the list will be ignored/filtered. --- >8 ---- List archives: http://www.xiph.org/archives/ Ogg project homepage: http://www.xiph.org/ogg/ To unsubscribe from this list, send a message to 'vorbis-dev-request@xiph.org' containing only the word 'unsubscribe' in the body. No subject is needed. Unsubscribe messages sent to the list will be ignored/filtered.
Possibly Parallel Threads
Wisdom of the Ancients
